(Master this, basically any perfect follow their own ideas, change the column.)
background:
Recently, a process of converting data to excel. The amount of data is so large, there are about 30 million. We need some rows become columns, and some columns become rows. The operation itself is more irritable.
Not to mention the amount of data in the case reached the hundreds of thousands, excel basically stuck up.
A city together as one
Type 2 type air is separated by the column
First posted sample:
After the conversion results:
Distressed for a long time.
practice:
melt function to explain,
frame, - data sets need to be addressed
id_vars = None, - no need to change the column
value_vars = None, - the need to convert the column name, if the rest of the columns all have to convert, do not write a
var_name = None, - set the name of the corresponding dimension
value_name = "value", - setting the corresponding metric name
col_level = None, - do not know
first_data_2 = self.pd.melt(deal_data, id_vars=['date', 'hour', 'type'], value_vars=city_data, var_name='city', value_name='count_clue').fillna(0)
Here, deal_data is the need to deal with data sets, id_vars the same column, date, hour, type, need to convert array of columns city_data [], in theory, should not have to fill all the conversion below.
Corresponding dimension name: city, together corresponding to the measure. count_clue.
This put the columns together.
The results show
Then we use piovt function, it also listed the column type transformation take place.
first_data_3 = self.pd.DataFrame( self.pd.pivot_table(first_data_2, index=['date', 'hour', 'city', ], columns='type', values='count_clue'))
piovt_table, I know too little. The Lifted
Posted a link to my understanding, index is the need of the column, then columns is to expand the column, value is the value you want to expand, that's all.
https://zhuanlan.zhihu.com/p/31952948