<Data analysis was performed using the Python - Second Edition> Chapter pandas entry reading notes

"Data analysis was performed using the Python-2nd Edition" Chapter V pandas Getting Started - Basic objects, operations, rules

Data structure introduced 5.1 pandas

1) Series: one-dimensional array is similar to the object tag (i.e., index) + data (a

In [11]: obj = pd.Series([4, 7, -5, 3])
In [12]: obj
Out[12]:
0 4
1 7
2 -5
3 3
dtype: int64
  • On the left is the row index, the value of the right side of the data
    line index Default: 0- (N-1): You can specify the row index
    Series: obj.value obj.index obj: Example series of
In [13]: obj.values
Out[13]: array([ 4, 7, -5, 3])
In [14]: obj.index # like range(4)
Out[14]: RangeIndex(start=0, stop=4, step=1)
  • Set the row index: 1. Initialization 2. Subsequent assignments, provided
In [15]: obj2 = pd.Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])
In [16]: obj2
Out[16]:
d 4
b 7
a -5
c 3
dtype: int64
In [17]: obj2.index        #可以根据索引选取值
Out[17]: Index(['d', 'b', 'a', 'c'], dtype='object')

In [18]: obj2['a']
Out[18]: -5
In [20]: obj2[['c', 'a', 'd']]    #根据索引顺序显示,Series显示是按行显示(不同于numpy数组
Out[20]:
c 3
a -5
d 6
dtype: int64
  • After using numpy function or operation -> retention index position
    Series: fixed length ordered dictionary -> index mapped to the data value / function can be used instead of the dictionary parameters:
In [24]: 'b' in obj2
Out[24]: True
  • Series dictionary can be created by: -> You can also pass sorted or different dictionary key to change the order
In [26]: sdata = {'Ohio': 35000, 'Texas': 71000, 'Oregon': 16000, 'Utah': 5000}
In [27]: obj3 = pd.Series(sdata)
In [28]: obj3
Out[28]:
Ohio 35000
Oregon 16000
Texas 71000
Utah 5000
dtype: int64

In [29]: states = ['California', 'Ohio', 'Oregon', 'Texas']
In [30]: obj4 = pd.Series(sdata, index=states)    #最后值是按索引决定
In [31]: obj4
Out[31]:
California NaN    #缺失值NaN
Ohio 35000.0
Oregon 16000.0
Texas 71000.0
dtype: float64
  • pd.isnull () / pd.notnull (): detecting missing data = (applied when the series) obj.isnull ()
In [32]: pd.isnull(obj4)
Out[32]:
California True
Ohio False
Oregon False
Texas False
dtype: bool

In [33]: pd.notnull(obj4)
Out[33]:
California False
Ohio True
Oregon True
Texas True
dtype: bool
  • Seies important functions: calculation based on an index tab do automatic alignment operation data (a step corresponding to the natural connection made, and the operation is the inner join other data alignment join =

  • obj.name: Column name obj.index.name data columns: name row index of the row (column index can be seen as the Series column index

2)DataFrame

  • DataFrame: tabular data structure indexed row / column index = Series composed of a plurality of dictionaries (dictionary keys: a column index

  • frame.index: row index frame.column: column index

  • Stores a two-dimensional structure higher dimensional hierarchical structure table index columns of data points = the dimension line =

  • The most commonly used construction DataFrame: ①: long lists of incoming or numpy array of dictionaries, etc.

  • data = { 'state': [ 'Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada', 'Nevada'], # each list / array is a
    'year': [2000, 2001 , 2002, 2001, 2002, 2003],
    'POP': [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]}

  • frame = pd.DataFrame (data)
    line index set: = Series disposed row index // frame: examples of DataFrame

  • frame.head (): Select the first five lines of the display

In [46]: frame.head()
Out[46]:
pop state year
0 1.5 Ohio 2000
1 1.7 Ohio 2001
2 3.6 Ohio 2002
3 2.4 Nevada 2001
4 2.9 Nevada 2002

In [47]: pd.DataFrame(data, columns=['year', 'state', 'pop'])    #指定列序列,即给列名的显示顺序
Out[47]:
year state pop
0 2000 Ohio 1.5
1 2001 Ohio 1.7
2 2002 Ohio 3.6
3 2001 Nevada 2.4
4 2002 Nevada 2.9
5 2003 Nevada 3.2
  • DataFrame each column = Series: frame [column] -> Assignment column is added automatically if there is no column of the column (columns only add rather than using a Frame frame.column [column])
In [51]: frame2['state']    #可用于查询也可用于添加
Out[51]:
one Ohio
two Ohio
three Ohio
four Nevada
five Nevada
six Nevada
Name: state, dtype: object

In [52]: frame2.year    #可用于查询但不可用于添加
Out[52]:
one 2000
two 2001
three 2002
four 2001
five 2002
six 2003
Name: year, dtype: int64
  • DataFrame can use this column index in two ways: frame [colum] applies to any column name, frame.column only when the column name is justified in order to use python variable name (full view

  • DataFrame use to obtain location data attributes loc :(. Loc column index with a direct relationship with quote looks like a copy of ???)

  • .iloc and loc different:
    1. iloc row index is always the number of rows sequence, the index column is a column name, the column can be ordinal number; loc column and row index is only the name of the column row

    2. iloc only be used for queries; and loc can be used to query can also be used to assign, add a new column (loc faster

In [53]: frame2.loc['three']    #df.loc[[1, 5], ['b', 'c']]  , df.loc[1:4,'b'] 取多列/多行的方式
Out[53]:
year 2002
state Ohio
pop 3.6
debt NaN
Name: three, dtype: object
  • del Delete Column
In [63]: del frame2['eastern']
  • frame.T transpose

  • The most commonly used construction DataFrame: ②: Nested dictionary or ③Series dictionary

In [65]: pop = {'Nevada': {2001: 2.4, 2002: 2.9},
....: 'Ohio': {2000: 1.5, 2001: 1.7, 2002: 3.6}}

In [66]: frame3 = pd.DataFrame(pop)    #可以用内层键作为行索引,也可以指定index作为行索引
In [67]: frame3
Out[67]:
Nevada Ohio
2000 NaN 1.5
2001 2.4 1.7
2002 2.9 3.6

In [70]: pdata = {'Ohio': frame3['Ohio'][:-1],
....: 'Nevada': frame3['Nevada'][:2]}
In [71]: pd.DataFrame(pdata)
Out[71]:
Nevada Ohio
2000 NaN 1.5
2001 2.4 1.7
  • Line Name frame.index.name frame.column.name index of the row of column names (index index index of the column
In [72]: frame3.index.name = 'year'; frame3.columns.name = 'state'
In [73]: frame3
Out[73]:
state Nevada Ohio
year
2000 NaN 1.5
2001 2.4 1.7
2002 2.9 3.6
  • frame.value // frame[column].value

  • dtype = object :( compatible data structure types of all columns: python type)

3) index object

  • row column index index index columns -> collectively Index object
    index object after dataframe / series new immutable -> index object that the security data structures shared among a plurality of
    index similar set (both are not the same set of fixed-size
In [85]: frame3
Out[85]:
state  Nevada  Ohio
year               
2000      NaN   1.52001      2.4   1.72002      2.9   3.6
In [86]: frame3.columns
Out[86]: Index(['Nevada', 'Ohio'], dtype='object', name='state')
In [87]: 'Ohio' in frame3.columns
Out[87]: True
In [88]: 2003 in frame3.index
Out[88]: False
  • Difference: pandas The index may contain duplicate label (not duplicate unique set of
In [89]: dup_labels = pd.Index(['foo', 'foo', 'bar', 'bar'])
In [90]: dup_labels
Out[90]: Index(['foo', 'foo', 'bar', 'bar'], dtype='object')
  • index commonly used methods: Usually taken individually used, index_tes = pd.Index () or = frame.columns / obj.index (assignment reference, can be individually removed

5.2 pandas basic data processing functions

1) re-index

  • obj.reindex ([weight Index List]): reindexing new object is re-arrangement structure according to the ordinal index of the original, if the original index value does not exist, NaN missing values ​​are introduced

    reindex参数:Dataframe.reindex(index=None,columns=None,**kwargs)

In [91]: obj = pd.Series([4.5, 7.2, -5.3, 3.6], index=['d', 'b', 'a', 'c'])
In [92]: obj
Out[92]:
d 4.5
b 7.2
a -5.3
c 3.6
dtype: float64
In [93]: obj2 = obj.reindex(['a', 'b', 'c', 'd', 'e'])    #NaN值可以填充也可以插值等
In [94]: obj2
Out[94]:
a -5.3
b 7.2
c 3.6
d 4.5
e NaN
dtype: float64
  • Ordered time-series data, reindex interpolation processing may be required: Substituted default value entry NaN -> may be implemented using an interpolation method parameter items (e.g. ffill: forward-filling
    (method: {None, 'backfill ' / 'bfill', 'pad' / 'ffill', 'nearest'}, optional. method parameter options
In [95]: obj3 = pd.Series(['blue', 'purple', 'yellow'], index=[0, 2, 4])
In [96]: obj3
Out[96]:
0 blue
2 purple
4 yellow
dtype: object
In [97]: obj3.reindex(range(6), method='ffill')    #填补原索引中未有的NaN项
Out[97]:
0 blue
1 blue
2 purple
3 purple
4 yellow
5 yellow
dtype: object

2) discard entries on a specified axis

  • frame.del (): Removes the specified column

  • obj.drop (): Delete the specified axis item (line / one parameter: axis / inpalce

In [107]: new_obj = obj.drop('c')    #默认指定行:axis=0 / 'index'
In [108]: new_obj
Out[108]:
a 0.0
b 1.0
d 3.0
e 4.0
dtype: float64
In [113]: data.drop('two', axis=1)    #axis=1 删除指定列
Out[113]:    
one three four
Ohio 0 2 3
Colorado 4 6 7
Utah 8 10 11
New York 12 14 15

In [115]: obj.drop('c', inplace=True)    #就地修改原对象,不会返回新对象
In [116]: obj
Out[116]:
a 0.0
b 1.0
d 3.0
e 4.0
dtype: float64

3) index, and the filter selection

  • obj []: Index / Slice -> default row index (ordinary arithmetic python different sections: the end of the index value comprises
In [121]: obj[2:4]    #各种切片运算
Out[121]:
c 2.0
d 3.0
dtype: float64
In [122]: obj[['b', 'a', 'd']]
Out[122]:
b 1.0
a 0.0
d 3.0
dtype: float64
In [123]: obj[[1, 3]]
Out[123]:
b 1.0
d 3.0
dtype: float64
In [124]: obj[obj < 2]    #布尔型数组选取索引
Out[124]:
a 0.0
b 1.0
dtype: float64
In [126]: obj['b':'c'] = 5    #利用切片进行赋值(视图

4) using the .loc be selected and .iloc

  • frame.loc ([], []): the left column of the right row LOC (column) line; loc (:, rows) a (.loc index comprises index does not include the final index .iloc last index

  • DataFrame Index Options

5) integer index

  • = pd.Series Ser (np.arange (. 3.))
    Ser
    Ser [-1] # integer index of the last row

  • Integer index (built-index) may conflict with integer index series generated: For non-integer index is not generated index

In [144]: ser
Out[144]:
1 0.0
2 1.0
3 2.0
dtype: float64
  • ser [1] => 0.0 or finger means 1.0, a contradiction bug

    It is generally accurate to use the index with a unified .loc (label) and .iloc (integer)

In [147]: ser[:1]
Out[147]:
0 0.0
dtype: float64
In [148]: ser.loc[:1]
Out[148]:
0 0.0
1 1.0
dtype: float64
In [149]: ser.iloc[:1]
Out[149]:
0 0.0
dtype: float64

6) arithmetic operations and data alignment

  • Different arithmetic = index object database connected NATURAL (alignment data) arithmetic + -> not overlapped with the index value is introduced NaN, missing values ​​will be propagated in an arithmetic operation process

7) In the arithmetic process filling value

  • If you do not want to introduce value NaN / propagation missing values ​​-> a padding value arithmetically r + + fill_value arithmetically use: an arithmetic operation inversion parameters
In [171]: df1.add(df2, fill_value=0)
Out[171]:
a b c d e
0 0.0 2.0 4.0 6.0 4.0
1 9.0 5.0 13.0 15.0 9.0
2 18.0 20.0 22.0 24.0 14.0
3 15.0 16.0 17.0 18.0 19.0
In [172]: 1 / df1
Out[172]:
a b c d
0 inf 1.000000 0.500000 0.333333
1 0.250000 0.200000 0.166667 0.142857
2 0.125000 0.111111 0.100000 0.090909
In [173]: df1.rdiv(1)    #df1.div(1)=df1/1    df1.rdiv(1)=1/df1 反转了参数两者
Out[173]:
a b c d
0 inf 1.000000 0.500000 0.333333
1 0.250000 0.200000 0.166667 0.142857
2 0.125000 0.111111 0.100000 0.090909

8) between arithmetic and Series DataFrame

  • Arithmetic (operation line) and between the DataFrame Series: Series according to the index matching dataframe columns for row broadcasting boardcasting along a row (series to make a particular row, and each row do arithmetic dataframe

    If both do not have a common index, and arithmetic operations as not to overlap partially filled with a NaN value; also be filled with a value arithmetically

  • Column computation: must be arithmetically (with the row arithmetic operation can also be directly arithmetically symbols)
    in the arithmetic process parameter passing (axis = 'index' or axis = 0) to be broadcast listed in the column

9) applications and a mapping function

1. numpy of ufuncs (Group series element method) may be the operation target pandas

In [192]: np.abs(frame)
Out[192]:
b d e
Utah 0.204708 0.478943 0.519439
Ohio 0.555730 1.965781 1.393406
Texas 0.092908 0.281746 0.769023
Oregon 1.246435 1.007189 1.296221

2. The function to effect a one-dimensional array of columns / rows each formed by: using the apply method dataframe

  • (Many common array of statistical methods have been implemented dataframe, such as sum / mean, so do not need apply method)
In [193]: f = lambda x: x.max() - x.min()            #计算一个Series的最大最小值之差,每列都执行了一次,结果返回值是一个Series,列作为新对象索引
In [194]: frame.apply(f)                                #将f作用于frame的各列(默认各列    #f可以用def也可以用lambda函数
Out[194]:                                                #函数必须是作用一行数据(如果是元素级python函数,则需要用applymap:意思就是作用到每个数据的函数
b 1.802165                                                #之所以叫applymap:是因为Series中含有一个应用元素级的map方法:frame['e'].map(f)
d 1.684034
e 2.689627
dtype: float64
In [196]: def f(x):
.....: return pd.Series([x.min(), x.max()], index=['min', 'max'])    #也可以返回多个Series,用了多个算术方法
In [197]: frame.apply(f)
Out[197]:
b d e
min -0.555730 0.281746 -1.296221
max 1.246435 1.965781 1.393406

10) sorting and ranking

  • sorting: The index lexicographically ordering (Dictionary
  • () Obj.sort: Sort defaults
  • obj.sort_index (): The sorted index (specified axis), a new object returns a sorted (series / dataframe
In [204]: frame.sort_index()    
Out[204]:
d a b c
one 4 5 6 7
three 0 1 2 3
In [205]: frame.sort_index(axis=1)
Out[205]:
a b c d
three 1 2 3 0
one 5 6 7 4
In [206]: frame.sort_index(axis=1, ascending=False)    #默认升序排序;False则是降序排序
Out[206]:
d c b a
three 0 3 2 1
one 4 7 6 5
  • obj.sort_value (): sorted according to the value, the new object returns a sorted
In [209]: obj = pd.Series([4, np.nan, 7, np.nan, -3, 2])
In [210]: obj.sort_values()
Out[210]:        #NaN值放到最后
4 -3.0
5 2.0
0 4.0
2 7.0
1 NaN
3 NaN
dtype: float64
In [211]: frame = pd.DataFrame({'b': [4, 7, -3, 2], 'a': [0, 1, 0, 1]})
In [213]: frame.sort_values(by='b')    #通过by参数指定按照某列的值进行排序
Out[213]:
a b
2 0 -3
3 1 2
0 0 4
1 1 7
In [214]: frame.sort_values(by=['a', 'b'])    #多列值排序,根据by的先后决定权重
Out[214]:
a b
2 0 -3
0 0 4
3 1 2
1 1 7
  • Series & DataFrame.rank (): the value of the object size, ranked by -> 1- (n-1) (default: assign a group for the same level of destruction average position relationship (by default behavior dataframe group, by axis = ' columns' parameters are listed by group)
In [215]: obj = pd.Series([7, -5, 7, 4, 2, 0, 4])
In [216]: obj.rank()    #当出现相同值时,两者取平均;默认按从小到大的顺序排序
Out[216]:
0 6.5    #如原本是第5和6,则两者都取平均为6.5
1 1.0
2 6.5
3 4.5
4 3.0
5 2.0
6 4.5
dtype: float64
In [217]: obj.rank(method='first')    #同时也可以设置method参数:first;表示第一次出现的数大于第二次出现的数,序数则也更大
Out[217]:
0 6.0        #于是第一次出现的7排6,第二次出现的7排5
1 1.0
2 7.0
3 4.0
4 3.0
5 2.0
6 5.0
dtype: float64
In [218]: obj.rank(ascending=False, method='max')    #降序排列,从大到小
Out[218]:
0 2.0
1 7.0
2 2.0
3 4.0
4 5.0
5 6.0
6 4.0
dtype: float64
  • Ranked same level for the destruction of the relationship method

11) shaft repeat tag with the index

  • The only non-mandatory index: obj.index.is_unique-> bool Returns the index value is unique

    Duplicate index value index to return a Series; unique index value returns a scalar index

5.3 Summary and calculated descriptive statistics

  • Common mathematical and statistical methods pandas object: You can have missing values ​​(so do ahead of time to fill the missing values ​​-> as df.sum () df.mean ()

    dataframe return a series / series returns a scalar
    Reduction type / Summary Statistics: Common

In [231]: df
Out[231]:
one two
a 1.40 NaN
b 7.10 -4.5
c NaN NaN
d 0.75 -1.3
In [232]: df.sum()    #默认按列进行操作,返回列索引转行索引
Out[232]:
one 9.25
two -5.80
dtype: float64
In [233]: df.sum(axis=1)    #设置按行进行操作    统计时无视NaN值
Out[233]:
a 1.40
b 2.60
c NaN
d -0.55
In [234]: df.mean(axis='columns', skipna=False)        #skipna要求正视NaN的存在,对于NaN无法进行统计,结果返回NaN
Out[234]:
a NaN
b 1.300
c NaN
d -0.275
dtype: float64
  • Reduction type of statistical methods mathematical parameters:

  • Indirect Statistical Methods: The maximum / minimum index: idxmax () / idxmin ()

In [235]: df.idxmax()    #值是df行最大值对应的索引
Out[235]:
one b
two d
dtype: object
  • Accumulative statistical methods: such as
In [236]: df.cumsum()
Out[236]:
one two
a 1.40 NaN
b 8.50 -4.5
c NaN NaN
d 9.25 -5.8
  • Neither a reduction type, nor is the cumulative type: as describe; generating a plurality of one-time summary statistics
In [237]: df.describe()    #对于数值
Out[237]:
one two
count 3.000000 2.000000
mean 3.083333 -2.900000
std 3.493685 2.262742
min 0.750000 -4.500000
25% 1.075000 -3.700000
50% 1.400000 -2.900000
75% 4.250000 -2.100000
max 7.100000 -1.300000
In [238]: obj = pd.Series(['a', 'a', 'b', 'c'] * 4)    #对于非数值
In [239]: obj.describe()
Out[239]:
count 16
unique 3
top a
freq 8
dtype: object

1) The correlation coefficient Covariance:

+ Correlation / covariance: parameters need to be calculated

import pandas_datareader.data as web
all_data = {ticker: web.get_data_yahoo(ticker)    #股票数据
    for ticker in ['AAPL', 'IBM', 'MSFT', 'GOOG']}
price = pd.DataFrame({ticker: data['Adj Close']    #股票价格
    for ticker, data in all_data.items()})
volume = pd.DataFrame({ticker: data['Volume']    #股票成交量
    for ticker, data in all_data.items()})


In [242]: returns = price.pct_change()    #计算价格的百分比变化(相关时间序列操作
In [243]: returns.tail()    #显示最后五条
Out[243]:
              AAPL GOOG IBM MSFT
Date
2016-10-17 -0.000680 0.001837 0.002072 -0.003483
2016-10-18 -0.000681 0.019616 -0.026168 0.007690
2016-10-19 -0.002979 0.007846 0.003583 -0.002255
2016-10-20 -0.000512 -0.005652 0.001719 -0.004867
2016-10-21 -0.003930 0.003011 -0.012474 0.042096
  • obj.corr (obj2): calculated in two superposed series, the non-NaN, aligned by index coefficient

    obj.cov (obj2): calculated two covariance

In [244]: returns['MSFT'].corr(returns['IBM'])
Out[244]: 0.49976361144151144
In [245]: returns['MSFT'].cov(returns['IBM'])
Out[245]: 8.8706554797035462e-05
  • frame.corr () / frame.cov (): returns the complete correlation / covariance matrix (coefficient matrix itself

  • frame.corrwith (): calculate the correlation coefficients between columns / rows with another Series / dataframe:
    Incoming Series Series return relevant coefficient values (computed for each column)
    incoming dataframe matching column names returned by the correlation coefficient value (index aligned

In [249]: returns.corrwith(returns.IBM)    #同样axis参数可以设置为1/columns使之按行运算
Out[249]:
AAPL 0.386817
GOOG 0.405099
IBM 1.000000
MSFT 0.499764obj.isin
dtype: float64
In [250]: returns.corrwith(volume)    #两个不同dataframe之间的相关系数
Out[250]:
AAPL -0.075565
GOOG -0.007067
IBM -0.204849
MSFT -0.092950
dtype: float64

2) a unique value, the value of the count and membership

  • obj.unique (): to give a unique value in the series array

  • obj.value_counts (): calculated value of the respective number of occurrences of each index value, a value of the number (Default descending
    = pd.value_counts (obj.value, sort = False ): for any array or sequence

  • obj.isin ([Set]): vectoring for determining membership set, data filtering Series / dataframe column; column returns a Boolean column size and the like,
    = pd.Index (small objects) .get_indexer (Large Object) : Analyzing large object column element is included in a small object, comprising a large object returns the index element in the sequence of small objects

In [256]: obj
Out[256]:
0 c
1 a
2 d
3 a
4 a
5 b
6 b
7 c
8 c
dtype: object
In [257]: mask = obj.isin(['b', 'c'])
In [258]: mask
Out[258]:
0 True
1 False
2 False
3 False
4 False
5 True
6 True
7 True
8 True
dtype: bool
In [259]: obj[mask]
Out[259]:
0 c
5 b
6 b
7 c
8 c
dtype: object

In [260]: to_match = pd.Series(['c', 'a', 'b', 'b', 'c', 'a'])
In [261]: unique_vals = pd.Series(['c', 'b', 'a'])
In [262]: pd.Index(unique_vals).get_indexer(to_match)
Out[262]: array([0, 2, 1, 1, 0, 2])
  • For example, to calculate each column appeared histogram how many different numbers:
In [263]: data = pd.DataFrame({'Qu1': [1, 3, 4, 3, 4],
.....: 'Qu2': [2, 3, 1, 2, 3],
.....: 'Qu3': [1, 5, 2, 4, 4]})
In [264]: data
Out[264]:
Qu1 Qu2 Qu3
0 1 2 1
1 3 3 5
2 4 1 2
3 3 2 4
4 4 3 4
In [265]: result = data.apply(pd.value_counts).fillna(0)        #利用apply函数,NaN值填充0
In [266]: result                            #行索引是所有列的唯一值,后面是该值在各列出现的次数
Out[266]:
Qu1 Qu2 Qu3
1 1.0 1.0 1.0
2 0.0 2.0 1.0
3 2.0 2.0 0.0
4 2.0 0.0 2.0
5 0.0 0.0 1.0

Guess you like

Origin www.cnblogs.com/ymjun/p/11980323.html