Pandas Data Analysis D1

datawhale hands-on data analysis learning record

Section 1 Data Loading and Preliminary Observation

---

1.1 Load data

---

Task 1: Import the library

import numpy as np
import pandas as pd

Task 2: Load the data

1. Absolute path and relative path to read data

df1 = pd.read_csv('./train.csv')
df2 = pd.read_csv(u'F:/Note/hands-on-data-analysis-master/第一单元项目集合/train.csv', encoding = "gbk")

The solution to the Chinese path in the path: python3 processes the Chinese path
encoding represents the encoding format of the file, commonly used encodings are utf-8, utf-16, gbk, gb2312, gb18030, etc. If the wrong encoding is specified, the data cannot be read, and the IPython interpreter will report a parsing error.

2. pd.read_table() read data
   Use pd.read_table() to read the separator can not be read. It can be processed with the split parameter sep.

df3 = pd.read_table('./train.csv', sep = ',', engine = 'python')

3. The difference between '.tsv' and '.csv' suffixes
   Read csv and tsv files and convert between them

• csv file: comma delimited file, can be opened with excel
• tsv file: tab separated file, can be opened with text document

Therefore, septwo types of files can be processed using the parameter: sep = '\t',sep = ','

Task 3: Every 1000 rows are a data module, read block by block

Chunked reading of Pandas datasets

• Read directly in blocks: Add chunksizeparameters to read_csv()
  The returned reader is not a DataFrame, but an iterable object (iteration). It should be noted that this iterable object cannot be accessed with subscripts

reader = pd.read_csv('./train.csv', usecols = ['PassengerId', 'Survived', 'Sex'], chunksize = 5)
for i in reader:
	print(i)

• First read the data set as an iterable object, and then read it in chunks.
  Specify the parameter iterator as True in the read_csv() method, and then traverse the reader in chunks. Pay attention to the get_chunk() method used and the parameters inside. Define chunk size

reader = pd.read_csv('./train.csv', usecols = ['PassengerId', 'Survived', 'Sex'], iterator= True)

while True:
    try:
        print(reader.get_chunk(5))
    except StopIteration:
        break

The category that type(reader)can be obtained through is an iterable object. The advantage of this block-by-block reading is to avoid insufficient memory after reading data at one time. pandas.read_csv reads large files in blocks (chunksize, iterator=True)reader<class 'pandas.io.parsers.readers.TextFileReader'>

Task 4: Change the header to Chinese

df = pd.read_csv('./train.csv')
df.columns = [' 乘客ID', '是否幸存', '乘客等级(1/2/3等舱位)',
             '乘客姓名','性别', '年龄', '堂兄弟/妹个数', '父母与小孩个数', 
             '船票信息', '票价', '客舱','登船港口']

df.rename(columns = {
    
    '性别': 'Sex'}, inplace = True)
df.head()

1.2 Preliminary exploration

---

# 查看数据基本信息
df.info()
# 观察表格前10行和后15行
df.head(10)
df.tail(15)
# 判断数据是否为空，为空的地方返回True，其余地方返回False
df.isnull()

1.3 Save data

---

df.to_csv('./train_chinese.csv', index = False, encoding = 'utf-8')

---

Section 2 Pandas Basics

---

1.4 Understanding the data

---

Task 1: There are two data types DateFrame and Series in pandas

s = pd.Series(data = [100, 'a', {
    
    'dict':5},
				index = pd.Index(['id', 20, 'third', name = 'my_idx'),
				dtype = 'object',
				name = 'my_name')
s

df = pd.DataFrame(data = np.array([['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada', 'Nevada'],
                         [2000, 2001, 2002, 2001, 2002, 2003],
                         [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]]).T,
                 index = ['row_{}'.format(i) for i in range(6)],
                 columns = ['state', 'year', 'pop'])
df

Tasks 2 and 3: Load the file and view the columns

df = pd.read_csv('./train.csv')
df.columns

Task 4: View all values ​​of the "Cabin" column

df['Cabin'].unique()

Task 5: Delete redundant columns

df_test = pd.read_csv('./test_1.csv')
df_test_copy = df_test.drop([i for i in df_test.columns if i not in df.columns],
                            axis = 1)
df_test_copy.info()

Task 6: Column elements are hidden, observe several other column elements

df.drop(['PassengerId','Name','Age','Ticket'], axis = 1).info()

1.5 Logic of screening

---

Task 1 and 2: "Age" is the filter condition

# 显示年龄在10岁以下的乘客信息
df[df['Age']<10]
# 将年龄在10岁以上和50岁以下的乘客信息显示出来，并将这个数据命名为midage
midage = df[(df['Age'] > 10) & (df['Age'] < 50)]

Task 3: Display the data of "Pclass" and "Sex" in row 100 of the midage data

midage = midage.reset_index(drop=True)
midage.loc[[100], ['Pclass', 'Sex']]

Tasks 4 and 5: Display data with loc and iloc methods

midage.loc[[100,105,108],['Pclass','Name','Sex']] 
midage.iloc[[100,105,108],[2,3,4]]

The difference between loc() and iloc() in Pandas.DataFrame

Section III Exploratory Data Analysis

Sorting, arithmetic calculations, and use of calculation description functions

---

1.6 Know your data?

---

Task 1: Use Pandas to sort the sample data, requiring ascending order

frame = pd.DataFrame(np.arange(8).reshape((2, 4)), 
                     index=['2', '1'], 
                     columns=['d', 'a', 'b', 'c'])
frame

# 以 c 的值排序，升序               
frame.sort_values(by='c', ascending=True)

Summary:
KaTeX parse error: Expected 'EOF', got '#' at position 27: …ort_values(by='#̲#',axis=0,ascen... Pandas-sorting function sort_values(
)

Task 2: Comprehensive sorting by two columns of ticket price and age (descending order)

df.sort_values(by=['票价', '年龄'], ascending=False).head()

Task 3: Use Pandas to perform arithmetic calculations and calculate the result of adding two DataFrame data

frame1_a = pd.DataFrame(np.arange(9.).reshape(3, 3),
                     columns=['a', 'b', 'c'],
                     index=['one', 'two', 'three'])
frame1_b = pd.DataFrame(np.arange(12.).reshape(4, 3),
                     columns=['a', 'e', 'c'],
                     index=['first', 'one', 'two', 'second'])
frame1_a + frame1_b

Task 4: How to calculate the number of members of the largest family on board the Titanic?

max(df['堂兄弟/妹个数'] + df['父母与小孩个数'])

Task 5: Learn to use the Pandas describe() function to view basic statistics of data

df.describe()

Task 6: Look at the basic statistical data of the fare, parents and children in the Titanic data set, what can you find?

df['票价'].describe()

The fare has a large variance and may contain more information. Combined with the previous fare and age ranking results, the fare may be a key factor.