[Study notes] 1 hour a day to learn data analysis, mining, cleaning, visualization from entry to actual project

Learn data analysis, mining, cleaning, and visualization for 1 hour a day.
Guide from entry to project practice: How to become a qualified business data analyst?

Data Analyst Career Analysis

technology stack

Capability interpretation

Academic requirements

Data Analysis Development Prospects

Data Analysis Development - Salary Level

1. Basic operation of Excel (Tina's video)

Knowledge points:

• data format, cell format
• Functions: summary statistics, text processing, time calculations, search and match
• visualization
• pivot table

Power Query will be taught in Power BI
After learning Python, give up VBA (so no need to learn)

1.1 cells

Select mode, input mode, edit mode (double-click to enter).
Press F2 to toggle between editing and typing mode.
"Ctrl+shift+down" can be used to check if there is a vacancy in a column.
"alt+enter" can break a line within a cell.
In the format, you can choose to automatically adjust the column width and row height.

1.2 Data format

Text format, left by default.
Numeric format, default to the right.

Some functions work on text, some on numbers

1.2.1 Custom format

The format can be customized.

• If set ##“次”, enter the value 25, it will display 25 times, but it is still in the numerical format.
• If it is set [>=75][蓝色]##"次";[<75][红色]##"次";, if it is greater than or equal to 75, it will display blue, followed by the unit times.

This changes the color of the value itself.
# Represents the original data format, 0 represents the numerical format, and 0000 represents insufficient digits, and fills in 4 digits.

1.2.2 Toolbar - Conditional Formatting

Data Bars, Color Scales, Rules

1.3 Protection function

• Protect Sheet
  Select Review, select "Protect Sheet"
  
• Protect Workbook
  Select Review, select "Protect Workbook"
  
• Partial cell editing
  Select review, select "Allow editing area"
  

1.4 Quick data input

Fill Handle - left mouse button down

Populate with custom columns

Multiple discontinuous cells with the same data

ctrl+鼠标左键Select multiple discontinuous cells, then enter content, ctrl+回车and then complete all selected cells.
This method can also be used to delete discontinuous cells.

text memory input

data verification

On the Data tab, choose Data Validation.

1.5 Import data

Note that if the data source is lost, so is the table information.
If the data source is modified, the modification can also be seen after the table information is refreshed. To click Refresh on the Table Design tab.

1.6 Data Types

1.7 Convert text to value

Adding a single quotation mark in English format in front of the cell will default to the left.

1.8 Date Standardization

1.9 Batch delete single quotes

Using the Format Painter

1.10 Key combination Alt+=

Alt+= will automatically sum the above elements.

1.11 Quick selection area

"ctrl+shift+direction" can quickly select cells in a certain direction.
This method can be used to find vacancies.

1.12 Freezing cells

1.13 Select cells

If there is a vacancy in a cell, neither the numerator nor the denominator will record this cell when calculating the average.

1.14 Data classification and aggregation

On the Data tab - select Subtotals

1.15 Copy Subtotal Results

If you directly copy the subtotal results in 1.14, even if the classification is hidden, the hidden cells will still be copied.
At this time, you can use "Find and Select", and then select "Visible Cells", so that if you copy again, hidden cells will not be copied.

1.16 Paste Special - Operation

1.17 The search function counts the number of the same fill color

1.18 Cell matching replacement

1.19 Use the auxiliary column-interlaced to insert a blank row

How to make the above table achieve the effect that one row has data, one row is empty, and one row has data?
Use the auxiliary row, first add the arithmetic sequence 1, 2, ...
and then add 1.1, 2.1, ...
to the blank line at the end, and finally sort the auxiliary row in ascending order, and then delete the auxiliary column to achieve the effect.

1.20 Paste Special, combine two columns of data into one column

1. Supplement Excel (ailsa video)

1.21 Lesson 1

The most intelligent shortcut key ctrl+e

smart fill

Text Stitching &

1.22 Formulas and functions

ctrl+~ can switch to display the original formula or value.

1.22.1 Address reference

Relative reference: you change it, such as A2:A5
Absolute reference: such as $A $2Mixed\; references:\; For\; example$
, whoever is in front ofA2 and A$2

1.22.2 Functions

• logic function

IF(判断条件，True的结果，False的结果) # 判断
AND(条件，条件） # 与
OR(条件，条件) # 或
NOT(条件) # 非

References to text in excel must be in double quotes

• text function

MID(文本，开始字符下标，几个字符长度) # 取子串
LEFT(文本，几个字符长度) # 从左取子串
RIGHT(文本，几个字符长度) # 从右取子串
LEN(文本) # 文本长度
TEXT(文本，文本格式) # 数字转化文本格式
# 文本格式如"yyyy-mm-dd"，可以参考单元格格式中的自定义的书写方法
REPLACE(原始文本，开始字符下标，几个字符长度，替换的内容) # 替换特定位置处的文本

• statistical function

# 简单的
INT() # 向下舍数取整
MOD(除数，被除数) # 求余数
ROUND(数值，几位小数) # 四舍五入，精确到指定小数位
AVERAGE(一组数据) # 求平均
MAX(一组数据) # 求最大
MIN(一组数据) # 求最小
SUM(一组数据) # 求和
COUNT(一组数据) # 数值计数
# 稍微有点难度的
SUMIF(条件区域，条件，求和的数据区域) # 条件求和
SUMIFS(求和的数据区域，条件区域1，条件1，条件区域2，条件2，...) # 多条件求和
COUNTIF(条件区域，条件) # 条件计数
COUNTIFS(条件区域1，条件1，条件区域2，条件2) # 多条件计数

• The application scenario of the lookup and reference function
  VLOOKUP function: the data source is a different table, but the format is similar, and I want to summarize it into one table.
  Returns a value for the row of the lookup object in the lookup range.

VLOOKUP(查找对象(如姓名)，查找区域(如姓名列和值列)，得到哪一列的值，查找方式)：垂直方向查找
# 查找对象必须在查找区域的第一列中，否则找不到
# 查找区域一般需要绝对引用
# 查找方式分为近似匹配(TRUE或1)和精确匹配(FALSE或0)，一般采用后者

Application scenario of the MATCH function: return the position of the search object in the search area

MATCH(查找对象， 查找区域，查找方式)：查找函数

• date function

TODAY() # 当前日期
NOW() # 当前日期和时间
DATE(年，月，日) # 组合得到给定的日期
DATEDIF(开始时间，结束时间，单位) # 计算日期差
# 单位是“D”表示天，“M”表示月，“Y”表示年
YEAR() # 求年
MONTH() # 求月
DAY() # 求日

1.23 PivotTable

1.23.1 PivotTable

When referencing data across tables, "table name +!" will be added before the reference

• Pivot Data
  Step 1: Put the mouse on the data area
  Step 2: Click "Insert Tab", click "Pivot Table"
  
  Step 3: Drag the brand to the row direction, drag the sales to the value direction, and drag the revenue to the value direction.

In the value field setting, you can choose the aggregation operation to be taken, such as summing, averaging, etc.

1.23.2 Precautions

1. Column name cannot be empty
2. cannot be repeated
3. There cannot be blank rows, so fill in the blank values
4. cannot have merged cells
5. Data types are consistent, especially numeric

1.23.3 Common functions

Click on the pivot table and the contextual tabs will appear.

1. The data is wrongly recorded and needs to be higher. How to update the pivot table?
   Answer: If the source data of the pivot table is updated. The pivot table needs to click "Refresh All" in the analysis tab in the context tab to refresh the created pivot table.
2. A new row or column is added to the content of the source data, what should I do with the PivotTable?
   Answer: If you add data inside the data, you only need to refresh it. But adding row or column at the bottom or edge, refresh is also useless. Because the range of the data source does not include these new columns or rows, you need to click "Change Data Source" next to "Refresh".
3. What if you want to arrange the calculation results in a certain order?
   A: Click on the row label and select "Additional sorting options".
4. Want to add calculated columns to your pivot?
   Answer: Select the column to be calculated, then click the "Analysis" tab under the "PivotTable Tools" tab, then click "Fields, Items and Sets", and click "Calculate Fields".
5. Want to add calculated rows to your pivot?
   Answer: Select the row to be calculated, then click the "Analysis" tab under the "PivotTable Tools" tab, then click "Fields, Items, and Sets", and click "Calculate Rows".
6. How to filter pivot content dynamically?
   Answer: Use a slicer. Click the Analysis tab, click Insert Slicer, and select the labels to slice. Select the slicer and press the Delete key to delete the slicer.

Under the "PivotTable Tools-Analysis" tab, you can click "Field List", "+/- button" to modify the display style of the pivot table. Under the "PivotTable Tools - Design" tab, click Report Layout, and click "Show in Table". In the subtotals, you can click "Do not display subtotals".
In "Analysis", you can click "Options", in the layout and format, you can choose "Merge and center cells with labels".

7. How to split multiple worksheets based on a certain category?
   A: Drag "Region" to "Filter" in the "PivotTable Field List", then click "Analyze", "Options", "Show Report Filter Page". In this way, the data of "Northeast" and "North China" will generate a worksheet respectively.
8. How to merge multiple Excel sheets?
   Answer: You need to use the multiple merge dialog box. This function is not available in the function area of ​​the foreground and needs to be set.
   Click the "File" tab, "Options", Customize the Ribbon, drop down in the "Drop-down location selection command", click the command "Not in the Ribbon", find the "Pivot Table and Pivot Chart Wizard", and set it Add it to the "New Group" under "Data Grouping". In this way, you can see "Pivot Table and Pivot Chart Wizard" in the ribbon,
   click "Pivot Table and Pivot Chart Wizard", then click "Multiple Consolidated Data Range", click Next, and then click "Create Single Page field", click Next, select the data area, and click Add.

The premise is that the content and format are extremely similar

9. How to create a pivot chart?
   A: Click Analyze, click PivotChart, and select a chart type (such as a bar chart).

1.24 Charts

slightly

1.25 Data Analysis Process

• Analysis process
  1. Clear requirements
  2. Determine ideas
  3. Process data (data cleaning, most of the time is here)
  4. Analyze data
  5. Display data and write reports
  6. Effect feedback

1.25.1 Data cleaning

• Select a subset
  to remove unnecessary columns
• column name rename
• remove duplicates
• Missing value handling
  Use the countif function to see if there are any null values.
  Solutions for missing values:

1. If the amount of data is relatively small and not important, delete it directly
2. If it is a text type, we will fill it out by human judgment alone
3. Fill missing values, for numeric, mean or median fill

• Consistent processing

二、Jupyter notebook

2.1 Cell state

• Status
  Selected state: the left side of the cell turns blue, and the cell itself can be operated
  Editing state: the left side of the cell turns green, and the cursor is flashing inside the cell, and the internal text can be operated
• State switching
  Selected state-"Edit state: 1. Enter or 2. Mouse click inside the cell
  Edit state-"Selected state: 1.ESC or 2. Mouse click outside the cell

2.2 Cell editing mode

• Edit mode
  Code code mode, Markdown text mode
• Mode switching (two methods)
  1. In the selected state, press y to switch to code mode, press m to switch to markdown mode
  2. Manually switch in the upper drop-down list with the mouse

2.3 Cell operation

Premise: The cell is selected
1. New cell:
Press a: Insert a new cell above the selected cell
Press b: Insert a new cell below the selected cell
2. Delete cell:
Press dd: Delete the selected cell
Press x: cut cell
3. copy cell
press c: copy the selected cell
4. paste
press v: paste the copied cell directly below the selected cell

2.4 run

Ctrl+Enter to run the selected cell, and then continue to select the current cell
Shift+Enter to run the selected cell, and select a cell below it, if there is no cell below it, add a new
ALt+Enter to run the selected cell, and Add a new cell below it

2.5 Help Documentation

1. hlep (object to query)
2. The object to query + ?
3. shift+tab

2.6 Magic commands

• %run
  runs the xx.py file

%run xx.py

• %time
  records the running time of a line of code

%%time 一行代码

• %%time
  records the running time of multiple lines of code

%%time
多行代码

• %timeit and %%timeit
  are similar to %time and %%timeit, but will run multiple times to take the average
• %who
  finds the names of all current objects
• %whos
  not only gets the names of all objects, but also types, and content or information

2.7 IPython input and output history

Jupyter notebook is based on IPython, so
In stores a list of strings composed of all input command strings, and elements can be obtained through In[index].
Out returns all the dictionaries that contain the sequence number of the output command and its output, and the elements can be obtained through Out[index].

3. Numpy

Numpy is the basis of the pandas library.

3.1 The difference between ordinary lists and array ndarray objects in Numpy

The elements in the ordinary list can be of any type, but the elements in the array ndarry must be of the same type (the element types will be automatically unified to facilitate operations).

3.2 Construction method of ndarray

ages = [17, 19, 20, 34]
ages_array = np.array(ages)

3.3 Access method of ndarray

Array elements can be accessed by index
Compared with lists, arrays provide some convenient methods, such as the sum methodndaaray.sum()

3.4 Two-dimensional construction and access

An array *2 will multiply each element in it by 2, and a list multiplied by 2 will copy and expand the original list.

# 构造二维数组
array2 = np.array([[1,2],[3,4]])

3.5 Quickly construct high-dimensional arrays

• Use the np.arange() method to construct an array

# 一个参数，则起点为0，参数为终点，步长为1
x = np.arange(5) # [0, 1, 2, 3, 4]
# 两个参数，则第一参数为起点，第二参数为终点，步长为1
y = np.arange(5, 10) # [5, 6, 7, 8, 9]
# 三个参数，则第一参数为起点，步长为1，第二参数为终点，第三个参数为步长
z = np.arange(5, 10, 0.5)

• Use the np.random module to build random arrays

np.random.randn(n) # 创建服从正太分布的n个随机数组成的一维数组
np.random.randint(整数x, 整数y, [行m, 列n]) # 创建由x到y之间的随机数组成的m*n数组

The ndaaray.reshape((a, b)) method can change the shape of the array into a row and b column.

4. pandas

pandas is widely used for fast data analysis, as well as data cleaning and preparation. Compared with Numpy, pandas is better at handling two-dimensional data.
There are two main data structures in pandas: series and DataFrame.
A series is like a one-dimensional array, but it also contains a set of indices.

4.1 Basic data types of pandas

• create series

import pandas as pd
s1 = pd.Series([“a”, "b", "c"], index=[“x”, "y", "z"])

• Creating a DataFrame
  DataFrame is a container for Series.

import pandas as pd
# 方法1：通过列表创建，可以通过index和col来配置行索引和列索引
a = pd.DataFrame(二维列表, index=[], col=[])

# 方法2.11：通过字典创建，字典键会变成列索引
先创建两个series(或者列表)，用变量names和ages存放
pd.DataFrame({
	"names": names,
	"ages": ages
})

# 方法2.2：如果想让字典键变成行索引，可以通过from_dict方式，并设置orient参数为index
c = pd.DataFrame.from_dict({"a": [1, 3, 5], "b": [2, 4, 6]}, orient="index")

# 方法2.3：通过把DataFrame当字典使用来赋予列
mdate = [1, 3, 4, 5]
score = [10, 20, 30, 40]
a = pd.DataFrame()
a["key1"] = mdata
a["key2"] = score

# 方法3：通过np.arrange()方法来创建二维数组，然后传递给DataFrame
pd.DataFrame(np.arange(12).reshape(3,4), index=[1,2,3], columns=[...])

4.2 Modify index

• DataFrame.set_index() method

# DataFrame.set_index可以将某列设置为行索引，但这样不会改变原DataFrame，而是返回一个新的DataFrame
a.set_index(“日期”)
# 要在原DataFrame上更改，要么直接赋值给原变量，要么使用参数inplace=True（如下）
a.set_index("日期", inplace=True)

• DataFrame.rename() method

# 可以重命名索引，也有inplace参数
a.rename(index={"A": "万科", ...}, columns={"date":"日期"}, inplace=True)

• DataFrame.reset_index() method

# 可以重置索引，让行索引变成常规列，也有inplace参数
a  = a.reset_index()

4.3 Read and write operations of excel and csv files

• file reading-excel

import pandas  as pd
data = pd.read_excel('data.xlsx')
# 指定sheet读取
pd.read_excel("path", sheet_name=1)
# 参数index，以表格中的某列或某些列来设置行索引
# 参数header，以表格中某行或某些行设置列索引
pd.read_excel(path, index=0, header=[0, 1])

• file write-excel

data.to_excel("mdata.xlsx")

• file read-csv

pd.read_csv("csv文件", sep=',', header=[0,1])

• file write-csv

data.to_csv("mdata2.csv")

4.4 Absolute path and relative path

The \ in python has an escape function, such as \n and so on. So just write one \, you need to add r in front to remove the escape function. Or write 2 \.

# windows下绝对路径举例
"E:\\大数据分析\\data.xlsx"
r"E:\大数据分析\data.xlsx"
"E:/大数据分析/data.xlsx" # 这种写法主要用于linux，也可以用于windows(python也可以读取)

4.5 pandas object access and filtering

• Select data by column

b = data["c1"] #读取c1列，返回Series
b = data[["c1"]] # 读取c1列， 返回DataFrame

• Read data by row

d = data.loc["r1"] # 读取r1行，返回成一列的Series
d = data.loc[["r2", "r3"]] # 读取r2行和r3行，返回DataFrame

• Use loc to read an element

records.loc[2, "身高"]

• read the first few lines

df.head(n) # 查看前n行，n不填默认为5

• Filter by specific conditions

data[data["c1"] > 1] # 筛选c1列数字大于1的行

4.6 pandas data operation

• The shape attribute looks at the number of rows and columns

data.shape # 返回(3, 3)

• describe() method to view statistical indicators

df.describe() # 返回每一列的计数，均值等

• info() method to view table data information

df.info() # 可以看到每一列数据的数据类型和非空数据个数

• The value_counts() method counts each column of data and its frequency

data["c1"].value_counts()

• pandas supports column-to-column direct operations

df["sum"] = df["c1"] + df["c2"]

• pandas supports aggregation methods

df.sum() # 默认对列进行求和，得到Series
df.mean() # 默认对列求均值
# 设置axis=1可以求行方向的
df.sum(axis=1)
df.mean(axis=1)

4.7 pandas mapping processing

• The map() method maps column data

先定义map_funct方法
records["身体"].map(map_funct) # 对身高列做map方法

4.8 pandas null filling and lookup

If the data in the table displays NaN (meaning Not a Number), it means a null value.

• fillna() method fills empty values

data.fillna(value=0, inplace=True)

• The isnull() method looks for null values

records.isnull() # 对每个数做空值检验，是空值的位置为Trure

# 求出每一列空值的个数
records.isnull().sum()
# 求出每一列空值的占比
records.isnull().mean()

Five, Matplotlib data visualization basis

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
%matplotlib inline # 这个魔法指令保证图像一定能渲染到jupyter上

5.1 Line chart plt.plot()

x = [1,2 3]
y = [2,4,7]
plt.plot(x, y)
plt.show() # 建议写上。不用的话，在jupyter上也能看到，但是是在Out里

5.2 Bar chart (histogram) plt.bar()

plt.bar(x, y, color=np.random.random((4,3))) # 利用0-1随机浮点数配置rgb颜色
plt.show()

5.3 Scatter plot plt.scatter()

x # 横轴数据
y #纵轴数据
size # 大小，也可以用来表示第三维数据
plt.scatter(x, y, s=size) # 这样就成了气泡图

5.4 Quick plotting techniques for the pandas library

Pandas draws a line chart

df # 10行3列的dataframe
df.plot() # 会把3列化成3条折线
df["人均收入"].plot(kind='line') # 和df.plot()一样
df["人均收入"].plot(kind='bar') # 绘制柱状图

df.T can transpose dataframe

5.5 Add text description

• title, x-axis label, y-axis label, legend

plt.plot(x, y, label="line of sin") # label就是图例要展示的信息
plt.title("xxx") # 添加标题
plt.xlabel("X") # 添加x轴
plt.ylabel("Y") # 添加y轴
plt.legend(loc="upper right") # 允许展示图例，且用loc参数配置位置，这里放在右上角

np.linspace(0, 2*np.pi, 100) Split 100 numbers from 0-2pi, arithmetic sequence

• The problem of Chinese display
  When using matplotlib to draw pictures, Chinese is not supported by default. Requires global configuration.

plt.rcParams['font.sans-serif'] = 'SimHei' # 用来正常显示中文标签，simhei：黑体字
# 更改了字体会导致不显示负号，所以要
plt.rcParams['axes.unicode_minus'] = False # 解决符号"-"显示为方块的问题

5.6 Drawing multiple graphs

The figure is the canvas, and there are many axes (subgraphs) below it.
The subplot() function contains 3 parameters, the first indicates the number of rows, the second indicates the number of columns, and the third indicates the serial number of the subplot. For example, subplot(2,2,1) means to draw a subgraph with 2 rows and 2 columns, and draw on the first subgraph.

plt.figure(figsize=(12, 8)) # 通过调整画布来调整整体大小
# 分别获取子画布绘图
ax1 = plt.subplot(221)
ax1.plot(x,y)
ax2 = plt.subplot(222)
ax2.plot(x,np.cos(x))
ax3 = plt.subplot(223)
ax3.plot(x,np.cos(x))
ax4 = plt.subplot(224)
ax4.plot(x,np.cos(x))

6. Summary of pandas files

• os module

import os
os.listdir('./') # 获取当前目录下的子目录和子文件
for root, dirs, filenames in os.walk('./)': # 迭代进入子目录，获取子文件
	...
os.path.join(root, filename) # 拼接路径

• pd.concat()

pd.concat(files) # 将多个dataframe组成的李彪进行级联

Seven, business data analysis thinking

7.1 Business Indicators

Expand from the "turnover rate" indicator.

p65-p72, temporarily omitted

7.2 Commonly used analytical methods

7.3 Logic tree analysis

7.4 Multi-dimensional dismantling analysis

7.5 Comparative Analysis

7.6 Attribution analysis (hypothesis testing method)

RFM analysis model

Funnel Analysis Model (AARRR Analysis Model)

DuPont Analysis (understand)

8. Statistics

slightly

9. Business Intelligence Power BI

PowerBI official website: https://powerbi.microsoft.com/zh-cn/

9.1 Introduction to Power BI

Pain points of traditional companies:
1. There are no dedicated analysts, but there are domain experts, but they are stuck in the use of tools. 2.
The amount of data is very complex, the tables are scattered, and a lot of work is spent on the concentration of tables. A lot of time is wasted in the idea-making process of reporting.
3. Lack of ETL process: source-integration-data governance-loading-big data architecture

There are many systems in the food delivery company, such as customer order system, logistics management system, merchant system, customer service and after-sales system, and these systems are independently developed. Then consider accessing the data center.
Data analysis platform: read-only. Such as domestic: FineReport, Sensors, Zhuge IO. United States: Tableau (started as a map), PowerBI (Microsoft).
APP: User operation: Buried point data.

To obtain more visual objects in PowerBI, you need an account, which is divided into ordinary accounts
1. Enterprise email or edu email, you can apply for free, and you can buy it from Taobao, within 10 yuan.
2. Pro account: Not only can you get more visual objects, you can publish report documents, rights management can be shared, 99% permanent. Personally do not recommend buying.

9.2 PowerBI practical operation 1_ Overall report creation

9.2.1 Function Area Introduction

• data source find data
  
• data cleaning
  
• visual object
  
• Various perspective
  
  processes: data source - data cleaning power query - build indicators to create new metrics power pivot - visualization power view - report completion The
  
  three on the left are (from top to bottom): report, data, table and between tables relational (data modeling)

PowerBI can be used to: get data, select other, select Web connection, and read the table information in the web page. It is suitable for reading some data websites with a lot of content, but the website is relatively old.
You can go to the PowerBI documentation: https://learn.microsoft.com/zh-cn/power-bi/. See how some functions are used.

9.2.2 Read folders and merge files

Get the data, load the folder, and then in the Power Query editor, find the contents of the loaded folder, delete other columns (such as creation time, etc.), keep only the Content column, and then select the merged file (note the merged file, be sure to pay attention The title corresponds to the same)

9.2.3 Load MySQL

The mysql-connector-net-...msi file must be installed in the mysql folder so that PowerBI can connect to Mysql.

9.2.4 Excel's powerquery editor

Excel can also enter the powerquery editor. (So ​​the following explanations are all on the powerquery editor of excel)
In the "Data Tab", select "From Table/Range".