Data analysis with R

Data analysis with R

Vol_0: Digital characteristics and correlation analysis of data

Import Data

Import text table data

Year  Nationwide  Rural  Urban
1978  184         138    405
1979  207         158    434
1980  236         178    496
1981  262         199    562
1982  284         221    576
1983  311         246    603
1984  354         283    662
1985  437         347    802
...

R code:

data <- read.table("./data.txt", header=TRUE)
data

result:

Import CSV data

 序号,省市区,11月,1～11月
 1,北京,35.22,499.8
 2,天津,10.41,161.37
 3,河北,17.22,273.29
 4,山西,10.7,134.79
 5,内蒙古,10.29,90.92
 ...

R code:

data <- read.csv("./data.csv")

Note: The header title in this data contains Chinese characters and special symbols, which will be automatically processed by R into:

> cat(names(data))
X.序号 省市区 X11月 X1.11月

You can change it manually:

data <- data[-1]  # remove "序号" col
names(data) <- c("Province", "X1", "X2")
data

result:

Note: One of the two imported data will be randomly used as an example later.

attach

In order to facilitate calling the data of each column in data.frame, we can:

attach(data)

Then you can directly refer to a column of data with the column name, for example:

print(X1)

It is no longer necessary to fetch the index through data:

print(data[2])

After use, remember to detach:

detach(data)

Mean, variance, standard deviation, coefficient of variation, skewness, kurtosis

Univariate

data: one variable, one "column" of data

x <- c(1, 2, 3, 4, 5)

mean:

$$\overline{x}=\frac{1}{n}\sum _{i=1}^n{x}_i$$

mean(x)

方差：
$$s^2=\frac{1}{n-1}\sum _{i=1}^n(x_i-\overline{x}{)}^2$$

var(x)

标准差：
$$s=\sqrt{s^2}=\sqrt{\frac{1}{n-1}\sum _{i=1}^n(x_i-\overline{x}{)}^2}$$

sd(x)

Coefficient of variation:
$$CV=\frac{s}{\overline{x}}$$

cv <- function(x) sd(x)/mean(x)

cv(x)

Note: The book is a percentage of $CV=100\times \frac{s}{\overline{x}}(\%)$.

偏度：
$$g_1=\frac{1}{(n-1)(n-2)}\frac{1}{s^3}\sum _{i=1}^n(x_i-\overline{x}{)}^3$$

g1 <- function(x) {
    n <- length(x)
    A <- n / ((n-1) * (n-2))
    B <- 1 / sd(x)^3
    S <- sum((x - mean(x))^3)
    A * B * S
}

g1(x)

峰度：
$$\begin{gathered} g_2=\frac{n(n+1)}{(n-1)(n-2)(n-3)}\frac{1}{s^4}\sum _{i=1}^n(x_i-\overline{x}{)}^4 \\ -\frac{3(n-1{)}^2}{(n-2)(n-3)} \end{gathered}$$

g2 <- function(x) {  # 峰度
    n <- length(x)
    A <- (n * (n+1)) / ((n-1) * (n-2) * (n-3))
    B <- 1 / sd(x)^4
    S <- sum((x - mean(x))^4)
    C <- (3 * (n-1)^2) / ((n-2) * (n-3))
    A * B * S - C
}

g2(x)

Act on data.frame

The data we import is all data.frame, and a column can be taken out separately, just like the above x:

x <- data[[2]]  # 取出 data 的第二列数据

mean(x)

But it is annoying to call each column once, so here is another way to apply mean or other functions to each column of data.frame at once (here we use the data imported from the first table above as an example):

apply(data[-1], MARGIN=2, FUN=mean)

result:

illustrate:

• Here the first column of data is character, and the mean is meaningless: use to data[-N]remove the Nth column (R 1starts indexing from )
• The second parameter MARGIN=2is to process column by column
• The third FUN is the function to be applied, here is the mean value. The variance is the same, just change this parameter to FUN=varsomething.

all in one

For convenience, we can encapsulate the process of calculating these things together:

describes <- function(df) {
    # TODO(CDFMLR): 优化重复计算
    cv <- function(x) sd(x)/mean(x)  # 变异系数

    g1 <- function(x) {  # 偏度
        n <- length(x)
        A <- n / ((n-1) * (n-2))
        B <- 1 / sd(x)^3
        S <- sum((x - mean(x))^3)
        A * B * S
    }

    g2 <- function(x) {  # 峰度
        n <- length(x)
        A <- (n * (n+1)) / ((n-1) * (n-2) * (n-3))
        B <- 1 / sd(x)^4
        S <- sum((x - mean(x))^4)
        C <- (3 * (n-1)^2) / ((n-2) * (n-3))
        A * B * S - C
    }
    
    itm <- matrix(c("均值", "方差", "标准差", "变异系数", "偏度", "峰度"), 6, 1)
    res <- apply(df, 2, 
                 function(x) c(mean(x), var(x), sd(x), cv(x), g1(x), g2(x)))
    
    cbind(itm, res)
}

The incoming parameter is a data.frame, and this function will calculate the mean, variance, etc. of each column (for example, for the csv data imported earlier):

describes(data[-1])

result:

It comes out all at once, which is very convenient.

Changeling

Of course, these operations are implemented by third-party packages, such as this one:

• psych

Install this package (written directly in R):

install.packages("psych")

Guide package:

library(psych)

Then you can use the contents of this bag.

This package provides kurtosis and skewness, the hardest to write before:

# g1、g2 是用 type=2: see help(skew)
g1 <- function(x) skew(x, type=2)
g2 <- function(x) kurtosi(x, type=2)  # help(kurtosi)

This package also provides a describefunction to find most of the previous values ​​at once (similar to what we wrote by hand describes):

describe(data[-1], type=2)

result:

Median, upper and lower quartiles, quartile range

You can first find five numbers : minimum value, lower quartile, median, upper quartile, maximum

fn <- apply(data[-1], 2, fivenum)
fn

Quartile range:

R1 <- function(Q3, Q1) Q3 - Q1

R1(Q3=fn[4,], Q1=fn[2,])

[Math Time]

p-quantile:

$$M_p=\{\begin{array}{ll}{x}_{([np]+1)},& np\text{is not an integer}\\ \frac{1}{2}(x_{(np)}+x_{(np+1)}),& np\text{is}\end{array}$$
Upper and lower quartiles:
$$Q_3=M_{0.75},Q_1=M_{0.25}$$
Interquartile range:
$$R_1=Q_3-Q_1$$

Note: R is used to calculate quantiles quantile, see for details help(quantile).

With the upper and lower quartiles and the quartile range, you can find an abnormal data:

Definition: lower and upper cutoff points:

$$Q_1-1.5R_1,Q_3+1.5R_1$$

Data greater than the "upper cutoff" and smaller than the "lower stage" are considered outliers :

abnormal <- function(x) {
    fn <- fivenum(x);
    Q1 <- fn[2];  Q3 <- fn[4];
    
    R1 <- Q3 - Q1
    
    QD <- Q1 - 1.5 * R1
    QU <- Q3 + 1.5 * R1
    
    x[(x < QD) | (x > QU)]
}

apply(data[-1], 2, abnormal)
# 若结果为空则没有异常值

Data Distribution Chart

stem and leaf diagram

stem(Nationwide)

histogram

The easiest is to use it directly hist(x), but we can make it look better.

Package:

histogram <- function(x, xname="x") {
    hist(x, prob=TRUE, main=paste("Histogram of" , xname))
    lines(density(x))
    rug(x) # show the actual data points
}

transfer:

histogram(X1, "X1")

result:

Empirical Distribution Function Plot

Package:

plot_ecdf <- function(x, xname="x") {
    
    
    plot(ecdf(x), do.points=FALSE, verticals=TRUE, main=paste("ecdf(" , xname, ")"))
    
    xs <- seq(min(x), max(x), 1/sqrt(length(x)))
    lines(xs, pnorm(xs, mean=mean(x), sd=sd(x)), lty=3, col="red")
}

Note xshere I choose to use $\frac{1}{\sqrt{n}}$The density, this value is more suitable for my data (the drawing is not too thin but not too dense), this can be changed at will.

transfer:

plot_ecdf(X1, "X1")

result:

Normal QQ plot

qqnorm(X1)

Pearson and Spearman correlation coefficient

Pearson correlation coefficient

2D population: $(X,Y{)}^T$

观测数据：$(x_1,y_1{)}^T,(x_2,y_2{)}^T,\cdots ,(x_n,y_n{)}^T$

记：$\overline{x}=\frac{1}{n}{\sum }_{i=1}^n{x}_i,\overline{y}=\frac{1}{n}{\sum }_{i=1}^n{y}_i$

则$X,$The variance of the observed data of$Y$
$$s_{xx}=\frac{1}{n-1}\sum _{i=1}^n(x_i-\overline{x}{)}^2{s}_{yy}=\frac{1}{n-1}\sum _{i=1}^n(y_i-\overline{y}{)}^2$$
$X,$The covariance of the observed data of$Y$
$$s_{xy}=\frac{1}{n-1}\sum _{i=1}^n(x_i-\overline{x}{)}^2(y_i-\overline{y}{)}^2$$
(Note: covariance matrix$S=\left[\begin{array}{cc}{s}_{xx}& s_{xy}\\ s_{yx}& s_{yy}\end{array}\right]$，其中 $s_{yx}=s_{xy}$）

Pearson correlation coefficient:
$$r_{xy}=\frac{s_{xy}}{\sqrt{s_{xx}}\sqrt{s_{yy}}}$$

This value $|r_{xy}|\le 1$ , to measure the degree of linear correlation between X and Y:

• $r_{xy}\to 1$ positive correlation
• $r_{xy}\to 0$ is not linearly correlated
• $r_{xy}\to -1$ negative correlation

Use R to calculate the correlation coefficient, and use cor(x, y, method="pearson")this function to directly find $r_{xy}$value. You can also use the following function, which will output more information:

cor.test(X1, X2, method="pearson")

Output: ( $\text{cor}=r_{xy}$)

[Math Time] Hypothesis testing about the above output:

Let the two-dimensional population $(X,Y{)}^{}$The distribution function of${}^T$$F(x,y)$

Overall correlation coefficient $r_{{}_{XY}}=\frac{Cov(X,Y}{\sqrt{Var\left(X\right)}\sqrt{Var\left(Y\right)}}$

$When\; n$ is sufficiently large, there is$r_{{}_{XY}}\approx r_{xy}$

Now the question is:

• rxy r_{xy} can always be obtained for any observation data$r_{xy}$, and it is generally not 0
• And if the overall $X$、$Y$ is uncorrelated ($r_{XY}=0$ ): At this time, use$r_{xy}$to measure $X$、$The\; relevance\; of\; Y$ is meaningless.

So do a hypothesis test:
$$H_0:r_{{}_{XY}}=0\leftrightarrow H_1:r_{{}_{XY}}\ne 0$$
If the population is two-dimensional normal, then$H_0$When true, the statistic
$$t=\frac{T_{xy}\sqrt{n-2}}{\sqrt{1-r_{xy}^2}}\sim t(n-2)$$ tt
calculated from the observed data$The\; t$ value is recorded as$t_0$then test $p$ 值：
$$p=P_{H_0}(|t|>|t_0|)=P(|t(n-2)|\ge |t_0|)$$
Given a significant level$\alpha$ ，$p<\alpha$ reject$H_0$, consider $X,Y$ is related, you can use$r_{xy}$Measure relevance.

Spearman correlation coefficient

Spearman is the rank correlation coefficient .

Sample rank: take observations $x_1,x_2,\cdots ,x_n$Sort from small to large, $x_i$Rank $R_i$Just how much.

e.g.
$$\begin{array}{rcccc}{x}_i:& 7& -3& -1& 5\\ R_i:& 4& 1& 2& 3\end{array}$$

remember:

• $x_1,x_2,\cdots ,x_n$The ranks are: $R_1,R_2,\cdots ,R_n$
• $\overline{R}=\frac{1}{n}{\sum }_{i=1}^n{R}_i=\frac{1}{n}{\sum }_{i=1}^{n}i=\frac{n+1}{2}$
• $y_1,y_2,\cdots ,y_n$The ranks are: $S_1,S_2,\cdots ,S_n$
• $\overline{S}=\frac{1}{n}{\sum }_{i=1}^n{S}_i=\frac{n+1}{2}$

则定义 Spearman 相关系数：
$$\begin{array}{lcc}{q}_{xy}& =& \frac{{\sum }_{i=1}^n(R_i-\overline{R})(S_i-\overline{S})}{\sqrt{{\sum }_{i=1}^n(R_i-\overline{R}{)}^2}\sqrt{{\sum }_{i=1}^n(S_i-\overline{S}{)}^2}}\end{array}$$
Use R to calculate ( rho = qxy \textrm{rho}=q_{xy} in the output$\text{rho}=q_{xy}$)：

cor.test(X1, X2, method="spearman")

Still the same, with a hypothesis test:
$$H_0:r_{{}_{XY}}=0\leftrightarrow H_1:r_{{}_{XY}}\ne 0$$

---

【EOF】

That's all for now. I'm busy recently, and if I have time later, I may write regression analysis, variance analysis... These are a complete set.

CDFMLR 2021.06.07

---

[PS 2021.07.15] I really don't have time to write, and I feel like it's unfinished again.

( If anyone wants to see the follow-up, you can chime in )

In fact, if necessary, the various codes of the entire data analysis are basically written in github.com/cdfmlr/daex , you can refer to it.