【Statistical Analysis】(task1) Hypothesis Test 1: Methodology and Unary Numerical Test

summarize

• Mathematically we cannot prove a hypothesis with a special sample, but we can use it to disprove (reject) a proposition. Therefore, hypothesis testing is essentially exploring how to reject the null hypothesis $H_{}$to accept the alternative hypothesis $H_{}$.
• The basic steps of hypothesis testing based on p-value (p-value, which is the minimum significance level given to reject the null hypothesis under the determined sample observation value):
  • Determine the alternative hypothesis $H_{}$, the sign of the alternative hypothesis determines which cumulative probability we use.
  • Clarify the formula of Test statistics. Different hypothesis tests have their own test statistics. Check the data to find it!
  • Specify the distribution to which the test statistic follows so that we can calculate the cumulative probability.
  • According to the alternative hypothesis $H_{}$Calculate p-values ​​with Test statistics.
  • Associate p-value with significance level $Alpha$ comparison, if$p>\alpha$ , the null hypothesis cannot be rejected; if$p<\alpha$ , the null hypothesis can be rejected.
• Test common hypotheses, according todata type, data characteristicsClassification, in practical applications, you can directly select the corresponding hypothesis test according to the characteristics of the data to be analyzed and the analysis task:
  • For some of the more common and important hypothesis tests (such as various t tests), briefly learn their principles;
  • For tests that are uncommon and difficult to explain (such as normality tests), you can use them.
• Different test statistics have different forms and different distributions, but the idea of ​​hypothesis testing has something in common:Construct the test statistic - output the quantile of the corresponding distribution - calculate the critical value (rejection domain) - make a judgment.
  

Zero, basic knowledge review

0.1 Classification of data

• Statistics: A variable used in statistical theory to analyze and test data. Inferring the properties of the whole from the sample, we usually infer it through statistics, such as inferring the lifespan of the bulbs produced by the factory by calculating the average lifespan of 100 bulbs. Common statistics are: sample mean, sample variance, sample moment, sample K-order center distance, sample skewness, sample kurtosis, etc.; there are also statistics constructed for statistical analysis needs, such as $z$ statistic,$t$ statistic,$h^2$ Statistics,$F$ statistic etc.
• Classification of statistical data (common practice):
  • Categorical data: Calculate frequency or frequency, mode and disparity ratios for each group, perform contingency table analysis and $h^2$ Inspection, etc.;
  • Ordinal data: calculate median and interquartile range, calculate rank correlation coefficient, etc.;
  • Numerical data (divided into discrete variables and continuous variables, the former such as the number of products, the latter such as temperature): There are more statistical methods for analysis, such as calculating various statistics, parameter estimation and testing, etc.

0.2 Graphical display of data

• Graphical display of the data:
  • Quality data: that is, categorical data (frequency distribution chart, divided into contingency table, cross table; pie chart, histogram, Pareto chart, ring chart, etc.) and sequential data (using cumulative frequency to draw line chart or frequency picture).
  • Numeric data:
    • If it is grouping, for example, after grouping at equal intervals, calculate the frequency within the group, and then use the histogram or bar chart;
    • If it is ungrouped, such as stem and leaf plots or boxplots, boxplots are less familiar. It is possible to compare the distribution characteristics of multiple sets of data, one of which is a set of 5 features: the maximum value, the minimum value, the median, and the two quartiles of the data. The following are boxplots of different distributions;
    • In the case of time series data, a line graph.
    • If it is multivariate data: scatter plot, bubble chart, radar chart, etc.

[Box Plot Xiao Lizi] Plot the 8 courses of 11 students into a box plot (picture from Jia Junping - Statistics 6th Edition P59):

0.3 Generalized measures of data

The basics of hypothesis testing

Reason for learning: When I study courses such as probability theory and statistical analysis, multivariate statistical analysis, etc., I always feel that I have learned a lot of hypothesis testing, but I lack a systematic generalization, so I don’t know what kind of data and what kind of data. requirements to select the corresponding hypothesis test.

(1) Several characteristics of hypothesis testing:

• Hypothesis testing runs through all aspects of statistical analysis. In mathematical modeling, we can not only conduct exploratory information mining on the data through hypothesis testing, but also give us the basis for choosing a model;
• After the modeling is completed, the validity of the model can be verified through a specific hypothesis test. It is necessary to select the corresponding hypothesis test according to the characteristics of the data and the needs of the task.
• Unlike regression analysis and other systematic and structured statistical analysis tasks, each field has its own hypothesis tests that can be used. For example, there are significance tests of model coefficients in regression analysis, and unit root tests, white noise tests, and so on in time series analysis.

(2) Two major categories of hypothesis testing (hypothesis testing based on statistical models and hypothesis testing not based on statistical models):

• The former is based on a known statistical model and "provides services" for the use of the model. The coefficient significance test of the linear regression model we mentioned earlier is a typical example. For this hypothesis test, the corresponding statistical model can be learned first. Learn more about hypothesis testing.
• The latter (the focus of this study) is to "start directly from the data" and directly test some properties of the data, such as normality test, two-sample t test, variance analysis, and so on.

1.1 The principle of hypothesis testing

(1) The essence of hypothesis testing

[Chestnut] Known: The average grade of the first class $\bar{x}=108.2$ , sample standard deviation$s=4$ , the number of people$n=25$ , and according to past experience, grades' test scores show a normal distribution. So, can a prefect consider a grade point average of at least 110?

Simple analysis:
Population: Mathematics scores of the entire grade
One sample: Mathematics scores of a class
Known: The sample mean $\bar{x}=108.2$
The needs of prefects: Can we inferwhetheroverallmeanreaches 110 through the sample of a class?

• We need to answer "yes" or "no" to the proposition that the overall grade point average is not less than 110. Such questions are called hypothesis testing questions.
• The process of mathematically testing and answering hypothesis testing questions is called hypothesis testing.

Summary: This kind of "yes or no" test and answer to a proposition describing the overall nature based on sample information and known information is the essence of hypothesis testing. That is, hypothesis testing does not verify the nature of the sample itself, but the nature of the population in which the sample is located .

Hypothesis testing can be roughly divided into two types: parametric hypothesis testing and nonparametric hypothesis testing.

• If the hypothesis is about a parameter or set of parameters of the population, the hypothesis test is a parametric hypothesis test. The hypothesis of Example.1 is about the population mean, and the mean is a parameter, so this is a parametric hypothesis test;
• If the hypothesis cannot be represented by a set of parameters, the hypothesis test is a nonparametric hypothesis test, a typical one is the normality test.

1.2 Derivation of Hypothesis Testing

(1) The establishment of the hypothesis

Chestnut 1: The math score of a student in a certain grade $X$ follows a normal distribution$X\sim N\left(m,p^2\right)$, take a class of students as a sample, the known sample mean$\bar{x}=108.2$ , sample standard deviation$s=4$ , the number of people in a class is$n=25$ , can it be considered that the population mean$m>110$ ?

The proposition "The population mean $m>Whether\; 110$ ” is correct involves the following two assumptions:
$$H_{}:m⩽110\leftrightarrow H_{}:m>110$$
$H_{}$is called the null hypothesis, $H_{}$Known as the alternative hypothesis, the two hypotheses must be mutually exclusive, because only then, reject the hypothesis $H_{}$is equivalent to accepting the hypothesis $H_{}$.

• The discussion of proposition establishment is transformed into rejecting the null hypothesis $H_{}$Instead of focusing on accepting the alternative hypothesis, this is because mathematically we cannot prove a hypothesis with a special sample, but we can use it to disprove (reject) a proposition. therefore,Hypothesis testing is essentially exploring how to reject the null hypothesis $H_{}$to accept the alternative hypothesis $H_{}$.
• So in Example.1, it is difficult for us to directly prove the proposition $m>110$ is true, but it can be proved by assuming$m⩽110$ error indirectly verify$m>110$ Establishment. In actual hypothesis testing,We usually take the proposition we want to test as the alternative hypothesis $H_{}$, by testing the null hypothesis $H_{}$Whether it is rejected to judge whether to accept $H_{}$.

(2) Three types of one-parameter tests and precautions for the null hypothesis

Although we usually take the proposition we want to test as the alternative hypothesis $H_{}$, but this is not a "criteria" - there are some hypothesis tests where the setting of the original/alternative hypothesis is fixed, such as when testing whether the population in which a sample is located obeys a specific distribution (such as a normality test ), we usually set two assumptions as follows
$$H_{}:\text{The population in which the sample is located obeys a certain distribution}\leftrightarrow H_{}:\text{The population where the sample is located does not obey a certain distribution}$$
. Another example,In the most common one-parameter tests, "=" will only appear in the null hypothesis $H_{}$, but not in the alternative hypothesis $H_{}$middle, i.e. we won't do things like
$$H_{}:m\ne 110\leftrightarrow H_{}:m=110$$
Assumptions.

The three most common one-parameter test problems, take the mean test as an example:
$$\begin{gathered} H_{}:m⩽m_{}\leftrightarrow H_{}:m>m_{} \\ {H}_{}:m⩾m_{}\leftrightarrow H_{}:m<m_{} \\ {H}_{}:m=m_{}\leftrightarrow H_{}:m\ne m_{} \end{gathered}$$
Among them, the first two tests are called unilateral test, and the third test is bilateral test. There is another, more common way of expressing the null hypothesis of the above three problems:
$$\begin{gathered} H_{}:m=m_{}\leftrightarrow H_{}:m>m_{} \\ {H}_{}:m=m_{}\leftrightarrow H_{}:m<m_{} \\ {H}_{}:m=m_{}\leftrightarrow H_{}:m\ne m_{} \end{gathered}$$

Q: Why are the null hypotheses all set to the "=" sign?
Answer: Let's take the first test problem as an example: if we accept $H_{}$, meaning $\mu$ is significantly larger than$m_{}$while rejection is equal to $m_{}$the assumption, and if even equal to $m_{}$The assumptions are unacceptable, less than $m_{}$Not to mention. therefore,Although this notation is not mutually exclusive, its result is equivalent to the previous notation.

In the subsequent parametric tests, the symbol of the null hypothesis will be uniformly set to "=", and only the alternative hypothesis $H_{}$That's it.

(3) Critical value, rejection domain, significance level

In Example.1, becauseSample mean $\bar{x}$ is the population meanμ \muunbiased estimate of$\mu$, then if the null hypothesis is rejected, that is, $m>110$ , then$\bar{x}$ has a high probability of being greater than 110, which means that ifactual samplecalculated$\bar{x}$ is much larger than 110, then the null hypothesisis very likely to fail. To give a criterion for rejecting the null hypothesis,we set a critical value$C$ , if the actual sample calculated$\bar{x}$ Satisfy$\bar{x}-110>C$ , we reject the null hypothesis. where,$\bar{x}-110>C$ is also called deny field:

$$\left\{\bar{x}:\bar{x}>110+C\right\}$$
Once the result of the sample calculation falls into the rejection region, we reject the null hypothesis; otherwise, the null hypothesis cannot be rejected. The rejection domains of different hypothesis tests are different, but the core logic is exactly the same.

1) Determine the critical value C with probability

The next question is, how do we determine the critical value $What\; about\; C$ ? Determine with probability.

Due to the randomness of sampling, there is always a chance of making mistakes in judging the nature of the population based on the information of the sample—that is, whether or not we reject the null hypothesis $H_{}$, we all have a probability of making one of the following two types of errors:

• Type 1 Error: Null Hypothesis $H_{}$is true, but the data falls into the rejection domain (hence the rejection $H_{}$judgment). The probability of making a Type 1 error is called the rejection probability $a$
• Type II Error: Null Hypothesis $H_{}$is false, but the data does not fall into the reject field (hence the accept $H_{}$judgment). The probability of making a Type II error is called a pseudoprobability $b$

The two error probabilities are opposed to each other and "contradict" each other. Given the sample size, if we want to reduce the probability of one type of error by adjusting the hypothesis testing rules, it will inevitably lead to an increase in the probability of another type of error. This means that we can't control them and keep them at a low level at the same time, based on this, we can only make compromises -A common practice is to limit only the probability of making a Type 1 error $a$.

2) The problem of determining the critical value

Type 1 error: The null hypothesis is true, but the null hypothesis is rejected.
Type II error: The null hypothesis is false, but the null hypothesis is not rejected.

Back to the problem of determining the critical value. When deciding on thresholds, we want to ensure thatProbability of Type 1 Error $\alpha$ needs to be at a given, small level (usually$a=0.05/0.1$ ), at this time $\alpha$ is also known as the significance level.

Determine the critical value $The\; criterion\; for\; C$ is: null hypothesis$H_{}$is true, but the probability that the data falls into the rejection region should be exactly the given $a$ . In Example.1, this probability can be written as:
$$P_{H_{}istrue}\left(\bar{x}-m_{}>C\right)=P\left(\bar{x}-110>C\right)=a$$

(4) Handling Probabilities with Distributions - Constructing Test Statistics

Process $P\left(\bar{x}-110>C\right)$ , this form is like the "quantile" of the distribution that we contacted in general learning, the next step is to construct it into the quantile form of a distribution, so that the critical value$C.$ _

Note: 110 is actually $\mu$ is in the null hypothesis$H_{}$correct value, and $AND\left(\bar{x}\right)=\mu$ , so the probability is actually:
$$P\left(\bar{x}-AND\left(\bar{x}\right)>C\right),AND\left(\bar{x}\right)=m_{}=110$$
Since$\stackrel{ˉSubject}{x}$ to normal distribution, and ssin Example.1$s$ is known, we can construct the t-statistic:
$$P\left(\bar{x}-AND\left(\bar{x}\right)>C\right)=P\left(\frac{\bar{x}-AND(\bar{x}}{s}>\frac{C}{s}\right)=a,\frac{\bar{x}-AND(\bar{x}}{s}\sim t_{n-1}$$
This means, $\frac{C}{s}$Exactly $t_{n-1}\left(1-\alpha \right)$ quantiles, which are known for a given distribution, so$C$ is solved
$$C=s\cdot t_{n-1}\left(1-\alpha \right)$$
We substitute it into the above formula to have
$$P\left(\frac{\bar{x}-AND(\bar{x}}{s}>\frac{C}{s}\right)=P\left(\frac{\bar{x}-m_{}}{s}>\frac{s\cdot t_{n-1}(1-a}{s}\right)=P\left(\bar{x}>m_{}+s\cdot t_{n-1}\left(1-a\right)\right)=\alpha$$
is: as long as$\bar{x}>110+s\cdot t_{n-1}\left(1-\alpha \right)$ , we can thenat the significance levelα \alphaThe null hypothesis is rejected under$\alpha$ .

• In the above process, we useknownThe statistic constructs a statistic that obeys a certain distribution to help us calculate the probability, and the constructed statistic is the test statistic .
• Different test statistics have different forms and different distributions, but the idea of ​​hypothesis testing has something in common:Construct the test statistic - output the quantile of the corresponding distribution - calculate the critical value (rejection domain) - make a judgment.

In the above example, the test statistic is
$$t=\frac{\bar{x}-m_{}}{s}$$
The quantile corresponding to the t distribution is
$$t_{n-1}\left(1-\alpha \right)$$
The rejection domain is
$$\bar{x}>110+s\cdot t_{n-1}\left(1-\alpha \right)$$
python manually implements the above hypothesis testing process:

## 加载包
import numpy as np
import pandas as pd
from scipy.stats import t

n=25 
x_bar=108.2
s=4
mu=110

# 计算检验统计量
tvalue=(x_bar-mu)/s
print('t值为：{}'.format(tvalue))

# 输出分位点
'''
ppf:单侧左分位点
isf:单侧右分位点
interval:双侧分位点
'''
T_isf=t.isf(0.05,n-1) #由于备择假设是大于号，因此应当选用单侧右分位点，0.05为显著性水平a，n-1为自由度
# 如果备择假设是小于号，则应选用单侧左分位点ppf，里面的参数设置不变，依次为显著性水平a与分布自由度

print('分位点为：{}'.format(T_isf))
# 拒绝域
Deny_domain=110+s*T_isf
print('拒绝域的临界点为：{}'.format(Deny_domain))

# 判断
print('样本均值是否位于拒绝域：{}'.format(x_bar>Deny_domain))
print('因此，不能拒绝原假设，不能认为总体均值大于110.')

#t值为：-1.7999999999999972
#分位点为：1.7108820799094282
#拒绝域的临界点为：116.84352831963771
#样本均值是否位于拒绝域：False
#因此，不能拒绝原假设，不能认为总体均值大于110.

Of course, the rejection region can also be directly represented by the test statistic and the corresponding distribution quantile
$$t>t_{n-1}\left(1-\alpha \right)$$
This is more convenient and more general, because no more time is spent calculating the critical value$C.$ _ The rejection domains of the three hypothesis tests (taking the normal distribution as an example) are shown in the following figure

It can be seen that the three hypothetical quantiles correspond to the right quantile, the left quantile, and the bilateral quantile, respectively. In practical applications, we can use the alternative hypothesis $H_{}$symbol, select the corresponding quantile to construct a rejection domain.

# 直接用检验统计量与分布分位点判断
print('检验统计量是否位于拒绝域：{}'.format(tvalue>T_isf))
# 检验统计量是否位于拒绝域：False

Assuming a two-sided test using the data in Example.1
$$H_{}:m=110\leftrightarrow H_{}:m\ne 110$$
Then the rejection domain is
$$|t|>\left|t_{n-1}\left(1-\frac{}{2}\right)\right|$$

# 进行双边检验
## 计算双侧分位点
T_int=t.interval(1-0.05,n-1) # 对于双侧检验（双侧分位点），分位点参数应该输入1-a，这里是1-0.05=0.95
print('检验统计量t的绝对值：{}'.format(np.abs(tvalue)))
print('双侧分位点：{}'.format(T_int))
print('显然，检验统计量不在拒绝域内，因此无法拒绝原假设')

#检验统计量t的绝对值：1.7999999999999972
#双侧分位点：(-2.0638985616280205, 2.0638985616280205)
#显然，检验统计量不在拒绝域内，因此无法拒绝原假设

1.3 Basic steps of hypothesis testing - based on p-values

• A disadvantage of using the rejection domain method for hypothesis testing is that the quantile value and the significance level $\alpha$ is relevant. If we want to test at different significance levels, we need to calculate different quantiles for comparison, which is very tedious.
• p-value: As long as the sample information and hypothesis are determined, you can rely on a constant indicator to judge whether to reject the null hypothesis. The p-value is the minimum significance level that can reject the null hypothesis under the determined sample observation value, and the p-value is only related to the sample observation value and the hypothesis test we do. The smaller the p-value, the more rejecting the null hypothesis.

The smaller the p value is, the more the null hypothesis can be rejected. For example, if the p value is 0.001, which is smaller than the confidence level of 0.01, we think that we can also reject the null hypothesis at the confidence level of 0.01; and if the p value is 0.025, the The confidence level of 0.01 is larger, but less than 0.05, then we think that we can reject the null hypothesis at the confidence level of 0.05, but not at the confidence level of 0.01.

The p-value is in the form of the alternative hypothesis we make $H_{}$related:

• If $H_{}$The notation is $\ne$，则：$pvalue=P\left(|X|>|Teststatistics|\right)$
• If $H_{}$的符号为>，则：$pvalue=P\left(X>Teststatistics\right)$
• If $H_{}$的符号为<，则：$pvalue=P\left(X<Teststatistics\right)$

in:

• X is a variable that obeys a specific distribution;
• Test statistics are the test statistics mentioned earlier.
• The p-value is essentially a cumulative probability. For the alternative hypothesis with the symbol >, the p-value is the cumulative probability on the right; for the alternative hypothesis with the symbol <, the p-value is the cumulative probability on the left; and for the same In terms of test statistics, the p-value of a two-sided test is twice that of a certain type of one-sided test.

# 利用example.1的数据进行三种假设检验
# 利用p值进行假设检验
'''
sf:右尾累积概率
cdf:左尾累积概率
'''
# 若备择假设为mu>110
pvalue=t.sf(tvalue,n-1) 
print('备择假设为mu>110的p值为：{}'.format(pvalue))

# 若备择假设为mu<110
pvalue=t.cdf(tvalue,n-1)
print('备择假设为mu<110的p值为：{}'.format(pvalue))

# 若备择假设为mu不等于110
pvalue=t.cdf(tvalue,n-1)*2 # 之所以是左尾累积概率的两倍，是因为右尾累积概率大于0.5，而p值不可能大于1。
print('备择假设为mu不等于110的p值为：{}'.format(pvalue))

#备择假设为mu>110的p值为：0.9577775745385242
#备择假设为mu<110的p值为：0.042222425461475775
#备择假设为mu不等于110的p值为：0.08444485092295155

Note: Using p-value for hypothesis testing is more common in practical applications. All packages for hypothesis testing in python will output test statistics and p-values. In subsequent learning, p-values ​​are uniformly used for hypothesis testing.

The python scipy.statsmodule contains many types of hypothesis testing APIs that can be used directly, but compared to SPSS and R, which specialize in statistical analysis, python has relatively few hypothesis testing functions. If a certain hypothesis testing python does not have its corresponding api, you need to manually calculate the p-value. For example, the subsequent Hotelling T2 test on the mean vector.

[Summary] The basic steps of hypothesis testing based on p value:

1. Determine the alternative hypothesis $H_{}$, the sign of the alternative hypothesis determines which cumulative probability we use.
2. Clarify the formula of Test statistics. Different hypothesis tests have their own test statistics. Check the data to find it!
3. Cumulative probability can only be calculated by specifying the distribution to which the test statistic follows.
4. According to the alternative hypothesis $H_{}$Calculate p-values ​​with Test statistics.
5. Associate p-value with significance level $Alpha$ comparison, if$p>\alpha$ , the null hypothesis cannot be rejected; if$p<\alpha$ , the null hypothesis can be rejected.

1.4 Classification of Hypothesis Testing

The common hypothesis tests are classified according to data types and data characteristics. In practical applications, the corresponding hypothesis tests can be selected directly according to the characteristics of the data to be analyzed and the analysis tasks:

• For some of the more common and important hypothesis tests (such as various t tests), briefly learn the principles;
• Learn how to use tests that are uncommon and difficult to explain, such as normality tests.

2. Hypothesis testing for unary numerical data

The content of hypothesis testing of between-group means in unary numerical data , learn how to test the properties of the mean of the population in which the sample data is used, and show how to implement each test in python. content:

1. normality test
2. A test that compares the population mean of a set of data for equality with a fixed value
3. A test to compare the equality between the population means of two sets of data
4. Tests for equality between population means of more than two groups of data

In the test of 2~4, it can be divided into two cases. If the data are roughly normally distributed, then parametric tests - t-tests can be used, which are more sensitive than nonparametric tests, but need to satisfy the normality assumption; if the data are not normally distributed, then some nonparametric tests can be used.

2.1 Normality test

Since parametric tests are more sensitive than nonparametric tests, we should use parametric tests once the data are normally distributed, and it is very necessary to test the data for normality.

Here, we provide three methods to help you judge the normality of data: visual judgment - probability map of normal distribution; Shapiro-Wilk test; D'Agostino's K-squared test

(1) Probability map

In statistics, along with many tools for visually assessing distributions, probability plots are one of them.

# 生成1000个服从正态分布的数据
data_norm = stats.norm.rvs(loc=10, scale=10, size=1000) # rvs(loc,scale,size):生成服从指定分布的随机数，loc：期望；scale：标准差；size：数据个数
# 生成1000个服从卡方分布的数据
data_chi=stats.chi2.rvs(2,3,size=1000)

# 画出两个概率图
fig=plt.figure(figsize=(12,6))
ax1=fig.add_subplot(1,2,1)
plot1=stats.probplot(data_norm,plot=ax1) # 正态数据
ax2=fig.add_subplot(1,2,2)
plot2=stats.probplot(data_chi,plot=ax2) # 卡方分布数据

For a given sample dataset, the probability plot:

• First put the data $Sort\; x$ from small to large, and calculate the sorted dataxxThe distribution quantile corresponding to$x\; ;$
• Then, plot the data points in a two-dimensional graph with the quantiles as the horizontal axis and the ordinal sample values ​​as the vertical axis.
• If the data roughly follow the target distribution, the data points will approximately follow the line $Y=x$ distribution, if the data does not follow the target distribution, we will observe that the data points deviate from the line$Y=x$。

(2) Two normality tests

Probability plots can only roughly tell whether the data is normal or not, but it is not precise. In order to more accurately judge whether the population of the sample is normally distributed, a strict normality test needs to be carried out.

The two assumptions of the normality test are as follows:
$$H_{}:\text{The population in which the sample is located obeys a normal distribution}\leftrightarrow H_{}:\text{There are many types of normality tests that the population of the sample does not obey the normal distribution}$$
. Here are only two of the most commonly used and most powerful hypothesis tests - Shapiro-Wilk test for small samples; D'Agostino's K for large samples. -squared test.

1) Shapiro-Wilk test (small sample normality test)

Shapiro–Wilk test is one of the most effective methods for normality testing. It is a method for testing normality in frequency statistics, and its theoretical principle is relatively complicated.

This method is suitable for normality testing problems with small samples because:This test works best when each sample value is unique, and once there are too many samples, it is inevitable that several sample values ​​will be repeated, which will greatly reduce the effectiveness of this method.

Scope of application of sample size: The sample size should not be less than 8, less than 50 is the best, less than 2000 is better, and it is no longer applicable if it exceeds 5000.

2) D'Agostino's K-squared test (large sample normality test)

D'Agostino's K-squared test mainly quantifies the difference and asymmetry between the data distribution curve and the standard normal distribution curve by calculating the skewness (Skewness) and kurtosis (Kurtosis), and then calculates these values ​​and the expected value of the normal distribution the degree of difference between.

This method is a common and powerful normality test method suitable for large samples. This is because the skewness and kurtosis of the distribution curve are easily affected by the amount of data. The larger the amount of data, the more accurate the calculation of skewness and kurtosis.

Scope of application of sample size: The sample size shall not be less than 4, otherwise the larger the better.

(3) Use multiple methods to judge normality at the same time

In practical applications, due to the complexity of the data, only using one method to judge normality may produce certain errors, so we usually use multiple methods to judge at the same time. If the conclusions drawn by different methods are different, it is necessary to carefully observe the characteristics of the data and find the reasons for the inconsistent results. For example: if the Shapiro-Wilk test is significant (non-normal) and the D'Agostino's K-squared test is not significant (normal), it may be because the sample size is large, or there are duplicate values ​​in the sample, if this is the case , then we should adopt the conclusion of D'Agostino's K-squared test rather than the conclusion of the Shapiro-Wilk test.

[Code Practice] Define a function in python that combines probability map, Shapiro-Wilk test, and D'Agostino's K-squared test.

data_small = stats.norm.rvs(0, 1, size=30) # 小样本正态性数据集
data_large = stats.norm.rvs(0, 1, size=6000) # 大样本正态性数据集

# 定义一个正态性检验函数，它可以输出：
## 正态概率图
## 小样本Shapiro-Wilk检验的p值
## 大样本D'Agostino's K-squared检验的p值

from statsmodels.stats.diagnostic import lilliefors
from typing import List

def check_normality(data: np.ndarray, show_flag: bool=True) -> List[float]:
    """
    输入参数
    ----------
    data : numpy数组或者pandas.Series
    show_flag : 是否显示概率图
    Returns
    -------
    两种检验的p值；概率图
    """

    if show_flag:
        _ = stats.probplot(data, plot=plt)
        plt.show()

    pVals = pd.Series(dtype='float64')
    # D'Agostino's K-squared test
    _, pVals['Omnibus'] = stats.normaltest(data) 

    # Shapiro-Wilk test
    _, pVals['Shapiro-Wilk'] = stats.shapiro(data)

    print(f'数据量为{
      
      len(data)}的数据集正态性假设检验的结果 : ----------------')
    print(pVals)

check_normality(data_small,show_flag=True)
check_normality(data_large,show_flag=False) # 当样本量大于5000，会出现警告

2.2 Mean Test

Carry out the test of the overall mean of the univariate numerical sample. Each mean test will have a corresponding parametric test (t test) and a non-parametric test to choose from.

(1) Test for the hypothesis of a single-group sample mean

Application scenario: test whether the mean of the population in which a sample is located is equal to a reference value , which is the test of the assumption of the mean of a single group of samples.

The test problem of Example.1 is actually this kind of test (although the alternative hypothesis needs to be replaced with $\ne$ ).

Example.2 In Bisheng Middle School, Mr. Chen's class finished an English test. Due to the large number of students in the class, it is difficult to complete the correction and statistics in a short period of time. Teacher Wang also wanted to know whether there is a significant difference between the average score of this class and the target of 137 set by the prefect, so he randomly selected The English grades of the top 10 students:

136,136,134,136,131,133,142,145,137,140

Q: Mr. Wang, can you think that there is no significant difference between the average grade of this class and the target of 137 for the class set by the prefect?
This is a typical test of the hypothesis of a single-group sample mean, comparing whether the overall mean (average of class English scores) represented by this sample (the English scores of 10 students) is equal to the reference value of 137. So for this kind of problem, we have two kinds of tests to use: one-sample t-test and wilcoxon test.

1) One Sample t-test (One Sample t-test)

The t-test requires the population to obey a normal distribution, that is,
$$x\sim N\left(m,p^2\right)$$
In Example.2, this means that the English scores of every student in Mr. Wang's class are required to obey a normal distribution. In a t-test, the standard deviation of the population$\sigma$ does not need to be known in advance, which is different from the z-test in probability theory and mathematical statistics, so the t-test is more widely used in practice.

Basic flow of hypothesis testing using p-values. (as per 1.1.3)

The two assumptions of the one-sample t-test are:
$$H_{}:m=m_{}\leftrightarrow H_{}:m\ne m_{}$$
The corresponding test statistics are:
$$Teststatistics=\frac{\bar{x}-m_{}}{s}$$
The distribution of the test statistics is:
$$Teststatistics\sim t_{n-1}$$
where, $n$ is the sample size. We can calculate the p-value based on the above information.

2) wilcoxon signed rank sum test

If the sample data is not normal, we should use the wilcoxon signed rank sum test. This test is a very classic non-parametric test, and the principle of non-parametric test is introduced:

(1) First of all, what is "rank".
Let $x_{},\cdots ,x_n$For simple random samples from a continuous distribution, we sort them from small to large to obtain ordered samples $x_{(1}⩽\cdots ⩽x_{\left(n\right)}$. Observations $x_i$ordinal rr in ordered samples$r$ is called$x_i$rank. Therefore, the rank is actually the sample value $x_i$The meaning of "smallest" in all samples.

Rank sum test, there must be a "sum of ranks". Let $x_{},\cdots ,x_n$are samples, do absolute value transformation on them, and then record $R_i$Because ∣ xi ∣ | x_i $|x_i|$在$\left(|x_{}|,\cdots ,|x_n|\right)$的秩。记
$$I\left(x_i>0\right)=\{\begin{array}{ll}1,& x_i>0\\ 0& x_i\ne \end{array}$$
则称
$${In}^{+}=\sum _{i=1}^n{R}_{i}I\left(x_i>0\right)$$
is the rank sum statistic.

(2) Wilcoxon signed rank sum test steps.
The two assumptions of the wilcoxon signed rank sum test for one-sample mean comparison are still
$$H_{}:m=m_{}\leftrightarrow H_{}:m\ne m_{}$$
For the samples to be analyzed $x_{},\cdots ,x_n$, let all samples subtract the contrast value $m_{}$，得：$x_{}-m_{},\cdots ,x_n-m_{}$, calculate their rank sum statistic ${In}^{+}$ .

(3) Test statistics can be calculated as
$$Teststatistics=\frac{{In}^{+}-\frac{n(n+1}{4}}{\sqrt{\frac{n(n+1)(2n+1}{twenty\; four}}}$$
Test statistics approximately follow the distribution
$$Teststatistics\to N\left(0,1\right)$$
where,$n$ is the sample size.

(4) Calculation of p-value and alternative hypothesis $H_{}$is related to the sign of , which is the same as the t-test.

This method is best used when the sample size is greater than 25, because then the test statistic is approximately normally distributed. The sample size in Example.2 is 10, which is not strictly suitable.

# 定义一个单组样本均值检验函数，使它可以同时输出t检验与wilcoxon符号秩和检验的p值
def check_mean(data,checkvalue,confidence=0.05,alternative='two-sided'):        
    '''
    输入参数
    ----------
    data : numpy数组或者pandas.Series
    checkvalue : 想要比较的均值
    confidence : 显著性水平
    alternative : 检验类型，这取决于我们备择假设的符号:two-sided为双侧检验、greater为右侧检验、less为左侧检验

    输出
    -------
    在两种检验下的p值
    在显著性水平下是否拒绝原假设
    '''
    pVal=pd.Series(dtype='float64')
    # 正态性数据检验-t检验
    _, pVal['t-test'] = stats.ttest_1samp(data, checkvalue,alternative=alternative)
    print('t-test------------------------')
    if pVal['t-test'] < confidence:
      print(('目标值{0:4.2f}在显著性水平{1:}下不等于样本均值(p={2:5.3f}).'.format(checkvalue,confidence,pVal['t-test'])))
    else:
      print(('目标值{0:4.2f}在显著性水平{1:}下无法拒绝等于样本均值的假设.(p={2:5.3f})'.format(checkvalue,confidence,pVal['t-test'])))

    # 非正态性数据检验-wilcoxon检验
    _, pVal['wilcoxon'] = stats.wilcoxon(data-checkvalue,alternative=alternative)
    print('wilcoxon------------------------')    
    if pVal['wilcoxon'] < confidence:
      print(('目标值{0:4.2f}在显著性水平{1:}下不等于样本均值(p={2:5.3f}).'.format(checkvalue,confidence,pVal['wilcoxon'])))
    else:
      print(('目标值{0:4.2f}在显著性水平{1:}下无法拒绝等于样本均值的假设.(p={2:5.3f})'.format(checkvalue,confidence,pVal['wilcoxon'])))
    
    return pVal

Whether it is a t test or a wilcoxon test, the p value is quite large. Obviously, we cannot reject the null hypothesis. Teacher Wang can think that the class is divided into 137.

(2) The mean equality test of the two groups of samples

1) Independence between groups

Two Sample t-test
Mannwhitneyu rank sum test

2) Paired groups

Paired t-test
Paired wilcoxon signed-rank-sum test

(3) Analysis of variance (test for equality of means among multiple groups of samples)

We have learned the mean test of a single group of samples and two groups of samples before, and then we start to learn to test the mean of the population of multiple groups of samples at the same time.

Analysis of variance (ANOVA): A statistical method for comparing the means of multiple populations. The following focuses on learning: the principles and ideas of one-way analysis of variance, and how to apply multi-factor variance.

1) Introduction to ANOVA

[Analysis of variance] A statistical method for comparing the means of multiple populations. Factors in One/Multivariate ANOVA? Examples of three ANOVA questions:

1. Whether the quality indicators of the same type of product with four different trademarks are consistent.
2. Does taking three different sales methods for the same product lead to significantly different sales volumes?
3. Whether there are significant differences in the purchasing power of residents in five different residential areas.

In these examples, trademarks, sales methods, residential areas, etc.Basis for differentiating groupsis a factor (also called a factor), usually with capital letters $A,B,C$ , etc. represent these factors, and the different states of a factor are called levels, with$A_{},A_{}$etc. said. In all three examples, there is only one factor, so they are all one-way ANOVA; if there are more than one factor, it is called multi-way ANOVA.

For one-way ANOVA, the number of samples to be compared is essentially the number of levels of the factor. For example, in Example 1., we are actually comparing the sample averages of four product quality indicators with different trademarks (the average of the population in which they are located). In this example, the number of factor levels for the factor "brand" is 4.

What is the comparison method for comparing multiple population means in ANOVA? It's not a comparison between the two, it'sCompare them at the same time, written in the form of hypothesis testing:
$$H_{}:m_{}=m_{}=m_i=\cdots m_k\leftrightarrow H_{}:m_i\text{not all equal}$$
where,$k$ is the number of samples (factor levels).
When the population means are not all equal:

2) One-way ANOVA

3) Two-way ANOVA

• One-way ANOVA has the function of comparing multiple population means, but its essence isInquiry Factor $Is\; A$ significant. If significant, it means that these population means are not equal due to factor $Caused\; by\; A$ ; if not significant, it means factor$A$ cannot cause them to be unequal.
• If we increase the number of factors to two, then the ANOVA becomes a two-way ANOVA. It is worth noting that the two-way ANOVA not only explores whether the two factors are significant, but also whether the interaction term of the two factors is significant (similar to regression analysis). For this kind of multivariate analysis of variance, borrowing linear regression models to solve the problem can do more with less.

3. Small exercises

Three lathes produce the same ball, and we extract 13, 14, and 16 products from each, and the measured diameters are:
A lathe: 15.0, 14.5, 15.2, 15.5, 14.8, 15.1, 15.2, 14.8, 13.9, 16.0, 15.1 , 14.5, 15.2;
B Lathe: 15.2, 15.0, 14.8, 15.2, 15.0, 15.0, 14.8, 15.1, 14.8, 15.0, 13.7, 14.1, 15.5, 15.9;
C Lathe: 14.6, 15.0, 14.7, 13.9, 13.8, 15.5 ,15.5,16.2,16.1,15.3,15.4,15,9,15.2,16.0,14.8,14.9

Suppose the significance level is $a=0.01$ , Q:

1. Do the diameters of balls produced by lathe A/B obey a normal distribution?

2. Is there a significant difference in the variance of the ball diameter produced by the A/B lathe?

3. Is there a significant difference in the diameter of the balls produced by the A/B lathe?

4. Is there a significant difference in the ball diameters produced by the three lathes? In one-way ANOVA, what are the factors in this question?

Attachment: time schedule

Task	content	time
Task01	Hypothesis Testing 1: Methodology and Unary Numerical Testing	8-13 - Thursday 8-18
Task02	Hypothesis Test 2: Multivariate Numerical Vector Test	8-29 - Saturday 8-20
Task03	Hypothesis Test 3: Categorical Data Testing	8-21 - 8-22 Monday
Task04	Applied Stochastic Processes and Simulation Systems	8-23 - 8-25 Thursday
Task05	Financial Quantitative Analysis and Stochastic Simulation	8-26 - Sunday 9-28

Reference

[1] https://github.com/Git-Model/Modeling-Universe/tree/main/Data-Story
[2] datawhale course
[3] Data normality test in statistics