[Comprehensive evaluation method] Common comprehensive evaluation methods and their implementation

---

1. Data preprocessing

1.1 Attributes that distinguish indicators

1. positive indicator
2. negative indicator
3. intermediate indicators
4. interval indicator

1.2 Positive indicators

1.2.1 Negative indicators

The positive method of negative indicators, also known as the indicator reversal method, refers to converting indicators that originally reflect negative situations into indicators that reflect positive situations to facilitate comparison and analysis. This method is widely used in evaluation indicators, market research, data analysis and other fields.

Specifically, the positive method of negative indicators can be divided into the following steps:

1. Identify metrics to forwardize

2. Determine what kind of positiveization is needed for negative indicators. Common methods include reciprocal, logarithmic, absolute value, etc.

3. The calculation formula for forwarding indicators depends on different forwarding methods.

4. Normalize the forwarded indicators to make them more comparable

The following introduces a positive method for negative indicators.

For a set of negative indicator data:
$$y_1,y_2,...,y_n$$
取出最大值：
y m a x = max ⁡ { y 1 , y 2 , . . . , y n } y_{max}=\max\left \{ y_{1},y_{2},...,y_{n} \right \} ymax​=max{ y1​,y2​,...,yn​}
Then use this value to update$y_i$ ：
$$y_i:=y_{max}-y_i$$

1.2.2 Intermediate indicators

Intermediate indicators mean that the value of the indicator should not be too small or too large. It is best to take a specific value. For example, the best pH value of a water body is 7. The following introduces a forwarding method for intermediate indicators:

For a set of intermediate indicator data:
$$y_1,y_2,...,y_n$$
First draw up an optimal value:
$$y_{best}$$
Then calculate the distance between each data in this set of data and the optimal value, and take the largest one:

M = max ⁡ { ∣ y 1 − y b e s t ∣ , ∣ y 2 − y b e s t ∣ , . . . , ∣ y n − y b e s t ∣ } M=\max\left \{ \left | y_{1} -y_{best}\right |, \left | y_{2} -y_{best}\right | , ... , \left | y_{n} -y_{best}\right | \right \} M=max{ ∣y1​−ybest​∣,∣y2​−ybest​∣,...,∣yn​−ybest​∣}

Then use this value to update $y_i$：
$$y_i:=1-\frac{|y_i-y_{best}|}{M}$$

1.2.3 Interval indicator

Interval-type indicators mean that the indicator value falls best within a certain range. For example, a person’s body temperature is at $36^{\circ }C$to$37^{\circ }C$is best. The following introduces a forwarding method for interval indicators:

For a set of interval indicator data:
$$y_1,y_2,...,y_n$$
First draw up an optimal interval:
$$\left(a,b\right)$$
Get the maximum and minimum values ​​of this set of data:
ymax = max ⁡ { y 1 , y 2 , . . . , yn } , ymin = min ⁡ { y 1 , y 2 , . . . , yn } y_{ max}=\max\left \{ y_{1},y_{2},...,y_{n} \right \} , y_{min}=\min\left \{ y_{1},y_{ 2},...,y_{n} \right \}ymax​=max{ y1​,y2​,...,yn​},ymin​=min{ y1​,y2​,...,yn​}

Then calculate a value $M$：
M = max ⁡ { a − y m i n , y m a x − b } M=\max\left \{ a-y_{min},y_{max}-b \right \} M=max{ a−ymin​,ymax​−b}

Then use the following formula to update $y_i$：
$$y_i:=\{\begin{array}{c}1-\frac{a-y_i}{a-y_{min}},y_{min}\le y_i<a\\ 1,a\le y_i\le b\\ 1-\frac{y_i-b}{y_{max}-b},b<y_i\le y_{max}\end{array}$$
It is more intuitive to use the following ladder diagram:

1.3 Standardization

1.3.1 Z-score normalization

For sample XXEvery feature in$X\; :$

$$X_{normalized}=\frac{(X-m}{p}$$

Among them, $\mu$ is the mean of the feature,$\sigma$ is the standard deviation of the feature.

1.3.2 Min-max normalization

For sample XXEach feature in$X\; :$

$$X_{normalized}=\frac{(X-X_{min})}{(X_{max}-X_{min})}$$

Among them, $X_{min}$is the minimum value of the feature, $X_{max}$is the maximum value of this feature.

1.3.3 Robust standardization

For sample XXEach feature in$X\; :$

$$X_{normalized}=\frac{(X-median)}{IQR/2}$$

where median is the median for the characteristic and IQR is the interquartile range (that is, the difference between the upper and lower quartiles).

1.3.4 Normalization

For sample XXEach feature in$X\; :$

$$X_{normalized}=\frac{X}{\sqrt{{\sum }_{i=1}^n{x}_i^2}}$$

Among them, $n$ is the number of features of the sample.

2. Fuzzy evaluation method (subjective) (not recommended)

• Suitable for evaluation problems without given indicators

3.Analytic hierarchy process (subjective) (not recommended)

• Suitable for evaluation problems without given indicators

4.PCA principal component analysis method (objective)

Principal component analysis is a commonly used unsupervised dimensionality reduction technique that projects the original data into a new low-dimensional space in order to preserve the maximum variance of the data. By choosing an appropriate number of principal components, we can capture the most important information in the data and reduce the dimensionality of the original data.

4.1 Steps

1. Data forwarding and standardization: Suppose we have ppSample dataX = ( x 1 , x 2 , . . . , xn ) in p$dimension$$\mathbf{X}=({\mathbf{x}}_1,{\mathbf{x}}_2,...,x_n)$ , the data mean of each dimension is 0 and the standard deviation is 1. The purpose of this step is to remove the dimensional influence between dimensions.
   $$x_j^{\prime }=\frac{x_j-\bar{x}}{p_j}(j=1,2,...,p)$$
   where,$\bar{x}$ is the mean of all sample data,$p_j$is $The\; standard\; deviation\; in\; the\; j$ dimension.

2. Calculate covariance matrix: Calculate the covariance matrix of the standardized data. The covariance matrix describes the correlation between different features.

   公式：
   $$S=\frac{1}{n-1}(X-\bar{X}{)}^T(X-\bar{X})$$
   where,$\Sigma$ is the covariance matrix,$X$ is the standardized data matrix,$\bar{X}$ is the mean of each feature,$n$ is the sample size.

3. Calculate eigenvalues ​​and eigenvectors: Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors. The feature vector represents the direction of the data in the new feature space.

4. Select principal components: Sort the eigenvectors according to the size of their eigenvalues, and select the top k eigenvectors as principal components. The eigenvalues ​​corresponding to these principal components are larger and contain more original data information.

5. Calculate projection: Project the original data onto the selected principal components to obtain the dimensionally reduced data.

   公式：$$Y=X_{\text{std}}W$$
   part,$Y$ is the dimensionally reduced data matrix,$X_{\text{std}}$is the standardized data matrix, $W$ is the projection matrix composed of the first k eigenvectors.

6. Optional: Reconstruct the data: Based on the dimensionally reduced data and the projection matrix, the data can be remapped into the original space through inverse transformation.

   公式：$$X_{\text{reconstructed}}=The{W}^T$$
   其中，$X_{\text{reconstructed}}$is the reconstructed data matrix.

4.2 Implementation

>>> import numpy as np
>>> from sklearn.decomposition import PCA

# 输入待降维数据 (5 * 6) 矩阵，6个维度，5个样本值
>>> A = np.array([[84,65,61,72,79,81],[64,77,77,76,55,70],[65,67,63,49,57,67],[74,80,69,75,63,74],[84,74,70,80,74,82]])
>>> print(A)
[[84 65 61 72 79 81]
 [64 77 77 76 55 70]
 [65 67 63 49 57 67]
 [74 80 69 75 63 74]
 [84 74 70 80 74 82]]
 
# 直接使用PCA进行降维
>>> pca = PCA(n_components=2) #降到 2 维
>>> pca.fit(A)
PCA(n_components=2)
>>> pca.transform(A) # 降维后的结果
array([[-16.14860528, -12.48396235],
       [ 10.61676743,  15.67317428],
       [ 23.40212697, -13.607117  ],
       [ -0.43966353,   7.77054621],
       [-17.43062559,   2.64735885]])
>>> pca.explained_variance_ratio_ # 降维后的各主成分的方差值占总方差值的比例，即方差贡献率
array([0.63506778, 0.339022  ])
>>> pca.explained_variance_ # 降维后的各主成分的方差值
array([306.29319053, 163.51030959])

5. Topsis method (objective)

The Topsis comprehensive evaluation method is a multi-dimensional decision analysis method that is suitable for a variety of complex evaluation and decision-making scenarios. To explain more clearly, I will explain each step in more detail.

First of all, the evaluation plan needs to consider multiple evaluation indicators at the same time. These indicators may be contradictory or have different weights. They need to be standardized through a certain mathematical model and weighted according to their relative importance. The Topsis method is based on this framework and uses the following calculation method to find the comprehensive score of each solution on each indicator.

5.1 Forwardization

See 1.2 for details

5.2 Standardization

Generally, the normalization method of 1.3.4 is used.

Assume that there are n solutions (or entities), each solution has m different evaluation indicators, and a comprehensive evaluation is performed among different evaluation indicators. For the j index of each plan i, its standardized value v(i,j) can be obtained through the following calculation:

$$v_{ij}=\frac{x_{ij}}{\sqrt{{\sum }_{i=1}^n{x}_{ij}^2}}$$

where $x_{ij}$Indicates the original data of the j-th indicator of the i-th plan. Standardization unifies the data range of different dimensions to between 0 and 1, and eliminates the impact of data magnitudes.

5.3 Calculate positive and negative ideal solutions

• If no forwarding is done:

For benefit indicators, such as price, income, etc., they need to be maximized; while for cost indicators, such as costs, liabilities, etc., they need to be minimized. The maximum and minimum values ​​of each indicator in the given n solutions can be calculated separately. Let the positive ideal solution of the jth index be $v_j^{+}$, the negative ideal solution is $v_j^{-}$. The specific calculation method is as follows:

For benefit indicators:

v j + = max ⁡ { v i j ∣ i = 1 , 2 , ⋯   , n } v^{+}_{j} = \max{\{v_{ij}| i = 1, 2, \cdots, n\}} vj+​=max{ vij​∣i=1,2,⋯,n}

v j − = min ⁡ { v i j ∣ i = 1 , 2 , ⋯   , n } v^{-}_{j} = \min{\{v_{ij}| i = 1, 2, \cdots, n\}} vj−​=min{ vij​∣i=1,2,⋯,n}

For cost indicators:

$$v_j^{+}=\min \{v_{ij}|i=1,2,\cdots ,n\}$$

$$v_j^{-}=\max \{v_{ij}|i=1,2,\cdots ,n\}$$

• If normalized:

Take the maximum value of each column vector.

5.4 Calculate the distance between each solution and the positive and negative ideal solutions

After normalization, the distance between each solution and the positive and negative ideal solutions can be calculated. Let the distance from the i-th scheme to the positive ideal solution be $S_i^{+}$, the distance to the negative ideal solution is $S_i^{-}$。

$$S_i^{+}=\sqrt{\sum _{j=1}^m{(v_{ij}-v_j^{+})}^2}$$

$$S_i^{-}=\sqrt{\sum _{j=1}^m{(v_{ij}-v_j^{-})}^2}$$

Among them, $m$ is the number of indicator dimensions. $S_i^{+}$Representation scheme $The\; distance\; between\; i$ and the ideal solution,$S_i^{-}$Representation scheme $The\; distance\; between\; i$ and the negative ideal solution. The smaller the value, the closer it is to the ideal solution. Therefore, the range of positive and negative ideal solutions can be expanded to$[0,1]$ as the basis for inspection indicators.

5.5 Calculate overall score

The final comprehensive score $s_i$This can be obtained by weighing each metric as follows:

$$s_i=\frac{S_i^{-}}{S_i^{+}+S_i^{-}}$$

Among them, $S_i^{+}$Represents the $The\; distance\; between\; i$ solutions and the ideal solution,$S_i^{-}$Represents the $The\; distance\; between\; i$ solutions and the negative ideal solution. Comprehensive score$s_i$It can be regarded as the weighted average of the evaluation indicators. When the comprehensive score is higher, it means that the $The\; i$ solutions are better.

Here's an example of how to use this method for decision-making. For example, a company wants to choose the most suitable machine learning platform taking into account multiple metrics. Their evaluation indicators include feature scores (such as size, accuracy of various model types, etc.), service quality scores (including ease of use, response time, data privacy, etc.), price scores, etc. We assume that there are three candidate machine learning platforms, and the evaluation indicators are as shown in the following table:

Candidate platform	Feature Score (0 - 1)	Service quality score (0 - 1)	Price Score (0 - 1)
Platform A	0.8	0.6	0.7
Platform B	0.6	0.8	0.6
Platform C	0.7	0.5	0.8

Use the Topsis method to calculate the score for each platform:

Standardize each evaluation index, calculate the positive and negative ideal solutions that meet each standardized index, and calculate the distance between each platform and the ideal solution:

Candidate platform	Feature score	Service quality score	price score	ideal solution	negative ideal solution	distance to ideal solution	distance to negative ideal solution	overall ratings
Platform A	0.8	0.6	0.7	0.8	0.5	0.2236	0.3606	0.3825
Platform B	0.6	0.8	0.6	0.8	0.5	0.2828	0.2828	0.5000
Platform C	0.7	0.5	0.8	0.8	0.5	0.2449	0.3317	0.4255

Through calculation, Platform B has the highest overall score, so this platform can be recommended as its preferred option for machine learning.

6. Gray relational analysis method (objective)

Gray correlation analysis is a commonly used multi-factor comprehensive evaluation method, which can be used to determine the correlation between different objects and a certain reference object. If we set this reference object as the ideal perfect object, then the gray correlation analysis method can analyze the advantages and disadvantages of different objects.

The specific implementation steps are as follows:

6.1 Collecting data

Establish an evaluation index matrix, in which each row corresponds to a factor (evaluation object), and each column corresponds to an evaluation index. Evaluation indicators can be quantitative indicators or qualitative (qualitative) indicators, but the specific meanings of the indicators must be the same. Let the evaluation index matrix be $X$ , where$x_{ij}$Represents the $i$ factor for$The\; values\; of\; j$ indicators.

Let's give an example first, the relevant data of the index evaluation of the candidate suppliers of a core enterprise:

Evaluation index	Object 1	Object 2	Object 3	Object 4	Object 5	Object 6
product quality	0.83	0.90	0.99	0.92	0.87	0.95
Product price (yuan)	326	295	340	287	310	303
Geographical location (km)	21	38	25	19	27	10
After-sales service (hours)	3.2	2.4	2.2	2.0	0.9	1.7
techinque level	0.20	0.25	0.12	0.33	0.20	0.09
Economic benefits	0.15	0.20	0.14	0.09	0.15	0.17
Supply capacity (pieces)	250	180	300	200	150	175
Market influence	0.23	0.15	0.27	0.30	0.18	0.26
Delivery status	0.87	0.95	0.99	0.89	0.82	0.95

6.2 Normalize and standardize and establish reference objects

Forwardize and standardize the evaluation index matrix, and convert each index into an evaluation index value under the same dimension. The standardization method generally adopts Min-max standardization. See 1.2 and 1.3 for details.

In the above example, product price, geographical location, and after-sales service are negative indicators , while others are positive indicators. The preprocessed data is as follows:

Evaluation index	Object 1	Object 2	Object 3	Object 4	Object 5	Object 6
Indicator 1	0	0.4375	1	0.5625	0.25	0.75
Indicator 2	0.2642	0.8491	0	1	0.566	0.6981
Indicator 3	0.6071	0	0.4643	0.6786	0.3929	1
Indicator 4	0	0.3478	0.4348	0.5217	1	0.6522
Indicator 5	0.4583	0.6667	0.125	1	0.4583	0
Indicator 6	0.5455	1	0.4545	0	0.5455	0.7273
Indicator 7	0.6667	0.2	1	0.3333	0	0.1667
Indicator 8	0.5333	0	0.8	1	0.2	0.7333
Indicator 9	0.2941	0.7647	1	0.4118	0	0.7059

Create a reference object as follows:

Evaluation index	Object 1	Object 2	Object 3	Object 4	Object 5	Object 6	Reference object
Indicator 1	0	0.4375	1	0.5625	0.25	0.75	1
Indicator 2	0.2642	0.8491	0	1	0.566	0.6981	1
Indicator 3	0.6071	0	0.4643	0.6786	0.3929	1	1
Indicator 4	0	0.3478	0.4348	0.5217	1	0.6522	1
Indicator 5	0.4583	0.6667	0.125	1	0.4583	0	1
Indicator 6	0.5455	1	0.4545	0	0.5455	0.7273	1
Indicator 7	0.6667	0.2	1	0.3333	0	0.1667	1
Indicator 8	0.5333	0	0.8	1	0.2	0.7333	1
Indicator 9	0.2941	0.7647	1	0.4118	0	0.7059	1

Since forward processing and Min-max normalization are performed here, the various indicators of the reference object here only need to take the maximum value for each row.

6.3 Determine weights

Determine the weight corresponding to each indicator. However, each row of the indicator matrix will not be weighted for the time being . These weights can be determined using the analytic hierarchy process.
$$oh=\left[{oh}_1,{oh}_2,...,{oh}_n\right],\sum _{i=1}^n{oh}_i=1These$$
weights will be used when calculating the gray correlation degree.

6.4 Calculation of gray correlation coefficient

We remember $x_i$for object $i$，参考对象为$x_0$。$x_i$与$x_0$都有m个指标，我们需要求出它们在第k个指标上的关联系数。关联系数越大，代表这个实际对象越贴近于参考对象。对于n个实际对象，m个指标，$x_i(j)$表示实际对象i的第j个指标的值，那么，$x_i$与$x_0$在第k个指标上的关联系数的计算公式如下：
$$\begin{gathered} {\xi }_i(k)=\frac{{\min }_{1\le s\le n}{\min }_{1\le t\le m}\left|x_0(t)-x_s(t)\right| \\ +\rho {\max }_{1\le s\le n}{\max }_{1\le t\le m}\left|x_0(t)-x_s(t)\right|}{\left|x_0(k)-x_i(k)\right| \\ +\rho {\max }_{1\le s\le n}{\max }_{1\le t\le m}\left|x_0(t)-x_s(t)\right|} \end{gathered}$$

其中，${\min }_{1\le s\le n}{\min }_{1\le t\le m}\left|x_0(t)-x_s(t)\right|$称为两极最小差，${\max }_{1\le s\le n}{\max }_{1\le t\le m}|x_0(t)-x_s(t)|$ is calledthe two-level maximum difference,$\rho$ is calledthe resolution coefficient.

The calculation process of the two-level minimum difference and the two-level maximum difference is the process of comparing each value of the indicator matrix with the reference object. Resolution coefficient $The\; larger\; \rho$ , the greater the resolution;$The\; smaller\; \rho$ , the smaller the resolution

In the above example, we can calculate that the minimum difference between the two levels is 0 and the maximum difference between the two levels is 1. This is due to the use of the Min-max normalization method.

6.5 Calculation and sorting of gray weighted correlation degree

The gray weighted correlation is the final score of each object , which is calculated using the following formula:

$$r_i=\sum _{k=1}^n{w}_i{X}_i(k)$$

Among them, $r_i$Represents the score of the object to be evaluated, $w_i$is the weight determined in 6.3.

Finally, they are sorted according to the scores of each evaluation object. A higher score means the closer the relationship with each indicator, the better.