Machine Learning Vectors

Vector plays a very important role in machine learning.

Basic concepts of vectors

Understanding scalars

A simple number, with only magnitude and no direction, can be expressed as a real number, and is called a scalar in the field of mathematics.

Recognition vector

A vector is an object that has both magnitude and direction.

vector representation

Represented by arrows: Vectors can be represented by lines containing arrows. The length of the line represents the magnitude of the vector, and the arrow represents the direction of the vector.

Vectors with the same magnitude and direction, but different positions. In the field of mathematics, different positions do not matter. They are the same vectors, also known as Identical Vectors.

Text representation: In the field of mathematics, vectors can sometimes be represented by right arrows above English letters, for example: vector $\underset{a}{\rightarrow}$

Including starting point and end point: Sometimes the English letters of the starting point and end point can also be used to represent the vector, and a right arrow must be added above the English letters: $\underset{AB}{\rightarrow}$

Position vector: The vector of the coordinate origin (0,0) is called the position vector.

$\underset{AB}{\rightarrow}=(1,2)$ or $\underset{AB}{\rightarrow}=(1 2)$ or $\underset{AB}{\rightarrow}=\binom{1}{2}$

Common vector representations in machine learning:

Machine learning often needs to deal with the mathematics of n-dimensional space. It is too complicated to use English letters containing starting points and end points to represent vectors. For simplicity, only one English letter is often used to represent vectors, but the letters are displayed in bold and blue.

Components of a vector: In the concept of coordinate axes in a two-dimensional space, the x and y coordinates are the components of this vector.

n-dimensional space vector:

The n-dimensional space vector representation of machine learning is as follows:

${\color{Blue} a}=(a_1\: a_2\: ...\, a_n)$

${\color{Blue} b}=(b_1\; b_2\; ...\; b_n)$

${\color{Blue} c}=(c_1\; c_2\; ...\; c_n)$

Zero vector: Each element of the vector is 0, called zero vector (), which can be expressed in the following ways:

$0=(0\, 0\, ... \, 0)$ Or $\underset{0}{\rightarrow}$

Compute vector components

To calculate $\underset{AB}{\rightarrow}$ the components, assuming that the coordinates of point A are (x1, y1) and the coordinates of point B are (x2, y2), the following method can be used:

(x2y2)-(x1y1)

The calculation method is as follows:

（4 4）-（3 2）=（1 2）

Python operation:

>>> import numpy as np
>>> a=np.array([3,2])
>>> b=np.array([4,4])
>>> b-a
array([1, 2])
>>>

relative position vector

A=（1 3）

B=（2 1）

C=（4 4）

For point A, the vector from A to B is (2 1)-(1 3) = (1-2);

For point A, the vector from A to C is (4 4)-(1 3) = (3-1).

For point B, the vector from B to C is (4 4) - (2 1) = (2 3).

Vector operations on different paths

$\underset{AB}{\rightarrow}+\underset{BC}{\rightarrow}=\underset{AC}{\rightarrow}$

(1-2)+(2 3)=(3 1)

Python operation:

>>> import numpy as np
>>> ab=np.array([1,-2])
>>> bc=np.array([2,3])
>>> ab+bc
array([3, 1])
>>>

Rules for vector addition

$a=(a_1\;a_2\;\cdots \;a_n)$

$b=(b_1\;b_2\;\cdots \;b_n)$

$c=(c_1\;c_2\;\cdots \;c_n)$

Vectors of the same dimension can be added

$a+b=(a_1+b_1\;a_2+b_2 \; \cdots \;a_n+b_n)$

Vectors of different dimensions cannot be added.

Vector addition conforms to the concept of commutative law

Commutative property is a commonly used mathematical term, which means that changing the order does not change the result.

a+b=b+a

Vector addition conforms to the concept of associativity

Associative Laws is a commonly used mathematical term. Its meaning is a formula containing more than two numbers that can be combined. The position of the numbers does not change and the result will not change.

(a+b)+c=a+(b+c)

Adding a vector to a zero vector does not change the result

There is a formula for vector addition: $a+z=a$

It is said $z$ to be a zero vector, which $z$ can also be marked as 0.

The result of adding a vector and its inverse is a zero vector

Opposite Vector refers to a vector that is equal in magnitude but opposite in direction. Assuming that the following formula is true, a and b are inverse quantities of each other.

a+b=0

Sometimes -a can also be used as the inverse of a.

>>> import numpy as np
>>> a=np.array([3,2])
>>> -a
array([-3, -2])
>>>

vector multiplied by title

Multiplying a vector a by a scalar c is equivalent to multiplying c by each vector element, as follows:

$c*a=(ca_1\;ca_2\;\cdots \;ca_n)$

>>> import numpy as np
>>> a=np.array([3,2])
>>> 3*a
array([9, 6])
>>>

vector divided by scalar

Multiply a vector by the reciprocal of a scalar.

Add vectors and multiply by scalars or add scalars and multiply by vectors

$(x+y)*a=xa+ya$

x(a+b)=xa+xb

Multiplication of titles and vectors also conforms to the associative law

(xy)a=x(ya)

vector multiplied by 1

The original vector can be obtained.

a*1=a

vector multiplied by -1

You can get the reverse of the original vector.

a*(-1)=-a

>>> import numpy as np
>>> a=np.array([3,2])
>>> a*-1
array([-3, -2])
>>>

vector multiplied by 0

can get a zero vector.

a*0=0

vector subtraction

If vectors are subtracted, it is equivalent to adding the opposite vector.

a-b=a+(-b)

>>> import numpy as np
>>> a=np.array([3,2])
>>> b=np.array([2,1])
>>> a-b
array([1, 1])
>>>

length of vector

Coordinates of home: (0,0)

Coordinates of the park: (1,3)

Coordinates of the store: (2,1)

Company coordinates: (4,4)

Distance from home to park $=\sqrt{1^2+3^2}=\sqrt{10}$

Distance from shop to park $=\sqrt{(4-2)^2+(4-1)^2}=\sqrt{2^2+3^2}=\sqrt{13}$

Suppose there is a vector a. The length of this vector is represented by: $\mid a\mid$ or $\parallel a \parallel$

Use the norm() method of numpy's linalg module.

Example of finding the distance from home to park:

>>> import numpy as np
>>> park=np.array([1,3])
>>> norm_park=np.linalg.norm(park)
>>> norm_park
3.1622776601683795
>>>

Example of finding the distance from store to company:

>>> import numpy as np
>>> store=np.array([2,1])
>>> office=np.array([4,4])
>>> store_office=office-store
>>> norm_store_office=np.linalg.norm(store_office)
>>> norm_store_office
3.605551275463989
>>>

vector equation

The so-called vector equation is an equation that uses vectors to represent graphics.

Equation of a straight line

Two points can form a line. For vectors, forming a line requires the direction of the vector and a point.

The coordinates of point A and point B are (-1, 2) and (1, 4) respectively.

$\underset{AB}{\rightarrow}=k=\begin{pmatrix} 1-(-1)\\ 4-2 \end{pmatrix}=\begin{pmatrix} 2\\ 2 \end{pmatrix}$

$\underset{a}{\rightarrow}=\begin{pmatrix} -1\\ 2 \end{pmatrix}$

b=a+pk #p is a constant

$\begin{pmatrix} x\\ y \end{pmatrix}=\begin{pmatrix} -1\\ 2 \end{pmatrix}+p\begin{pmatrix} 2\\ 2 \end{pmatrix}$

x=-1+2p

y=2+2p

x-y=-3

y=x+3

The slope is 1 and the y-intercept is 3.

Python practice equations connecting two points

Calculate the values of a and b.

from sympy import Symbol, solve

a = Symbol('a')
b = Symbol('b')
eq1 = -a + b -2
eq2 = a + b - 4
ans = solve((eq1, eq2))
print('a = {}'.format(ans[a]))
print('b = {}'.format(ans[b]))

operation result:

[Running] python -u "c:\Users\a-xiaobodou\OneDrive - Microsoft\Projects\tempCodeRunnerFile.py"
a = 1
b = 3

[Done] exited with code=0 in 3.895 seconds

Reasons to use vectors to build regression lines

omission

vector inner product

The concept of collaborative work

omission

Calculate the help B

omission

Definition of vector inner product

The English word for vector inner product is inner product, and its mathematical representation is: $a\cdot b$

Pronounced a dot b, the result of the vector inner product is a scalar.

The definition of geometric angle vector inner product is the product of the magnitude of two vectors and the cosine of their angle. Concept: $\left \| a \right \|\left \| b \right \| cos\theta$

The length of vector a: $\left \| a \right \|$

The length of vector b projected onto vector a: $\left \| b \right \| cos\theta$

The formula for vector inner product: $a\cdot b$ = $\left \| a \right \|\left \| b \right \| cos\theta$

Commutative law: $a\cdot b=b\cdot a$

Distributive law: $a\cdot (b+c)=a\cdot b+a\cdot c$

The code definition of the inner product is to multiply the corresponding elements of each group of two vectors of equal length, and then sum them.

$a=(a_1\;a_2\;\cdots \;a_n)$

$b=(b_1\;b_2\;\cdots \;b_n)$

Vector product definition: $a\cdot b=\sum_{i=1}^{n}a_ib_i=a_1b_1+a_2b_2+\cdots +a_nb_n$

Calculate the vector inner product using the dot() method of the numpy module:

>>> import numpy as np
>>> a=np.array([1,3])
>>> b=np.array([4,2])
>>> np.dot(a,b)
10
>>>

two direct angles

$cos\theta =\frac{a_1b_1+a_2b_2}{\left \| a \right \|\left \| b \right \|}$

Assume that the coordinate plane has four points A, B, C, and D.

The coordinates of A (1,1)

Coordinates of B (5,5)

Coordinates of C (1,5)

The coordinates of D (5,1)

Calculate the angle between these two vectors.

import numpy as np
import math

a = np.array([1, 1])
b = np.array([5, 5])
c = np.array([1, 5])
d = np.array([5, 1])

ab = b - a                              # 向量ab
cd = d - c                              # 向量bc

norm_a = np.linalg.norm(ab)             # 计算向量大小
norm_b = np.linalg.norm(cd)             # 计算向量大小
                    
dot_ab = np.dot(ab, cd)                 # 计算向量内积

cos_angle = dot_ab / (norm_a * norm_b)  # 计算cos值
rad = math.acos(cos_angle)              # acos转成弧度
deg = math.degrees(rad)                 # 转成角度
print('角度是 = {}'.format(deg))

operation result:

[Running] python -u "c:\Users\a-xiaobodou\OneDrive - Microsoft\Projects\tempCodeRunnerFile.py"
角度是 = 90.0

[Done] exited with code=0 in 4.425 seconds

Properties of vector inner products

The inner product of vectors is positive and the angle between two vectors is less than 90 degrees.

The inner product of vectors is 0, and the angle between two vectors is equal to 90 degrees.

The inner product of vectors is negative, and the angle between two vectors is greater than 90 degrees.

Angle is often used in 3D game design. Assuming that the player's observation point can only see the character's animated expression at an angle of more than 90 degrees, it needs to be drawn. When the angle is less than 90 degrees, the player cannot see the character's animated expression, which means there is no need to draw it. .

cosine similarity

cosine similarity= $cos\theta =\frac{a_1b_1+a_2b_2}{\left \| a \right \|\left \| b \right \|}$

Determine the similarity of the following sentences.

(1) Machines and machinery.

(2) Learn machine code.

(3) Robot learning.

The table below shows the number of occurrences of each word.

serial number	sentence	machine	device	and	machinery	study	habit	code	people
1	Machines and Machinery	2	1	1	1	0	0	0	0
2	Learn machine code	1	1	0	0	1	1	1	0
3	Robot learning	1	1	0	0	1	1	0	1

The following vectors can be created:

a=(2 1 1 1 0 0 0 0)

b=(1 1 0 0 1 1 1 0)

c=(1 1 0 0 1 1 0 1)

code show as below:

import numpy as np

def cosine_similarity(va, vb):
    norm_a = np.linalg.norm(va)                 # 计算向量大小
    norm_b = np.linalg.norm(vb)                 # 计算向量大小
    dot_ab = np.dot(va, vb)                     # 计算向量内积
    return (dot_ab / (norm_a * norm_b))         # 回传相似度

a = np.array([2, 1, 1, 1, 0, 0, 0, 0])
b = np.array([1, 1, 0, 0, 1, 1, 1, 0])
c = np.array([1, 1, 0, 0, 1, 1, 0, 1])
print('a 和 b 相似度 = {0:5.3f}'.format(cosine_similarity(a, b)))
print('a 和 c 相似度 = {0:5.3f}'.format(cosine_similarity(a, c)))
print('b 和 c 相似度 = {0:5.3f}'.format(cosine_similarity(b, c)))

The running results are as follows:

[Running] python -u "c:\Users\a-xiaobodou\OneDrive - Microsoft\Projects\tempCodeRunnerFile.py"
a 和 b 相似度 = 0.507
a 和 c 相似度 = 0.507
b 和 c 相似度 = 0.800

[Done] exited with code=0 in 3.242 seconds

Pearson correlation coefficient

In statistics, the Pearson Correlation Coefficient is commonly used to measure the degree of correlation between two variables x and y. The value range of this coefficient is -1~1. The basic concept is as follows:

(1) The coefficient value is 1: It means that all data are on a straight line, and the y value increases as the x value increases. The closer the coefficient value is to 1, the higher the degree of positive correlation between the x and y variables.

(2) The coefficient value is -1: it means that all data are on a straight line, and the y value decreases as the x value increases. The closer the coefficient value is to -1, the higher the negative correlation between the x and y variables.

(3) The coefficient value is 0: It means there is no linear relationship between the two variables, that is, the change in the y value is completely unrelated to the x value. The closer the coefficient is to 0, the higher the degree of complete irrelevance between the x and y variables.

The following are common definitions of correlation coefficients.

Correlation	just	burden
powerful	0.6~1.0	-1.0~(-0.6)
middle	0.3~0.6	-0.6~(-0.3)
weak	0.1~0.3	-0.3~(-0.1)
none	-0.09	-0.09~(-0.0)

Pearson correlation coefficient definition

The Pearson correlation coefficient is the quotient of the covariance and standard deviation between two variables. Generally, r is used as the variable of the Pearson coefficient. The formula is as follows:

$r=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^{n}((x_i-\bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i-\bar{y})^2}}$

Online shopping questionnaire case explanation

An online shopping company conducted a questionnaire survey in December 2019 to ask consumers about their satisfaction with the entire shopping. At the same time, in January 2021, it asked the respondents of the previous year about the number of times they purchased goods again in 2020. The data obtained is as follows:

Questionnaire number	Satisfaction	Number of re-purchases
1	8	12
2	9	15
3	10	16
4	7	18
5	8	6
6	9	11
7	5	3
8	7	12
9	9	11
10	8	16

Below is a table for calculating the above data.

(1) Calculate satisfaction-average satisfaction (the calculated average satisfaction is 0): $(x_i-\bar{x})$

(2) Calculate the number of repurchases - the average number of repurchases (the calculated average number of repurchases is 12): $(y_i-\bar{y})$

Questionnaire number	Satisfaction - average satisfaction	Number of repurchases - average number of repurchases
1	0	0
2	1	3
3	2	4
4	-1	6
5	0	-6
6	1	-1
7	-3	-9
8	-1	0
9	1	-1
10	0	4

Substituting the data into the Pearson coefficient company, the following data table can be obtained.

Questionnaire number	$(x_i-\bar{x})(y_i-\bar{y})$	$(x_i-\bar{x})^2$	$(y_i-\bar{y})^2$
1	0	0	0
2	3	1	9
3	8	4	16
4	-6	1	36
5	0	0	36
6	-1	1	1
7	27	9	81
8	0	1	0
9	-1	1	1
10	0	0	16
total	30	18	196

Substituting the above values into Pearson's Coefficient, we get the following results.

$r=\frac{30}{\sqrt{18}\sqrt{196}}=0.505$

From the above execution results, we can see that consumer satisfaction is positively correlated with repurchase, but the correlation strength is medium.

Verify the above results with a Python program.

import numpy as np

x = np.array([8, 9, 10, 7, 8, 9, 5, 7, 9, 8])
y = np.array([12, 15, 16, 18, 6, 11, 3, 12, 11, 16])             
x_mean = np.mean(x)
y_mean = np.mean(y)

xi_x = [v - x_mean  for v in x]
yi_y = [v - y_mean  for v in y]

data1 = [0]*10
data2 = [0]*10
data3 = [0]*10
for i in range(len(x)):
    data1[i] = xi_x[i] * yi_y[i]
    data2[i] = xi_x[i]**2
    data3[i] = yi_y[i]**2

v1 = np.sum(data1)
v2 = np.sum(data2)
v3 = np.sum(data3)
r = v1 / ((v2**0.5)*(v3**0.5))
print('coefficient = {}'.format(r))

operation result:

[Running] python -u "c:\Users\a-xiaobodou\OneDrive - Microsoft\Projects\tempCodeRunnerFile.py"
coefficient = 0.5050762722761054

[Done] exited with code=0 in 2.014 seconds

Plot a scatter plot of consumer satisfaction versus the number of repurchases.

import numpy as np
import matplotlib.pyplot as plt

x = np.array([8, 9, 10, 7, 8, 9, 5, 7, 9, 8])
y = np.array([12, 15, 16, 18, 6, 11, 3, 12, 11, 16])             
x_mean = np.mean(x)
y_mean = np.mean(y)
xpt1 = np.linspace(0, 12, 12)      
ypt1 = [y_mean for xp in xpt1]          # 平均购买次数
ypt2 = np.linspace(0, 20, 20)
xpt2 = [x_mean for yp in ypt2]          # 平均满意度

plt.scatter(x, y)                       # 满意度 vs 购买次数
plt.plot(xpt1, ypt1, 'g')               # 平均购买次数
plt.plot(xpt2, ypt2, 'g')               # 平均满意度
plt.grid()
plt.show()

operation result:

There are only 9 points above because the data (9,11) overlap. If it is highly correlated, the number of repurchases should be higher if satisfaction is high, and the number of repurchases if satisfaction is low should be low. This means that the data should be concentrated in the upper right and lower left of the green cross line.

Vector inner product calculation coefficient

Don't understand, omit it

vector outer product

Vector outer product is also called Cross Product or Vector Product, which is a two-dimensional operation for two vectors in three-dimensional space. Therefore, to perform the outer product operation of vectors a and b, we must first assume that a and b are on the same plane. Two vectors a and b perform an outer product, so the symbol used is X, and the representation method is as follows: $a\times b$

Vector outer product is often used in mathematics, physics and machine learning.

(1) The result of vector outer product is not a scalar but a vector.

(2) For a and b, the vector outer product is the vector perpendicular to the two vectors, also called the normal vector.

(3) The size of the above normal vector is the area of the parallelogram composed of vectors a and b.

normal vector

The vector product of a and b is a vector perpendicular to the two vectors.

Assume that the contents of the a and b vectors are as follows:

$a=\begin{pmatrix} a_1\\ a_2\\ a_3 \end{pmatrix}$ $b=\begin{pmatrix} b_1\\ b_2\\ b_3 \end{pmatrix}$

The formula for calculating the outer product of vectors a and b is as follows:

$a\times b=a=\begin{pmatrix} a_2b_3-a_3b_2\\ a_3b_1-a_1b_3\\ a_1b_2-a_2b_1 \end{pmatrix}$

The numpy module has the cross() method to perform this vector outer product calculation.

>>> import numpy as np
>>> a=np.array([0,1,2])
>>> b=np.array([2,0,2])
>>> np.cross(a,b)
array([ 2,  4, -2])
>>>

Calculate area

The size of the normal vector is equal to the area of the parallelogram formed by the two vectors.

To calculate the area of a triangle formed by two vectors, the steps are as follows:

(1) Calculate the length of two vectors;

(2) Calculate the angle between two vectors;

(3) Apply the above formula to calculate the area of the parallelogram;

(4) Divide the result of step (3) by 2 to get the area of the triangle formed by the two vectors.

import numpy as np
import math

a = np.array([4, 2])
b = np.array([1, 3])

norm_a = np.linalg.norm(a)              # 计算向量大小
norm_b = np.linalg.norm(b)              # 计算向量大小
                    
dot_ab = np.dot(a, b)                   # 计算向量内积

cos_angle = dot_ab / (norm_a * norm_b)  # 计算cos值
rad = math.acos(cos_angle)              # acos转成弧度

area = norm_a * norm_b * math.sin(rad) / 2
print('area = {0:5.2f}'.format(area))

The running results are as follows:

[Running] python -u "c:\Users\a-xiaobodou\OneDrive - Microsoft\Projects\tempCodeRunnerFile.py"
area =  5.00

[Done] exited with code=0 in 2.393 seconds

Use the vector outer product concept to calculate the area of a triangle formed by two vectors.

import numpy as np

a = np.array([4, 2])
b = np.array([1, 3])

ab_cross = np.cross(a, b)               # 计算向量外积
area = np.linalg.norm(ab_cross) / 2     # 向量长度除以2
                    
print('area = {0:5.2f}'.format(area))

The running results are as follows:

[Running] python -u "c:\Users\a-xiaobodou\OneDrive - Microsoft\Projects\tempCodeRunnerFile.py"
area =  5.00

[Done] exited with code=0 in 2.142 seconds