Python 被称为是最接近 AI 的语言。最近一位名叫Anna-Lena Popkes的小姐姐在GitHub上分享了自己如何使用Python（3.6及以上版本）实现7种机器学习算法的笔记，并附有完整代码。所有这些算法的实现都没有使用其他机器学习库。这份笔记可以帮大家对算法以及其底层结构有个基本的了解，但并不是提供最有效的实现。
七种算法包括：
线性回归算法
Logistic 回归算法
感知器
K 最近邻算法
K 均值聚类算法
含单隐层的神经网络
多项式的 Logistic 回归算法
扫描二维码关注公众号，回复： 707279 查看本文章
1. 线性回归算法
在线性回归中，我们想要建立一个模型，来拟合一个因变量 y 与一个或多个独立自变量(预测变量) x 之间的关系。
给定：
数据集
是d-维向量
是一个目标变量，它是一个标量
线性回归模型可以理解为一个非常简单的神经网络：
它有一个实值加权向量
它有一个实值偏置量 b
它使用恒等函数作为其激活函数
线性回归模型可以使用以下方法进行训练
a) 梯度下降法
b) 正态方程(封闭形式解)：
其中 X 是一个矩阵，其形式为
而正态方程需要计算
线性回归模型的训练过程有不同的步骤。首先(在步骤 0 中)，模型的参数将被初始化。在达到指定训练次数或参数收敛前，重复以下其他步骤。
第 0 步：
用0 (或小的随机值)来初始化权重向量和偏置量，或者直接使用正态方程计算模型参数
第 1 步(只有在使用梯度下降法训练时需要)：
计算输入的特征与权重值的线性组合，这可以通过矢量化和矢量传播来对所有训练样本进行处理：
其中 X 是所有训练样本的维度矩阵，其形式为
第 2 步(只有在使用梯度下降法训练时需要)：
用均方误差计算训练集上的损失：
第 3 步(只有在使用梯度下降法训练时需要):
对每个参数，计算其对损失函数的偏导数：
所有偏导数的梯度计算如下：
第 4 步(只有在使用梯度下降法训练时需要）:
更新权重向量和偏置量：
其中，
In [4]:
1import numpy as np
2import matplotlib.pyplot as plt
3from sklearn.model_selection import train_test_split
4np.random.seed(123)
数据集
In [5]:
1# We will use a simple training set
2X = 2 * np.random.rand(500, 1)
3y = 5 + 3 * X + np.random.randn(500, 1)
4fig = plt.figure(figsize=(8,6))
5plt.scatter(X, y)
6plt.title("Dataset")
7plt.xlabel("First feature")
8plt.ylabel("Second feature")
9plt.show()
In [6]:
1# Split the data into a training and test set
2X_train, X_test, y_train, y_test = train_test_split(X, y)
3print(f'Shape X_train: {X_train.shape}')
4print(f'Shape y_train: {y_train.shape}')
5print(f'Shape X_test: {X_test.shape}')
6print(f'Shape y_test: {y_test.shape}')Shape X_train: (375, 1)Shape y_train: (375, 1)Shape X_test: (125, 1)Shape y_test: (125, 1)
线性回归分类
In [23]:
1class LinearRegression:
2
3 def __init__(self):
4 pass
5
6 def train_gradient_descent(self, X, y, learning_rate=0.01, n_iters=100):
7 """
8 Trains a linear regression model using gradient descent
9 """
10 # Step 0: Initialize the parameters
11 n_samples, n_features = X.shape
12 self.weights = np.zeros(shape=(n_features,1))
13 self.bias = 0
14 costs = []
15
16 for i in range(n_iters):
17 # Step 1: Compute a linear combination of the input features and weights
18 y_predict = np.dot(X, self.weights) + self.bias
19
20 # Step 2: Compute cost over training set
21 cost = (1 / n_samples) * np.sum((y_predict - y)**2)
22 costs.append(cost)
23
24 if i % 100 == 0:
25 print(f"Cost at iteration {i}: {cost}")
26
27 # Step 3: Compute the gradients
28 dJ_dw = (2 / n_samples) * np.dot(X.T, (y_predict - y))
29 dJ_db = (2 / n_samples) * np.sum((y_predict - y))
30
31 # Step 4: Update the parameters
32 self.weights = self.weights - learning_rate * dJ_dw
33 self.bias = self.bias - learning_rate * dJ_db
34
35 return self.weights, self.bias, costs
36
37 def train_normal_equation(self, X, y):
38 """
39 Trains a linear regression model using the normal equation
40 """
41 self.weights = np.dot(np.dot(np.linalg.inv(np.dot(X.T, X)), X.T), y)
42 self.bias = 0
43
44 return self.weights, self.bias
45
46 def predict(self, X):
47 return np.dot(X, self.weights) + self.bias
使用梯度下降进行训练
In [24]:
1regressor = LinearRegression()
2w_trained, b_trained, costs = regressor.train_gradient_descent(X_train, y_train, learning_rate=0.005, n_iters=600)
3fig = plt.figure(figsize=(8,6))
4plt.plot(np.arange(n_iters), costs)
5plt.title("Development of cost during training")
6plt.xlabel("Number of iterations")
7plt.ylabel("Cost")
8plt.show()
Cost at iteration 0: 66.45256981003433 Cost at iteration 100: 2.2084346146095934 Cost at iteration 200: 1.2797812854182806 Cost at iteration 300: 1.2042189195356685 Cost at iteration 400: 1.1564867816573 Cost at iteration 500: 1.121391041394467
测试（梯度下降模型）
In [28]:
1n_samples, _ = X_train.shape
2n_samples_test, _ = X_test.shape
3
4y_p_train = regressor.predict(X_train)
5y_p_test = regressor.predict(X_test)
6
7error_train = (1 / n_samples) * np.sum((y_p_train - y_train) ** 2)
8error_test = (1 / n_samples_test) * np.sum((y_p_test - y_test) ** 2)
9
10print(f"Error on training set: {np.round(error_train, 4)}")
11print(f"Error on test set: {np.round(error_test)}")
Error on training set: 1.0955
Error on test set: 1.0
使用正规方程（normal equation）训练
1# To compute the parameters using the normal equation, we add a bias value of 1 to each input example
2X_b_train = np.c_[np.ones((n_samples)), X_train]
3X_b_test = np.c_[np.ones((n_samples_test)), X_test]
4
5reg_normal = LinearRegression()
6w_trained = reg_normal.train_normal_equation(X_b_train, y_train)
测试（正规方程模型）
1y_p_train = reg_normal.predict(X_b_train)
2y_p_test = reg_normal.predict(X_b_test)
3
4error_train = (1 / n_samples) * np.sum((y_p_train - y_train) ** 2)
5error_test = (1 / n_samples_test) * np.sum((y_p_test - y_test) ** 2)
6
7print(f"Error on training set: {np.round(error_train, 4)}")
8print(f"Error on test set: {np.round(error_test, 4)}")
Error on training set: 1.0228
Error on test set: 1.0432
可视化测试预测
1# Plot the test predictions
2
3
4fig = plt.figure(figsize=(8,6))
5plt.scatter(X_train, y_train)
6plt.scatter(X_test, y_p_test)
7plt.xlabel("First feature")
8plt.ylabel("Second feature")
9plt.show()
2. Logistic 回归算法
在 Logistic 回归中，我们试图对给定输入特征的线性组合进行建模，来得到其二元变量的输出结果。例如，我们可以尝试使用竞选候选人花费的金钱和时间信息来预测选举的结果(胜或负)。Logistic 回归算法的工作原理如下。
给定：
数据集
是d-维向量
是一个二元的目标变量
Logistic 回归模型可以理解为一个非常简单的神经网络：
它有一个实值加权向量
它有一个实值偏置量 b
它使用 sigmoid 函数作为其激活函数
与线性回归不同，Logistic 回归没有封闭解。但由于损失函数是凸函数，因此我们可以使用梯度下降法来训练模型。事实上，在保证学习速率足够小且使用足够的训练迭代步数的前提下，梯度下降法(或任何其他优化算法)可以是能够找到全局最小值。
训练 Logistic 回归模型有不同的步骤。首先(在步骤 0 中)，模型的参数将被初始化。在达到指定训练次数或参数收敛前，重复以下其他步骤。
第 0 步：用 0 (或小的随机值)来初始化权重向量和偏置值
第 1 步：计算输入的特征与权重值的线性组合，这可以通过矢量化和矢量传播来对所有训练样本进行处理：
其中 X 是所有训练样本的维度矩阵，其形式为
第 2 步：用 sigmoid 函数作为激活函数，其返回值介于0到1之间：
第 3 步：计算整个训练集的损失值。
我们希望模型得到的目标值概率落在 0 到 1 之间。因此在训练期间，我们希望调整参数，使得模型较大的输出值对应正标签(真实标签为 1)，较小的输出值对应负标签(真实标签为 0 )。这在损失函数中表现为如下形式：
第 4 步：对权重向量和偏置量，计算其对损失函数的梯度。
关于这个导数实现的详细解释，可以参见这里（https://stats.stackexchange.com/questions/278771/how-is-the-cost-function-from-logistic-regression-derivated）。
一般形式如下：
对于偏置量的导数计算，此时为 1。
第 5 步：更新权重和偏置值。
其中，表示学习率。
In [24]:
1import numpy as np
2from sklearn.model_selection import train_test_split
3from sklearn.datasets import make_blobs
4import matplotlib.pyplot as plt
5np.random.seed(123)
6
7% matplotlib inline
数据集
In [25]:
1# We will perform logistic regression using a simple toy dataset of two classes
2X, y_true = make_blobs(n_samples= 1000, centers=2)
3
4fig = plt.figure(figsize=(8,6))
5plt.scatter(X[:,0], X[:,1], c=y_true)
6plt.title("Dataset")
7plt.xlabel("First feature")
8plt.ylabel("Second feature")
9plt.show()
In [26]:
1# Reshape targets to get column vector with shape (n_samples, 1)
2y_true = y_true[:, np.newaxis]
3# Split the data into a training and test set
4X_train, X_test, y_train, y_test = train_test_split(X, y_true)
5print(f'Shape X_train: {X_train.shape}')
6print(f'Shape y_train: {y_train.shape}')
7print(f'Shape X_test: {X_test.shape}')
8print(f'Shape y_test: {y_test.shape}')
Shape X_train: (750, 2)
Shape y_train: (750, 1)
Shape X_test: (250, 2)
Shape y_test: (250, 1)
Logistic回归分类
In [27]:
1class LogisticRegression:
2
3 def __init__(self):
4 pass
5
6 def sigmoid(self, a):
7 return 1 / (1 + np.exp(-a))
8
9 def train(self, X, y_true, n_iters, learning_rate):
10 """
11 Trains the logistic regression model on given data X and targets y
12 """
13 # Step 0: Initialize the parameters
14 n_samples, n_features = X.shape
15 self.weights = np.zeros((n_features, 1))
16 self.bias = 0
17 costs = []
18
19 for i in range(n_iters):
20 # Step 1 and 2: Compute a linear combination of the input features and weights,
21 # apply the sigmoid activation function
22 y_predict = self.sigmoid(np.dot(X, self.weights) + self.bias)
23
24 # Step 3: Compute the cost over the whole training set.
25 cost = (- 1 / n_samples) * np.sum(y_true * np.log(y_predict) + (1 - y_true) * (np.log(1 - y_predict)))
26
27 # Step 4: Compute the gradients
28 dw = (1 / n_samples) * np.dot(X.T, (y_predict - y_true))
29 db = (1 / n_samples) * np.sum(y_predict - y_true)
30
31 # Step 5: Update the parameters
32 self.weights = self.weights - learning_rate * dw
33 self.bias = self.bias - learning_rate * db
34
35 costs.append(cost)
36 if i % 100 == 0:
37 print(f"Cost after iteration {i}: {cost}")
38
39 return self.weights, self.bias, costs
40
41 def predict(self, X):
42 """
43 Predicts binary labels for a set of examples X.
44 """
45 y_predict = self.sigmoid(np.dot(X, self.weights) + self.bias)
46 y_predict_labels = [1 if elem > 0.5 else 0 for elem in y_predict]
47
48 return np.array(y_predict_labels)[:, np.newaxis]
初始化并训练模型
In [29]:
1regressor = LogisticRegression()
2w_trained, b_trained, costs = regressor.train(X_train, y_train, n_iters=600, learning_rate=0.009)
3
4fig = plt.figure(figsize=(8,6))
5plt.plot(np.arange(600), costs)
6plt.title("Development of cost over training")
7plt.xlabel("Number of iterations")
8plt.ylabel("Cost")
9plt.show()
Cost after iteration 0: 0.6931471805599453
Cost after iteration 100: 0.046514002935609956
Cost after iteration 200: 0.02405337743999163
Cost after iteration 300: 0.016354408151412207
Cost after iteration 400: 0.012445770521974634
Cost after iteration 500: 0.010073981792906512
测试模型
In [31]:
1y_p_train = regressor.predict(X_train)
2y_p_test = regressor.predict(X_test)
3
4print(f"train accuracy: {100 - np.mean(np.abs(y_p_train - y_train)) * 100}%")
5print(f"test accuracy: {100 - np.mean(np.abs(y_p_test - y_test))}%")
train accuracy: 100.0%
test accuracy: 100.0%
3. 感知器算法
感知器是一种简单的监督式的机器学习算法，也是最早的神经网络体系结构之一。它由 Rosenblatt 在 20 世纪 50 年代末提出。感知器是一种二元的线性分类器，其使用 d- 维超平面来将一组训练样本( d- 维输入向量)映射成二进制输出值。它的原理如下：
给定：
数据集
是d-维向量
是一个目标变量，它是一个标量
感知器可以理解为一个非常简单的神经网络：
它有一个实值加权向量
它有一个实值偏置量 b
它使用 Heaviside step 函数作为其激活函数
感知器的训练可以使用梯度下降法，训练算法有不同的步骤。首先(在步骤0中)，模型的参数将被初始化。在达到指定训练次数或参数收敛前，重复以下其他步骤。
第 0 步：用 0 (或小的随机值)来初始化权重向量和偏置值
第 1 步：计算输入的特征与权重值的线性组合，这可以通过矢量化和矢量传播法则来对所有训练样本进行处理：
其中 X 是所有训练示例的维度矩阵，其形式为
第 2 步：用 Heaviside step 函数作为激活函数，其返回一个二进制值：
第 3 步：使用感知器的学习规则来计算权重向量和偏置量的更新值。
其中，表示学习率。
第 4 步：更新权重向量和偏置量。
In [1]:
1import numpy as np
2import matplotlib.pyplot as plt
3from sklearn.datasets import make_blobs
4from sklearn.model_selection import train_test_split
5np.random.seed(123)
6
7% matplotlib inline
数据集
In [2]:
11X, y = make_blobs(n_samples=1000, centers=2)
22fig = plt.figure(figsize=(8,6))
33plt.scatter(X[:,0], X[:,1], c=y)
44plt.title("Dataset")
55plt.xlabel("First feature")
66plt.ylabel("Second feature")
77plt.show()
In [3]:
1y_true = y[:, np.newaxis]
2
3X_train, X_test, y_train, y_test = train_test_split(X, y_true)
4print(f'Shape X_train: {X_train.shape}')
5print(f'Shape y_train: {y_train.shape})')
6print(f'Shape X_test: {X_test.shape}')
7print(f'Shape y_test: {y_test.shape}')
Shape X_train: (750, 2)
Shape y_train: (750, 1))
Shape X_test: (250, 2)
Shape y_test: (250, 1)
感知器分类
In [6]:
1class Perceptron():
2
3 def __init__(self):
4 pass
5
6 def train(self, X, y, learning_rate=0.05, n_iters=100):
7 n_samples, n_features = X.shape
8
9 # Step 0: Initialize the parameters
10 self.weights = np.zeros((n_features,1))
11 self.bias = 0
12
13 for i in range(n_iters):
14 # Step 1: Compute the activation
15 a = np.dot(X, self.weights) + self.bias
16
17 # Step 2: Compute the output
18 y_predict = self.step_function(a)
19
20 # Step 3: Compute weight updates
21 delta_w = learning_rate * np.dot(X.T, (y - y_predict))
22 delta_b = learning_rate * np.sum(y - y_predict)
23
24 # Step 4: Update the parameters
25 self.weights += delta_w
26 self.bias += delta_b
27
28 return self.weights, self.bias
29
30 def step_function(self, x):
31 return np.array([1 if elem >= 0 else 0 for elem in x])[:, np.newaxis]
32
33 def predict(self, X):
34 a = np.dot(X, self.weights) + self.bias
35 return self.step_function(a)
初始化并训练模型
In [7]:
1p = Perceptron()
2w_trained, b_trained = p.train(X_train, y_train,learning_rate=0.05, n_iters=500)
测试
In [10]:
1y_p_train = p.predict(X_train)
2y_p_test = p.predict(X_test)
3
4print(f"training accuracy: {100 - np.mean(np.abs(y_p_train - y_train)) * 100}%")
5print(f"test accuracy: {100 - np.mean(np.abs(y_p_test - y_test)) * 100}%")
training accuracy: 100.0%
test accuracy: 100.0%
可视化决策边界
In [13]:
1def plot_hyperplane(X, y, weights, bias):
2 """
3 Plots the dataset and the estimated decision hyperplane
4 """
5 slope = - weights[0]/weights[1]
6 intercept = - bias/weights[1]
7 x_hyperplane = np.linspace(-10,10,10)
8 y_hyperplane = slope * x_hyperplane + intercept
9 fig = plt.figure(figsize=(8,6))
10 plt.scatter(X[:,0], X[:,1], c=y)
11 plt.plot(x_hyperplane, y_hyperplane, '-')
12 plt.title("Dataset and fitted decision hyperplane")
13 plt.xlabel("First feature")
14 plt.ylabel("Second feature")
15 plt.show()
In [14]:
1plot_hyperplane(X, y, w_trained, b_trained)
4. K 最近邻算法
k-nn 算法是一种简单的监督式的机器学习算法，可以用于解决分类和回归问题。这是一个基于实例的算法，并不是估算模型，而是将所有训练样本存储在内存中，并使用相似性度量进行预测。
给定一个输入示例，k-nn 算法将从内存中检索 k 个最相似的实例。相似性是根据距离来定义的，也就是说，与输入示例之间距离最小(欧几里得距离)的训练样本被认为是最相似的样本。
输入示例的目标值计算如下：
分类问题：
a) 不加权：输出 k 个最近邻中最常见的分类
b) 加权：将每个分类值的k个最近邻的权重相加，输出权重最高的分类
回归问题：
a) 不加权：输出k个最近邻值的平均值
b) 加权：对于所有分类值，将分类值加权求和并将结果除以所有权重的总和
加权版本的 k-nn 算法是改进版本，其中每个近邻的贡献值根据其与查询点之间的距离进行加权。下面，我们在 sklearn 用 k-nn 算法的原始版本实现数字数据集的分类。
In [1]:
1import numpy as np
2import matplotlib.pyplot as plt
3from sklearn.datasets import load_digits
4from sklearn.model_selection import train_test_split
5np.random.seed(123)
6
7% matplotlib inline
数据集
In [2]:
1# We will use the digits dataset as an example. It consists of the 1797 images of hand-written digits. Each digit is
2# represented by a 64-dimensional vector of pixel values.
3
4
5digits = load_digits()
6X, y = digits.data, digits.target
7
8X_train, X_test, y_train, y_test = train_test_split(X, y)
9print(f'X_train shape: {X_train.shape}')
10print(f'y_train shape: {y_train.shape}')
11print(f'X_test shape: {X_test.shape}')
12print(f'y_test shape: {y_test.shape}')
13
14# Example digits
15fig = plt.figure(figsize=(10,8))
16for i in range(10):
17 ax = fig.add_subplot(2, 5, i+1)
18 plt.imshow(X[i].reshape((8,8)), cmap='gray')
X_train shape: (1347, 64)
y_train shape: (1347,)
X_test shape: (450, 64)
y_test shape: (450,)
K 最邻近类别
In [3]:
1class kNN():
2 def __init__(self):
3 pass
4
5 def fit(self, X, y):
6 self.data = X
7 self.targets = y
8
9 def euclidean_distance(self, X):
10 """
11 Computes the euclidean distance between the training data and
12 a new input example or matrix of input examples X
13 """
14 # input: single data point
15 if X.ndim == 1:
16 l2 = np.sqrt(np.sum((self.data - X)**2, axis=1))
17
18 # input: matrix of data points
19 if X.ndim == 2:
20 n_samples, _ = X.shape
21 l2 = [np.sqrt(np.sum((self.data - X[i])**2, axis=1)) for i in range(n_samples)]
22
23 return np.array(l2)
24
25 def predict(self, X, k=1):
26 """
27 Predicts the classification for an input example or matrix of input examples X
28 """
29 # step 1: compute distance between input and training data
30 dists = self.euclidean_distance(X)
31
32 # step 2: find the k nearest neighbors and their classifications
33 if X.ndim == 1:
34 if k == 1:
35 nn = np.argmin(dists)
36 return self.targets[nn]
37 else:
38 knn = np.argsort(dists)[:k]
39 y_knn = self.targets[knn]
40 max_vote = max(y_knn, key=list(y_knn).count)
41 return max_vote
42
43 if X.ndim == 2:
44 knn = np.argsort(dists)[:, :k]
45 y_knn = self.targets[knn]
46 if k == 1:
47 return y_knn.T
48 else:
49 n_samples, _ = X.shape
50 max_votes = [max(y_knn[i], key=list(y_knn[i]).count) for i in range(n_samples)]
51 return max_votes
初始化并训练模型
In [11]:
1knn = kNN()
2knn.fit(X_train, y_train)
3
4print("Testing one datapoint, k=1")
5print(f"Predicted label: {knn.predict(X_test[0], k=1)}")
6print(f"True label: {y_test[0]}")
7print()
8print("Testing one datapoint, k=5")
9print(f"Predicted label: {knn.predict(X_test[20], k=5)}")
10print(f"True label: {y_test[20]}")
11print()
12print("Testing 10 datapoint, k=1")
13print(f"Predicted labels: {knn.predict(X_test[5:15], k=1)}")
14print(f"True labels: {y_test[5:15]}")
15print()
16print("Testing 10 datapoint, k=4")
17print(f"Predicted labels: {knn.predict(X_test[5:15], k=4)}")
18print(f"True labels: {y_test[5:15]}")
19print()
Testing one datapoint, k=1
Predicted label: 3
True label: 3
Testing one datapoint, k=5
Predicted label: 9
True label: 9
Testing 10 datapoint, k=1
Predicted labels: [[3 1 0 7 4 0 0 5 1 6]]
True labels: [3 1 0 7 4 0 0 5 1 6]
Testing 10 datapoint, k=4
Predicted labels: [3, 1, 0, 7, 4, 0, 0, 5, 1, 6]
True labels: [3 1 0 7 4 0 0 5 1 6]
测试集精度
In [12]:
1# Compute accuracy on test set
2y_p_test1 = knn.predict(X_test, k=1)
3test_acc1= np.sum(y_p_test1[0] == y_test)/len(y_p_test1[0]) * 100
4print(f"Test accuracy with k = 1: {format(test_acc1)}")
5
6y_p_test8 = knn.predict(X_test, k=5)
7test_acc8= np.sum(y_p_test8 == y_test)/len(y_p_test8) * 100
8print(f"Test accuracy with k = 8: {format(test_acc8)}")
Test accuracy with k = 1: 97.77777777777777
Test accuracy with k = 8: 97.55555555555556
5. K均值聚类算法法
K-Means 是一种非常简单的聚类算法(聚类算法都属于无监督学习)。给定固定数量的聚类和输入数据集，该算法试图将数据划分为聚类，使得聚类内部具有较高的相似性，聚类与聚类之间具有较低的相似性。
算法原理
1. 初始化聚类中心，或者在输入数据范围内随机选择，或者使用一些现有的训练样本(推荐)
2. 直到收敛
将每个数据点分配到最近的聚类。点与聚类中心之间的距离是通过欧几里德距离测量得到的。
通过将聚类中心的当前估计值设置为属于该聚类的所有实例的平均值，来更新它们的当前估计值。
目标函数
聚类算法的目标函数试图找到聚类中心，以便数据将划分到相应的聚类中，并使得数据与其最接近的聚类中心之间的距离尽可能小。
给定一组数据X1，...，Xn和一个正数k，找到k个聚类中心C1，...，Ck并最小化目标函数：
这里：
决定了数据点
是否属于类
表示类
的聚类中心
表示欧几里得距离
K-Means 算法的缺点：
聚类的个数在开始就要设定
聚类的结果取决于初始设定的聚类中心
对异常值很敏感
不适合用于发现非凸聚类问题
该算法不能保证能够找到全局最优解，因此它往往会陷入一个局部最优解
In [21]:
1import numpy as np
2import matplotlib.pyplot as plt
3import random
4from sklearn.datasets import make_blobs
5np.random.seed(123)
6
7% matplotlib inline
数据集
In [22]:
1X, y = make_blobs(centers=4, n_samples=1000)
2print(f'Shape of dataset: {X.shape}')
3
4fig = plt.figure(figsize=(8,6))
5plt.scatter(X[:,0], X[:,1], c=y)
6plt.title("Dataset with 4 clusters")
7plt.xlabel("First feature")
8plt.ylabel("Second feature")
9plt.show()
Shape of dataset: (1000, 2)
K均值分类
In [23]:
1class KMeans():
2 def __init__(self, n_clusters=4):
3 self.k = n_clusters
4
5 def fit(self, data):
6 """
7 Fits the k-means model to the given dataset
8 """
9 n_samples, _ = data.shape
10 # initialize cluster centers
11 self.centers = np.array(random.sample(list(data), self.k))
12 self.initial_centers = np.copy(self.centers)
13
14 # We will keep track of whether the assignment of data points
15 # to the clusters has changed. If it stops changing, we are
16 # done fitting the model
17 old_assigns = None
18 n_iters = 0
19
20 while True:
21 new_assigns = [self.classify(datapoint) for datapoint in data]
22
23 if new_assigns == old_assigns:
24 print(f"Training finished after {n_iters} iterations!")
25 return
26
27 old_assigns = new_assigns
28 n_iters += 1
29
30 # recalculate centers
31 for id_ in range(self.k):
32 points_idx = np.where(np.array(new_assigns) == id_)
33 datapoints = data[points_idx]
34 self.centers[id_] = datapoints.mean(axis=0)
35
36 def l2_distance(self, datapoint):
37 dists = np.sqrt(np.sum((self.centers - datapoint)**2, axis=1))
38 return dists
39
40 def classify(self, datapoint):
41 """
42 Given a datapoint, compute the cluster closest to the
43 datapoint. Return the cluster ID of that cluster.
44 """
45 dists = self.l2_distance(datapoint)
46 return np.argmin(dists)
47
48 def plot_clusters(self, data):
49 plt.figure(figsize=(12,10))
50 plt.title("Initial centers in black, final centers in red")
51 plt.scatter(data[:, 0], data[:, 1], marker='.', c=y)
52 plt.scatter(self.centers[:, 0], self.centers[:,1], c='r')
53 plt.scatter(self.initial_centers[:, 0], self.initial_centers[:,1], c='k')
54 plt.show()
初始化并调整模型
1kmeans = KMeans(n_clusters=4)
2kmeans.fit(X)
Training finished after 4 iterations!
描绘初始和最终的聚类中心
1kmeans.plot_clusters(X)
6. 简单的神经网络
在这一章节里，我们将实现一个简单的神经网络架构，将 2 维的输入向量映射成二进制输出值。我们的神经网络有 2 个输入神经元，含 6 个隐藏神经元隐藏层及 1 个输出神经元。
我们将通过层之间的权重矩阵来表示神经网络结构。在下面的例子中，输入层和隐藏层之间的权重矩阵将被表示为，隐藏层和输出层之间的权重矩阵为。除了连接神经元的权重向量外，每个隐藏和输出的神经元都会有一个大小为 1 的偏置量。
我们的训练集由 m = 750 个样本组成。因此，我们的矩阵维度如下：
训练集维度： X = (750，2)
目标维度： Y = (750，1)
维度：(m，nhidden) = (2,6)
维度：(bias vector)：(1，nhidden) = (1,6)
维度： (nhidden，noutput)= (6,1)
维度：(bias vector)：(1，noutput) = (1,1)
损失函数
我们使用与 Logistic 回归算法相同的损失函数：
对于多类别的分类任务，我们将使用这个函数的通用形式作为损失函数，称之为分类交叉熵函数。
训练
我们将用梯度下降法来训练我们的神经网络，并通过反向传播法来计算所需的偏导数。训练过程主要有以下几个步骤：
1. 初始化参数(即权重量和偏差量)
2. 重复以下过程，直到收敛：
通过网络传播当前输入的批次大小，并计算所有隐藏和输出单元的激活值和输出值。
针对每个参数计算其对损失函数的偏导数
更新参数
前向传播过程
首先，我们计算网络中每个单元的激活值和输出值。为了加速这个过程的实现，我们不会单独为每个输入样本执行此操作，而是通过矢量化对所有样本一次性进行处理。其中：
表示对所有训练样本激活隐层单元的矩阵
表示对所有训练样本输出隐层单位的矩阵
隐层神经元将使用 tanh 函数作为其激活函数：
输出层神经元将使用 sigmoid 函数作为激活函数：
激活值和输出值计算如下(·表示点乘)：
反向传播过程
为了计算权重向量的更新值，我们需要计算每个神经元对损失函数的偏导数。这里不会给出这些公式的推导，你会在其他网站上找到很多更好的解释(https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/)。
对于输出神经元，梯度计算如下(矩阵符号)：
对于输入和隐层的权重矩阵，梯度计算如下：
权重更新
In [3]:
1import numpy as np
2import pandas as pd
3import matplotlib.pyplot as plt
4from sklearn.datasets import make_circles
5from sklearn.model_selection import train_test_split
6np.random.seed(123)
7% matplotlib inline
数据集
In [4]:
1X, y = make_circles(n_samples=1000, factor=0.5, noise=.1)
2fig = plt.figure(figsize=(8,6))
3plt.scatter(X[:,0], X[:,1], c=y)
4plt.xlim([-1.5, 1.5])
5plt.ylim([-1.5, 1.5])
6plt.title("Dataset")
7plt.xlabel("First feature")
8plt.ylabel("Second feature")
9plt.show()
In [5]:
1# reshape targets to get column vector with shape (n_samples, 1)
2y_true = y[:, np.newaxis]
3# Split the data into a training and test set
4X_train, X_test, y_train, y_test = train_test_split(X, y_true)
5print(f'Shape X_train: {X_train.shape}')
6print(f'Shape y_train: {y_train.shape}')
7print(f'Shape X_test: {X_test.shape}')
8print(f'Shape y_test: {y_test.shape}')
Shape X_train: (750, 2)
Shape y_train: (750, 1)
Shape X_test: (250, 2)
Shape y_test: (250, 1)
Neural Network Class
以下部分实现受益于吴恩达的课程
https://www.coursera.org/learn/neural-networks-deep-learning
1class NeuralNet():
2 def __init__(self, n_inputs, n_outputs, n_hidden):
3 self.n_inputs = n_inputs
4 self.n_outputs = n_outputs
5 self.hidden = n_hidden
6 # Initialize weight matrices and bias vectors
7 self.W_h = np.random.randn(self.n_inputs, self.hidden)
8 self.b_h = np.zeros((1, self.hidden))
9 self.W_o = np.random.randn(self.hidden, self.n_outputs)
10 self.b_o = np.zeros((1, self.n_outputs))
11 def sigmoid(self, a):
12 return 1 / (1 + np.exp(-a))
13 def forward_pass(self, X):
14 """
15 Propagates the given input X forward through the net.
16 Returns:
17 A_h: matrix with activations of all hidden neurons for all input examples
18 O_h: matrix with outputs of all hidden neurons for all input examples
19 A_o: matrix with activations of all output neurons for all input examples
20 O_o: matrix with outputs of all output neurons for all input examples
21 """
22 # Compute activations and outputs of hidden units
23 A_h = np.dot(X, self.W_h) + self.b_h
24 O_h = np.tanh(A_h)
25 # Compute activations and outputs of output units
26 A_o = np.dot(O_h, self.W_o) + self.b_o
27 O_o = self.sigmoid(A_o)
28 outputs = {
29 "A_h": A_h,
30 "A_o": A_o,
31 "O_h": O_h,
32 "O_o": O_o,
33 }
34 return outputs
35 def cost(self, y_true, y_predict, n_samples):
36 """
37 Computes and returns the cost over all examples
38 """
39 # same cost function as in logistic regression
40 cost = (- 1 / n_samples) * np.sum(y_true * np.log(y_predict) + (1 - y_true) * (np.log(1 - y_predict)))
41 cost = np.squeeze(cost)
42 assert isinstance(cost, float)
43 return cost
44 def backward_pass(self, X, Y, n_samples, outputs):
45 """
46 Propagates the errors backward through the net.
47 Returns:
48 dW_h: partial derivatives of loss function w.r.t hidden weights
49 db_h: partial derivatives of loss function w.r.t hidden bias
50 dW_o: partial derivatives of loss function w.r.t output weights
51 db_o: partial derivatives of loss function w.r.t output bias
52 """
53 dA_o = (outputs["O_o"] - Y)
54 dW_o = (1 / n_samples) * np.dot(outputs["O_h"].T, dA_o)
55 db_o = (1 / n_samples) * np.sum(dA_o)
56 dA_h = (np.dot(dA_o, self.W_o.T)) * (1 - np.power(outputs["O_h"], 2))
57 dW_h = (1 / n_samples) * np.dot(X.T, dA_h)
58 db_h = (1 / n_samples) * np.sum(dA_h)
59 gradients = {
60 "dW_o": dW_o,
61 "db_o": db_o,
62 "dW_h": dW_h,
63 "db_h": db_h,
64 }
65 return gradients
66 def update_weights(self, gradients, eta):
67 """
68 Updates the model parameters using a fixed learning rate
69 """
70 self.W_o = self.W_o - eta * gradients["dW_o"]
71 self.W_h = self.W_h - eta * gradients["dW_h"]
72 self.b_o = self.b_o - eta * gradients["db_o"]
73 self.b_h = self.b_h - eta * gradients["db_h"]
74 def train(self, X, y, n_iters=500, eta=0.3):
75 """
76 Trains the neural net on the given input data
77 """
78 n_samples, _ = X.shape
79 for i in range(n_iters):
80 outputs = self.forward_pass(X)
81 cost = self.cost(y, outputs["O_o"], n_samples=n_samples)
82 gradients = self.backward_pass(X, y, n_samples, outputs)
83 if i % 100 == 0:
84 print(f'Cost at iteration {i}: {np.round(cost, 4)}')
85 self.update_weights(gradients, eta)
86 def predict(self, X):
87 """
88 Computes and returns network predictions for given dataset
89 """
90 outputs = self.forward_pass(X)
91 y_pred = [1 if elem >= 0.5 else 0 for elem in outputs["O_o"]]
92 return np.array(y_pred)[:, np.newaxis]
初始化并训练神经网络
1nn = NeuralNet(n_inputs=2, n_hidden=6, n_outputs=1)
2print("Shape of weight matrices and bias vectors:")
3print(f'W_h shape: {nn.W_h.shape}')
4print(f'b_h shape: {nn.b_h.shape}')
5print(f'W_o shape: {nn.W_o.shape}')
6print(f'b_o shape: {nn.b_o.shape}')
7print()
8print("Training:")
9nn.train(X_train, y_train, n_iters=2000, eta=0.7)
Shape of weight matrices and bias vectors:
W_h shape: (2, 6)
b_h shape: (1, 6)
W_o shape: (6, 1)
b_o shape: (1, 1)
Training:
Cost at iteration 0: 1.0872
Cost at iteration 100: 0.2723
Cost at iteration 200: 0.1712
Cost at iteration 300: 0.1386
Cost at iteration 400: 0.1208
Cost at iteration 500: 0.1084
Cost at iteration 600: 0.0986
Cost at iteration 700: 0.0907
Cost at iteration 800: 0.0841
Cost at iteration 900: 0.0785
Cost at iteration 1000: 0.0739
Cost at iteration 1100: 0.0699
Cost at iteration 1200: 0.0665
Cost at iteration 1300: 0.0635
Cost at iteration 1400: 0.061
Cost at iteration 1500: 0.0587
Cost at iteration 1600: 0.0566
Cost at iteration 1700: 0.0547
Cost at iteration 1800: 0.0531
Cost at iteration 1900: 0.0515
测试神经网络
1n_test_samples, _ = X_test.shape
2y_predict = nn.predict(X_test)
3print(f"Classification accuracy on test set: {(np.sum(y_predict == y_test)/n_test_samples)*100} %")
Classification accuracy on test set: 98.4 %
可视化决策边界
1X_temp, y_temp = make_circles(n_samples=60000, noise=.5)
2y_predict_temp = nn.predict(X_temp)
3y_predict_temp = np.ravel(y_predict_temp)
1fig = plt.figure(figsize=(8,12))
2ax = fig.add_subplot(2,1,1)
3plt.scatter(X[:,0], X[:,1], c=y)
4plt.xlim([-1.5, 1.5])
5plt.ylim([-1.5, 1.5])
6plt.xlabel("First feature")
7plt.ylabel("Second feature")
8plt.title("Training and test set")
9ax = fig.add_subplot(2,1,2)
10plt.scatter(X_temp[:,0], X_temp[:,1], c=y_predict_temp)
11plt.xlim([-1.5, 1.5])
12plt.ylim([-1.5, 1.5])
13plt.xlabel("First feature")
14plt.ylabel("Second feature")
15plt.title("Decision boundary")Out[11]:Text(0.5,1,'Decision boundary')
7. Softmax 回归算法
Softmax 回归算法，又称为多项式或多类别的 Logistic 回归算法。
给定：
数据集
是d-维向量
是对应于
的目标变量，例如对于K=3分类问题，
Softmax 回归模型有以下几个特点：
对于每个类别，都存在一个独立的、实值加权向量
这个权重向量通常作为权重矩阵中的行。
对于每个类别，都存在一个独立的、实值偏置量b
它使用 softmax 函数作为其激活函数
它使用交叉熵( cross-entropy )作为损失函数
训练 Softmax 回归模型有不同步骤。首先(在步骤0中)，模型的参数将被初始化。在达到指定训练次数或参数收敛前，重复以下其他步骤。
第 0 步：用 0 (或小的随机值)来初始化权重向量和偏置值
第 1 步：对于每个类别k，计算其输入的特征与权重值的线性组合，也就是说为每个类别的训练样本计算一个得分值。对于类别k，输入向量为,则得分值的计算如下：
其中表示类别k的权重矩阵，·表示点积。
我们可以通过矢量化和矢量传播法则计算所有类别及其训练样本的得分值：
其中 X 是所有训练样本
第 2 步：用 softmax 函数作为激活函数，将得分值转化为概率值形式。属于类别 k 的输入向量的概率值为：
同样地，我们可以通过矢量化来对所有类别同时处理，得到其概率输出。模型预测出的表示的是该类别的最高概率。
第 3 步：计算整个训练集的损失值。
我们希望模型预测出的高概率值是目标类别，而低概率值表示其他类别。这可以通过以下的交叉熵损失函数来实现：
在上面公式中，目标类别标签表示成独热编码形式( one-hot )。因此为1时表示的目标类别是 k，反之则为 0。
第 4 步：对权重向量和偏置量，计算其对损失函数的梯度。
关于这个导数实现的详细解释，可以参见这里（http://ufldl.stanford.edu/tutorial/supervised/SoftmaxRegression/）。
一般形式如下：
对于偏置量的导数计算，此时为1。
第 5 步：对每个类别k，更新其权重和偏置值。
其中，表示学习率。
In [1]:
1from sklearn.datasets import load_iris
2import numpy as np
3from sklearn.model_selection import train_test_split
4from sklearn.datasets import make_blobs
5import matplotlib.pyplot as plt
6np.random.seed(13)
数据集
In [2]:
1X, y_true = make_blobs(centers=4, n_samples = 5000)
2fig = plt.figure(figsize=(8,6))
3plt.scatter(X[:,0], X[:,1], c=y_true)
4plt.title("Dataset")
5plt.xlabel("First feature")
6plt.ylabel("Second feature")
7plt.show()
In [3]:
1# reshape targets to get column vector with shape (n_samples, 1)
2y_true = y_true[:, np.newaxis]
3# Split the data into a training and test set
4X_train, X_test, y_train, y_test = train_test_split(X, y_true)
5print(f'Shape X_train: {X_train.shape}')
6print(f'Shape y_train: {y_train.shape}')
7print(f'Shape X_test: {X_test.shape}')
8print(f'Shape y_test: {y_test.shape}')
Shape X_train: (3750, 2)
Shape y_train: (3750, 1)
Shape X_test: (1250, 2)
Shape y_test: (1250, 1)
Softmax回归分类
1class SoftmaxRegressor:
2 def __init__(self):
3 pass
4 def train(self, X, y_true, n_classes, n_iters=10, learning_rate=0.1):
5 """
6 Trains a multinomial logistic regression model on given set of training data
7 """
8 self.n_samples, n_features = X.shape
9 self.n_classes = n_classes
10 self.weights = np.random.rand(self.n_classes, n_features)
11 self.bias = np.zeros((1, self.n_classes))
12 all_losses = []
13 for i in range(n_iters):
14 scores = self.compute_scores(X)
15 probs = self.softmax(scores)
16 y_predict = np.argmax(probs, axis=1)[:, np.newaxis]
17 y_one_hot = self.one_hot(y_true)
18 loss = self.cross_entropy(y_one_hot, probs)
19 all_losses.append(loss)
20 dw = (1 / self.n_samples) * np.dot(X.T, (probs - y_one_hot))
21 db = (1 / self.n_samples) * np.sum(probs - y_one_hot, axis=0)
22 self.weights = self.weights - learning_rate * dw.T
23 self.bias = self.bias - learning_rate * db
24 if i % 100 == 0:
25 print(f'Iteration number: {i}, loss: {np.round(loss, 4)}')
26 return self.weights, self.bias, all_losses
27 def predict(self, X):
28 """
29 Predict class labels for samples in X.
30 Args:
31 X: numpy array of shape (n_samples, n_features)
32 Returns:
33 numpy array of shape (n_samples, 1) with predicted classes
34 """
35 scores = self.compute_scores(X)
36 probs = self.softmax(scores)
37 return np.argmax(probs, axis=1)[:, np.newaxis]
38 def softmax(self, scores):
39 """
40 Tranforms matrix of predicted scores to matrix of probabilities
41 Args:
42 scores: numpy array of shape (n_samples, n_classes)
43 with unnormalized scores
44 Returns:
45 softmax: numpy array of shape (n_samples, n_classes)
46 with probabilities
47 """
48 exp = np.exp(scores)
49 sum_exp = np.sum(np.exp(scores), axis=1, keepdims=True)
50 softmax = exp / sum_exp
51 return softmax
52 def compute_scores(self, X):
53 """
54 Computes class-scores for samples in X
55 Args:
56 X: numpy array of shape (n_samples, n_features)
57 Returns:
58 scores: numpy array of shape (n_samples, n_classes)
59 """
60 return np.dot(X, self.weights.T) + self.bias
61 def cross_entropy(self, y_true, scores):
62 loss = - (1 / self.n_samples) * np.sum(y_true * np.log(scores))
63 return loss
64 def one_hot(self, y):
65 """
66 Tranforms vector y of labels to one-hot encoded matrix
67 """
68 one_hot = np.zeros((self.n_samples, self.n_classes))
69 one_hot[np.arange(self.n_samples), y.T] = 1
70 return one_hot
初始化并训练模型
1regressor = SoftmaxRegressor()
2w_trained, b_trained, loss = regressor.train(X_train, y_train, learning_rate=0.1, n_iters=800, n_classes=4)
3fig = plt.figure(figsize=(8,6))
4plt.plot(np.arange(800), loss)
5plt.title("Development of loss during training")
6plt.xlabel("Number of iterations")
7plt.ylabel("Loss")
8plt.show()Iteration number: 0, loss: 1.393Iteration number: 100, loss: 0.2051
Iteration number: 200, loss: 0.1605
Iteration number: 300, loss: 0.1371
Iteration number: 400, loss: 0.121
Iteration number: 500, loss: 0.1087
Iteration number: 600, loss: 0.0989
Iteration number: 700, loss: 0.0909
测试模型
1n_test_samples, _ = X_test.shape
2y_predict = regressor.predict(X_test)
3print(f"Classification accuracy on test set: {(np.sum(y_predict == y_test)/n_test_samples) * 100}%")
测试集分类准确率：99.03999999999999%
原文链接：
https://github.com/zotroneneis/machine_learning_basics