ML-1 逻辑回归和梯度下降

Author：吾爱北方的母老虎

原创链接：https://mp.csdn.net/postedit/80244886

Logistic Regression

The data

我们将建立一个逻辑回归模型来预测一个学生是否被大学录取。假设你是一个大学系的管理员，你想根据两次考试的结果来决定每个申请人的录取机会。你有以前的申请人的历史数据，你可以用它作为逻辑回归的训练集。对于每一个培训例子，你有两个考试的申请人的分数和录取决定。为了做到这一点，我们将建立一个分类模型，根据考试成绩估计入学概率。

In [3]:

#三大件

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

%matplotlib inline

In [11]:

import os

path = 'data' + os.sep + 'LogiReg_data.txt'

# 自己指定csv开头的标题是什么

pdData = pd.read_csv(path, header=None, names=['Exam 1', 'Exam 2', 'Admitted'])

pdData.head()

# pdData

Out[11]:

	Exam 1	Exam 2	Admitted
0	34.623660	78.024693	0
1	30.286711	43.894998	0
2	35.847409	72.902198	0
3	60.182599	86.308552	1
4	79.032736	75.344376	1

In [9]:

pdData.shape

Out[9]:

(100, 3)

对数据的一个可视化，先对数据有一个直观的感受

In [18]:

# 标签为1 的标记为正例，标签为0的标记为负例

positive = pdData[pdData['Admitted'] == 1] # returns the subset of rows such Admitted = 1, i.e. the set of *positive* examples

negative = pdData[pdData['Admitted'] == 0] # returns the subset of rows such Admitted = 0, i.e. the set of *negative* examples

​

fig, ax = plt.subplots(figsize=(10,5))    #  创建一个绘图区域 

# help(plt.subplots)  

print(fig)

print(ax)

ax.scatter(positive['Exam 1'], positive['Exam 2'], s=30, c='b', marker='o', label='Admitted')

ax.scatter(negative['Exam 1'], negative['Exam 2'], s=30, c='r', marker='x', label='Not Admitted')

ax.legend()  #  这个可以显示右上交的小标志

ax.set_xlabel('Exam 1 Score')

ax.set_ylabel('Exam 2 Score')

Figure(720x360)
AxesSubplot(0.125,0.125;0.775x0.755)

Out[18]:

Text(0,0.5,'Exam 2 Score')

The logistic regression

目标：建立分类器（求解出三个参数 θ0θ1θ2）

设定阈值，根据阈值判断录取结果

要完成的模块

• sigmoid : 映射到概率的函数

• model : 返回预测结果值

• cost : 根据参数计算损失

• gradient : 计算每个参数的梯度方向

• descent : 进行参数更新

• accuracy: 计算精度

sigmoid 函数

g(z)=11+e−z

In [19]:

def sigmoid(z):

    return 1 / (1 + np.exp(-z))

In [21]:

nums = np.arange(-10, 10, step=1) #creates a vector containing 20 equally spaced values from -10 to 10

fig, ax = plt.subplots(figsize=(12,4))   # 288*3  = 864  

print(fig)  

print(ax)

ax.plot(nums, sigmoid(nums), 'r')   # 给出了横坐标和纵坐标的数据

Figure(864x288)
AxesSubplot(0.125,0.125;0.775x0.755)

Out[21]:

[<matplotlib.lines.Line2D at 0x10b8eff60>]

Sigmoid

• g:ℝ→[0,1]
• g(0)=0.5
• g(−∞)=0
• g(+∞)=1

In [27]:

# 这里定义一个假设函数

​

def model(X, theta):

    return sigmoid(np.dot(X, theta.T))   # 把Z向量化的式子带入   两个矩阵的相乘

(θ0θ1θ2)×1x1x2=θ0+θ1x1+θ2x2

In [28]:

​

pdData.insert(0, 'Ones', 1) # in a try / except structure so as not to return an error if the block si executed several times

​

​

# set X (training data) and y (target variable)

orig_data = pdData.as_matrix() # convert the Pandas representation of the data to an array useful for further computations

cols = orig_data.shape[1]

X = orig_data[:,0:cols-1]

y = orig_data[:,cols-1:cols]

​

# convert to numpy arrays and initalize the parameter array theta

#X = np.matrix(X.values)

#y = np.matrix(data.iloc[:,3:4].values) #np.array(y.values)

theta = np.zeros([1, 3])

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-28-22ac317a1267> in <module>()
      1 
----> 2 pdData.insert(0, 'Ones', 1) # in a try / except structure so as not to return an error if the block si executed several times
      3 
      4 
      5 # set X (training data) and y (target variable)

~/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in insert(self, loc, column, value, allow_duplicates)
   2421         value = self._sanitize_column(column, value, broadcast=False)
   2422         self._data.insert(loc, column, value,
-> 2423                           allow_duplicates=allow_duplicates)
   2424 
   2425     def assign(self, **kwargs):

~/anaconda3/lib/python3.6/site-packages/pandas/core/internals.py in insert(self, loc, item, value, allow_duplicates)
   3808         if not allow_duplicates and item in self.items:
   3809             # Should this be a different kind of error??
-> 3810             raise ValueError('cannot insert {}, already exists'.format(item))
   3811 
   3812         if not isinstance(loc, int):

ValueError: cannot insert Ones, already exists

In [36]:

X[:5]

# print(X)

Out[36]:

array([[ 1.        , 34.62365962, 78.02469282],
       [ 1.        , 30.28671077, 43.89499752],
       [ 1.        , 35.84740877, 72.90219803],
       [ 1.        , 60.18259939, 86.3085521 ],
       [ 1.        , 79.03273605, 75.34437644]])

In [30]:

y[:5]

Out[30]:

array([[0.],
       [0.],
       [0.],
       [1.],
       [1.]])

In [31]:

theta

Out[31]:

array([[0., 0., 0.]])

In [32]:

X.shape, y.shape, theta.shape

Out[32]:

((100, 3), (100, 1), (1, 3))

损失函数

将对数似然函数去负号

D(hθ(x),y)=−ylog(hθ(x))−(1−y)log(1−hθ(x))

求平均损失

J(θ)=1n∑i=1nD(hθ(xi),yi)

In [33]:

# 定义损失函数  从函数式中可以看出需要三个参数

def cost(X, y, theta):

    left = np.multiply(-y, np.log(model(X, theta)))

    right = np.multiply(1 - y, np.log(1 - model(X, theta)))

    return np.sum(left - right) / (len(X))

In [34]:

cost(X, y, theta)

Out[34]:

0.6931471805599453

计算梯度

∂J∂θj=−1m∑i=1n(yi−hθ(xi))xij

In [37]:

def gradient(X, y, theta):

    grad = np.zeros(theta.shape)

    error = (model(X, theta)- y).ravel()   #  定义了error

    for j in range(len(theta.ravel())): #for each parmeter      reval的作用是将多维数组变成一维

        term = np.multiply(error, X[:,j])

        grad[0, j] = np.sum(term) / len(X)

    return grad

Gradient descent

比较3中不同梯度下降方法

In [48]:

STOP_ITER = 0

STOP_COST = 1

STOP_GRAD = 2

​

def stopCriterion(type, value, threshold):

    #设定三种不同的停止策略

    if type == STOP_ITER:        return value > threshold

    elif type == STOP_COST:      return abs(value[-1]-value[-2]) < threshold    # 阈值

    elif type == STOP_GRAD:      return np.linalg.norm(value) < threshold

In [49]:

import numpy.random

#洗牌

def shuffleData(data):

    np.random.shuffle(data)   # 将原始收的书序打乱，这样数据的分布会更均匀

    cols = data.shape[1]   #

    X = data[:, 0:cols-1]   # 去掉数据的列

    y = data[:, cols-1:]

    return X, y

# print(X)

print(orig_data.shape)

print(orig_data[:5])   #  原始数据data

(100, 4)
[[ 1.         34.62365962 78.02469282  0.        ]
 [ 1.         30.28671077 43.89499752  0.        ]
 [ 1.         35.84740877 72.90219803  0.        ]
 [ 1.         60.18259939 86.3085521   1.        ]
 [ 1.         79.03273605 75.34437644  1.        ]]

In [52]:

import time

​

def descent(data, theta, batchSize, stopType, thresh, alpha):

    #梯度下降求解

    init_time = time.time()

    i = 0 # 迭代次数

    k = 0 # batch

    X, y = shuffleData(data)

    grad = np.zeros(theta.shape) # 计算的梯度

    costs = [cost(X, y, theta)] # 损失值    然后对损失值进行求导，进行梯度下降

​

    while True:

        grad = gradient(X[k:k+batchSize], y[k:k+batchSize], theta)

        k += batchSize #取batch数量个数据

        if k >= n: 

            k = 0 

            X, y = shuffleData(data) #重新洗牌

        theta = theta - alpha*grad # 参数更新

        costs.append(cost(X, y, theta)) # 计算新的损失

        i += 1 

​

        if stopType == STOP_ITER:       value = i

        elif stopType == STOP_COST:     value = costs

        elif stopType == STOP_GRAD:     value = grad

        if stopCriterion(stopType, value, thresh): break

    return theta, i-1, costs, grad, time.time() - init_time

In [53]:

# 主要是实现对数据的一个可视化

​

def runExpe(data, theta, batchSize, stopType, thresh, alpha):

    #import pdb; pdb.set_trace();

    theta, iter, costs, grad, dur = descent(data, theta, batchSize, stopType, thresh, alpha)

    name = "Original" if (data[:,1]>2).sum() > 1 else "Scaled"

    name += " data - learning rate: {} - ".format(alpha)

    if batchSize==n: strDescType = "Gradient"

    elif batchSize==1:  strDescType = "Stochastic"

    else: strDescType = "Mini-batch ({})".format(batchSize)

    name += strDescType + " descent - Stop: "

    if stopType == STOP_ITER: strStop = "{} iterations".format(thresh)

    elif stopType == STOP_COST: strStop = "costs change < {}".format(thresh)

    else: strStop = "gradient norm < {}".format(thresh)

    name += strStop

    print ("***{}\nTheta: {} - Iter: {} - Last cost: {:03.2f} - Duration: {:03.2f}s".format(

        name, theta, iter, costs[-1], dur))

    fig, ax = plt.subplots(figsize=(12,4))

    ax.plot(np.arange(len(costs)), costs, 'r')

    ax.set_xlabel('Iterations')

    ax.set_ylabel('Cost')

    ax.set_title(name.upper() + ' - Error vs. Iteration')

    return theta   

不同的停止策略

设定迭代次数

In [54]:

#选择的梯度下降方法是基于所有样本的

n=100    #  选择迭代此时作物停止的阈值

runExpe(orig_data, theta, n, STOP_ITER, thresh=5000, alpha=0.000001)

***Original data - learning rate: 1e-06 - Gradient descent - Stop: 5000 iterations
Theta: [[-0.00027127  0.00705232  0.00376711]] - Iter: 5000 - Last cost: 0.63 - Duration: 0.98s

Out[54]:

array([[-0.00027127,  0.00705232,  0.00376711]])

根据损失值停止

设定阈值 1E-6, 差不多需要110 000次迭代 

In [55]:

# 选择损失作为迭代停止的条件

runExpe(orig_data, theta, n, STOP_COST, thresh=0.000001, alpha=0.001)

***Original data - learning rate: 0.001 - Gradient descent - Stop: costs change < 1e-06
Theta: [[-5.13364014  0.04771429  0.04072397]] - Iter: 109901 - Last cost: 0.38 - Duration: 26.31s

Out[55]:

array([[-5.13364014,  0.04771429,  0.04072397]])

根据梯度变化停止

设定阈值 0.05,差不多需要40 000次迭代

In [56]:

runExpe(orig_data, theta, n, STOP_GRAD, thresh=0.05, alpha=0.001)

***Original data - learning rate: 0.001 - Gradient descent - Stop: gradient norm < 0.05
Theta: [[-2.37033409  0.02721692  0.01899456]] - Iter: 40045 - Last cost: 0.49 - Duration: 9.91s

Out[56]:

array([[-2.37033409,  0.02721692,  0.01899456]])

对比不同的梯度下降方法

Stochastic descent

In [57]:

runExpe(orig_data, theta, 1, STOP_ITER, thresh=5000, alpha=0.001)

***Original data - learning rate: 0.001 - Stochastic descent - Stop: 5000 iterations
Theta: [[-0.37489917 -0.02291539 -0.01156707]] - Iter: 5000 - Last cost: 1.84 - Duration: 0.31s

Out[57]:

array([[-0.37489917, -0.02291539, -0.01156707]])

有点爆炸。。。很不稳定,再来试试把学习率调小一些

In [58]:

# batchsize = 1  这里选用的下降方法是用随机梯度进行下降做到的

runExpe(orig_data, theta, 1, STOP_ITER, thresh=15000, alpha=0.000002)

***Original data - learning rate: 2e-06 - Stochastic descent - Stop: 15000 iterations
Theta: [[-0.00202068  0.01005956  0.0010061 ]] - Iter: 15000 - Last cost: 0.63 - Duration: 1.09s

Out[58]:

array([[-0.00202068,  0.01005956,  0.0010061 ]])

速度快，但稳定性差，需要很小的学习率

Mini-batch descent

In [61]:

# batch-size = 16   每次梯度下降的时候选用的数据是16个（样本的数量）

runExpe(orig_data, theta, 16, STOP_ITER, thresh=15000, alpha=0.001)

***Original data - learning rate: 0.001 - Mini-batch (16) descent - Stop: 15000 iterations
Theta: [[-1.0329506   0.02385259  0.01382789]] - Iter: 15000 - Last cost: 0.62 - Duration: 1.48s

Out[61]:

array([[-1.0329506 ,  0.02385259,  0.01382789]])

浮动仍然比较大，我们来尝试下对数据进行标准化 将数据按其属性(按列进行)减去其均值，然后除以其方差。最后得到的结果是，对每个属性/每列来说所有数据都聚集在0附近，方差值为1

In [63]:

from sklearn import preprocessing as pp

​

​

# 对数据进行了预处理，就会说数据归一化，这里可以看出，效果得到了明显的提升

scaled_data = orig_data.copy()

scaled_data[:, 1:3] = pp.scale(orig_data[:, 1:3])

print(orig_data[:5])

print("______________________")

print(scaled_data[:5])

​

runExpe(scaled_data, theta, n, STOP_ITER, thresh=5000, alpha=0.001)

[[ 1.         64.17698887 80.90806059  1.        ]
 [ 1.         40.23689374 71.16774802  0.        ]
 [ 1.         94.09433113 77.15910509  1.        ]
 [ 1.         80.366756   90.9601479   1.        ]
 [ 1.         69.36458876 97.71869196  1.        ]]
______________________
[[ 1.         -0.07578684  0.7942862   1.        ]
 [ 1.         -1.31231814  0.26748769  0.        ]
 [ 1.          1.46947562  0.59152637  1.        ]
 [ 1.          0.76043181  1.33794685  1.        ]
 [ 1.          0.1921582   1.70347834  1.        ]]
***Scaled data - learning rate: 0.001 - Gradient descent - Stop: 5000 iterations
Theta: [[0.3080807  0.86494967 0.77367651]] - Iter: 5000 - Last cost: 0.38 - Duration: 1.25s

Out[63]:

array([[0.3080807 , 0.86494967, 0.77367651]])

它好多了！原始数据，只能达到达到0.61，而我们得到了0.38个在这里！ 所以对数据做预处理是非常重要的

In [64]:

runExpe(scaled_data, theta, n, STOP_GRAD, thresh=0.02, alpha=0.001)

***Scaled data - learning rate: 0.001 - Gradient descent - Stop: gradient norm < 0.02
Theta: [[1.0707921  2.63030842 2.41079787]] - Iter: 59422 - Last cost: 0.22 - Duration: 15.38s

Out[64]:

array([[1.0707921 , 2.63030842, 2.41079787]])

更多的迭代次数会使得损失下降的更多！

In [35]:

theta = runExpe(scaled_data, theta, 1, STOP_GRAD, thresh=0.002/5, alpha=0.001)

***Scaled data - learning rate: 0.001 - Stochastic descent - Stop: gradient norm < 0.0004
Theta: [[ 1.14848169  2.79268789  2.5667383 ]] - Iter: 72637 - Last cost: 0.22 - Duration: 7.05s

随机梯度下降更快，但是我们需要迭代的次数也需要更多，所以还是用batch的比较合适！！！

In [65]:

runExpe(scaled_data, theta, 16, STOP_GRAD, thresh=0.002*2, alpha=0.001)

***Scaled data - learning rate: 0.001 - Mini-batch (16) descent - Stop: gradient norm < 0.004
Theta: [[0.99833532 2.48373387 2.27860621]] - Iter: 49879 - Last cost: 0.22 - Duration: 5.74s

Out[65]:

array([[0.99833532, 2.48373387, 2.27860621]])

精度

In [66]:

#设定阈值

def predict(X, theta):

    return [1 if x >= 0.5 else 0 for x in model(X, theta)]

In [68]:

scaled_X = scaled_data[:, :3]

y = scaled_data[:, 3]

predictions = predict(scaled_X, theta)

correct = [1 if ((a == 1 and b == 1) or (a == 0 and b == 0)) else 0 for (a, b) in zip(predictions, y)]

accuracy = (sum(map(int, correct)) % len(correct))

print ('accuracy = {0}%'.format(accuracy))

accuracy = 60%

在 python2 中zip可以将两个列表并入一个元组列表，如：

a = [1,2,3,4]

b = [5,6,7,8]

c = zip(a,b)

结果：c 

[(1,5),(2,6),(3,7),(4,8)]

In [ ]:

​