Machine Learning - Training a Model

Enterprise 2023-06-05 13:32:14 views: null

Machine Learning - Training a Model

linear regression

y ^ = h Θ ( x ) = Θ ∗ x where Θ is the parameter vector of the model, which includes the bias term Θ 0 and the feature weights Θ 1 to Θ nx are the feature vectors of the instance, including from x 0 to xn , x 0 Always 0 Θ ∗ x is the vector Θ and x dot product h Θ is the hypothesis function, using model parameters Θ \widehat{y} = h_{\Theta}(x) = \Theta*x \\where \Theta is the model's Parameter vector, including bias item \Theta_0 and feature weight \Theta_1 to \Theta_n \\x is the feature vector of the instance, including from x_0 to x_n, x_0 is always 0 \\ \Theta*x is the vector \Theta and x point The product \\h_{\Theta} is the hypothesis function, using the model parameter \Thetay ​=hTh(x)=Th∗xwhere Θ is the parameter vector of the model, which includes the bias term Θ0​and feature weights Θ1​to Θn​x is the eigenvector of the instance, including0​to xn​,x0​always 0Th∗x is the vector Θ dot product with xhThis the hypothetical function, using the model parameter Θ

MSE cost function for linear regression model

M S E = ( X , h Θ ) = 1 / m ∑ i = 1 m ( Θ T x ( i ) − y ( I ) ) 2 MSE = (X,h_{\Theta}) = 1/m\sum_{i = 1}^{m}(\Theta^Tx^{(i)}-y^{(I)})^2 MSE=(X,hTh)=1/mi=1∑m​( ThTx(i)−y(I))2

standard equation

Θ ^ = ( XTX ) − 1 XT y In the equation, Θ ^ is the Θ value that minimizes the cost function y is the target value vector \widehat{\Theta}=(X^TX) including y ( 1 ) to y ( m ) ^{-1}X^Ty \\In the equation, \widehat{\Theta} is the \Theta value that minimizes the cost function\\y is the target value including y^{(1)} to y^{(m)} vectorTh =(XTX)−1XTyin the equationTh is the value of Θ that minimizes the cost functiony is including y( 1 ) toy( m ) target value vector

gradient descent

batch gradient descent


Partial derivative of $$ cost function:

\ \frac{ \partial MSE(\theta)}{\partial\theta_j} = 2/m\sum_{i = 1}{m}(\ThetaTx{(i)-y{i}})x_{j}^{(i)}
$$

Form: ∇ Θ MSE ( Θ ) = 2 / m XT ( X Θ − y ) Form: \nabla_{\Theta}MSE(\Theta) = 2/mX^T(X\Theta-y) .Gradient vector: ∇ThMSE(Θ)=2/mXT(XΘ−y)

Gradient descent step: Θ next step = Θ − η ∇ Θ MSE ( Θ ) ( η is the learning rate) Gradient descent step: \Theta^{next step} = \Theta-\eta\nabla_{\Theta}MSE(\Theta )\ \ \ \ \ (\eta is the learning rate)Gradient descent step: ΘNext step=Th−η∇ThMSE ( Θ ) ( η is the learning rate )     

eta = 0.1
n_iteration = 1000#迭代次数
m = 100

theta = np.random.randn(2,1)
for n in range(n_iteration):
    gradienters = 2/m*x_b.T.dot(x_b.dot(theta)-y)#偏导计算
    theta = theta-eta*gradienters #更改theta

stochastic gradient descent

'''随机梯度下降'''
n_epochs = 50#向前或向后迭代次数

t0,t1 = 5,50 #学习步长超参数

def learning_schedule(t):#计算步长
    return t0/(t+t1)#步长逐渐减小

theta = np.random.randn(2,1)
for n in range(n_epochs):
    for i in range(m):
        random_index = np.random.randint(m)
        xi = x_b[random_index:random_index+1]
        yi = y[random_index:random_index+1]
        gradienters = 2*xi.T.dot(xi.dot(theta)-yi)
        eta = learning_schedule(n*m+i)
        theta = theta-eta*gradienters

sklearn implements stochastic gradient descent

sgd_reg = SGDRegressor(max_iter=1000,tol = 1e-3,penalty=None,eta0=0.1)
sgd_reg.fit(x,y.ravel())
print(sgd_reg.intercept_,sgd_reg.coef_)

Mini-batch gradient descent

'''小批量梯度下降'''
theta_path_mgd = []

n_iterations = 50
minibatch_size = 20

np.random.seed(42)
theta = np.random.randn(2,1)  # random initialization

t0, t1 = 200, 1000
def learning_schedule(t):
    return t0 / (t + t1)

t = 0
for epoch in range(n_iterations):
    shuffled_indices = np.random.permutation(m)
    x_b_shuffled = x_b[shuffled_indices]
    y_shuffled = y[shuffled_indices]
    for i in range(0, m, minibatch_size):
        t += 1
        xi = x_b_shuffled[i:i+minibatch_size]
        yi = y_shuffled[i:i+minibatch_size]
        gradients = 2/minibatch_size * xi.T.dot(xi.dot(theta) - yi)
        eta = learning_schedule(t)
        theta = theta - eta * gradients
        theta_path_mgd.append(theta)

polynomial regression

'''多项式回归'''
m = 100
x = 6*np.random.rand(m,1)-3
y = 0.5*x**2+x+2+np.random.randn(m,1)

poly_features = PolynomialFeatures(degree=2,include_bias=False)#聚类特征
x_poly = poly_features.fit_transform(x)

lin_reg = LinearRegression()
lin_reg.fit(x_poly,y)
#画图
X_new=np.linspace(-3, 3, 100).reshape(100, 1)
X_new_poly = poly_features.transform(X_new)
y_new = lin_reg.predict(X_new_poly)
plt.plot(x, y, "b.")
plt.plot(X_new, y_new, "r-", linewidth=2, label="Predictions")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.legend(loc="upper left", fontsize=14)
plt.axis([-3, 3, 0, 10])

plt.show()

PolynomialFeatures can also add all combinations of features to a given polynomial order

regularized linear model

Ridge regression

Cost function
J ( Θ ) = MSE ( Θ ) + α 1 2 ∑ i = 1 n Θ i 2 The hyperparameter α controls how much to regularize the model. If α = 0 , ridge regression is only linear regression. If α is very large, all weights end up very close to zero, and the result is a flat line through the data mean J(\Theta) = MSE(\Theta)+\alpha\frac{1}{2}\sum_{i =1}^{n}\Theta_i^2 \\hyperparameter\alpha controls how much to regularize the model. If \alpha=0, ridge regression is linear regression only. \\If \alpha is very large, all weights end up very close to zero, and the result is a flat line through the mean of the dataJ(Θ)=MSE(Θ)+a21​i=1∑n​Thi2​The hyperparameter α controls how much to regularize the model. if α=0 , then ridge regression is just linear regression.If α is very large, all weights end up very close to zero, and the result is a flat line through the mean of the data Ridge
regression with closed-form solution:
Θ ^ = ( XTX + α A ) − 1 XT y \widehat{\Theta} = (X^TX+\alpha A)^{-1}X^TyTh =(XTX+αA)−1XTy

from sklearn.linear_model import Ridge
np.random.seed(42)
m = 20
X = 3 * np.random.rand(m, 1)
y = 1 + 0.5 * X + np.random.randn(m, 1) / 1.5
X_new = np.linspace(0, 3, 100).reshape(100, 1)

ridge_reg = Ridge(alpha=1, solver="cholesky", random_state=42)
ridge_reg.fit(X, y)
ridge_reg.predict([[1.5]])
ridge_reg = Ridge(alpha=1, solver="sag", random_state=42)
ridge_reg.fit(X, y)
ridge_reg.predict([[1.5]])

Lasso returns

Least Absolute Shrinkage and Selection Operator Regression

Form:
J ( Θ ) = MSE ( Θ ) + α ∑ i = 1 n ∣ Θ i ∣ J(\Theta) = MSE(\Theta)+\alpha\sum_{i=1}^{n}|\; Theta_i|J(Θ)=MSE(Θ)+ai=1∑n​∣Θi​∣
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-jufNfNa1-1658889211621)(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAj4AAAEYCAYAAABcL/waAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABUKklEQVR4nO3dd3hUVfrA8e9JgRQIkABBSghVepBEKaIGISKsiOu69oKrlLVXLKhgxYK4a0HlJwoqCurqYqEIaBQkIEFJKIIrHSQhQAgQAkkm5/fHSSWTOnf6+3meeZLcuXPvO5PJm3fOPUVprRFCCCGE8AcB7g5ACCGEEMJVpPARQgghhN+QwkcIIYQQfkMKHyGEEEL4DSl8hBBCCOE3pPARQgghhN+QwkcIIYQQfsMphY9SqotS6qRS6sNq9rlXKZWhlMpRSr2rlGrojFiEEL5B8ooQwgrOavF5A1hb1Z1KqeHAw8BQIBboCDzppFiEEL5B8ooQwmGWFz5KqauBI8Dyana7CZiltd6ktc4GngbGWB2LEMI3SF4RQlglyMqDKaUigKcwn7huqWbXnsCCcj+nAdFKqSit9SE7xx0HjAMIDw+P79atm3VBCyGcat26dQe11i3q+3hn5BXJKUJ4L0dziqWFD+YT1iyt9R6lVHX7NQJyyv1c8n1joFLho7WeCcwESEhI0KmpqdZEK4RwOqXULgcPYXlekZwihPdyNKdYVvgopfoCw4CzarH7cSCi3M8l3x+zKh4hhPeTvCKEsJqVLT6JmA6Fu4s/lTUCApVSPbTW/U7bdxMQB3xS/HMckGnvMpcQwq8lInlFCGEhKzs3zwQ6AX2Lb28B3wDD7ez7PnCLUqqHUqoZ8Bgw28JYhBC+QfKKEMJSlhU+WusTWuuMkhum2fmk1jpLKRWjlDqulIop3ncx8CLwPbCr+DbZqliEEL5B8ooQwmpWd24upbWeUu773Zgm6vL3TwemO+v8QgjfI3lFCOEoWbJCCCGEEH7DaS0+7nL06FEOHDhAQUGBu0MRdgQHB9OyZUsiIiJq3lkIDyF5RVRH8pp38anC5+jRo2RmZtKmTRtCQ0OpYc4P4WJaa/Ly8ti3bx+AJAnhFSSviOpIXvM+PnWp68CBA7Rp04awsDBJTh5IKUVYWBht2rThwIED7g5HiFqRvCKqI3nN+/hU4VNQUEBoaKi7wxA1CA0NlUsGwmtIXhG1IXnNe/hU4QPIJzIvIL8j4W3kPStqIu8R7+FzhY8QQgghRFWk8BFCCCGE35DCRwghhBB+QwofAcDmzZsZOnQoYWFhtG7dmieeeAKbzebusIQQXkzyivBEPjWPj6if7Oxshg0bRo8ePViwYAHbtm3j/vvvp6ioiGeeecbd4QkhvJDkFeGppPARvPXWW+Tl5fH5558TERFBUlISR48eZcqUKUycOFEm5BJC1JnkFeGp5FKXhzp8+DDjx4+nefPmREZG8tBDDwEwePBgXnvtNUvPtWjRIoYPH14hEV199dXk5eXxww8/WHouIYT7SF4RQlp8PNKRI0cYNGgQjRo14t1332Xjxo1MmjSJ2NhYtm3bxtixYyvsr7Wu1XXzoCD7v+4tW7Zw4YUXVtgWExNDWFgYW7ZsYdSoUfV/MkIIjyB5RQhDWnw80LPPPktGRgaLFy/m0ksv5dFHH6V58+Y88sgjPPDAA4SEhFTYf86cOQQHB9d4q0p2djZNmzattL1Zs2ZkZ2db/fSEEG4geUUIwy9afNw5oabWdd1fM2fOHMaOHUvz5s1Ltzdp0oScnBwmTJhQ6TGjRo1i7dq1DsVpb9ZRrbXMRipEFSSv1EzyivBEflH41DVJuNPWrVvJysoiKSmpwnabzcbdd99NeHh4pcdERkbSpEmTep+zWbNmHDlypNL2nJwcu5/YhBCSV2oieUV4KrnU5WF27NgBQPv27Uu3rVq1ip07d9K3b1+7j3G0Sbpbt25s2bKlwrY9e/aQm5tLt27dHH9SQgi3krwiRBnLW3yUUh8CQ4FwIAN4UWv9jp39xgCzgLxymy/RWidbHZM3CQwMBMzoCzDNwhMnTiz93h5Hm6RHjBjBSy+9xLFjx2jcuDEA8+fPJzQ0lAsuuKDexxXCCpJTHCd5RYhytNaW3oCeQMPi77thElW8nf3GACvrevz4+Hhdlc2bN1d5n7c4ePCgDgkJ0YMHD9aLFi3SY8aM0d27d9ddunTR1157rd6xY4fl5zx8+LBu1aqVHjZsmF66dKl+++23dXh4uJ40aZLl5yrhC78rUTtAqvbSnKK1b7xX/SWvuJsvvFe8gaM5xfJLXVrrTVrrUyU/Ft86WX0eXxUVFcWcOXPYu3cvl156Kenp6XzzzTdMmjSJBQsW8Mknn1h+zmbNmrF8+XJsNhujRo1i8uTJ3HvvvTz55JOWn0uIupKc4jjJK0KUUdoJPfSUUjMwn75CgV+B87XWx0/bZwzwBqZZ+jDwATBVa11o53jjgHEAMTEx8bt27bJ73t9++43u3btb9jyE88jvyn8opdZprRMcPIZbcgrIe1XUnrxXXMPRnOKUzs1a69uAxsB5wOfAKTu7/Qj0AloCfwOuAR6s4ngztdYJWuuEFi1aOCNkIYQHk5wihLCK00Z1aa1tWuuVQFvgn3bu36613qG1LtJabwCeAq5wVjxCCO8mOUUIYQVXDGcPonbX4zUgs1oJIWoiOUUIUW+WFj5KqZZKqauVUo2UUoFKqeGY5ubv7Ow7QikVXfx9N+BxYIGV8QghvJvkFCGE1axu8dGYJui9QDYwDbhHa71AKRWjlDqulIop3ncokK6UygUWYq7bP2dxPEII7yY5RQhhKUsnMNRaZwF2Z6bSWu8GGpX7+QHgASvPL4TwLZJThBBWkyUrhBBCCOE3pPARQgghhN+QwkcIIYQQfkMKHyGEEEL4DSl8BACbN29m6NChhIWF0bp1a5544glsNlu1j/njjz8YP348cXFxBAYGkpiY6JpghRAez1PzQ21yXWJiIkopu7eUlBQ3RS6sYumoLuGdsrOzGTZsGD169GDBggVs27aN+++/n6KiIp555pkqH7dp0yYWLlzIgAEDyM/Pd2HEQghP54n5oba5bsaMGRw9erTCY5944gl+/fVXzj77bFeHLSwmhY/grbfeIi8vj88//5yIiAiSkpI4evQoU6ZMYeLEiURERNh93KhRoxg9ejQAV1xxBQcPHnRl2EIID+aJ+aG2ua5Hjx4VHpefn09qaipXXXUVQUHyb9Ndftz1Iy+tesnh48ilLg91+PBhxo8fT/PmzYmMjOShhx4CYPDgwbz22muWnmvRokUMHz68QoFz9dVXk5eXxw8//FDl4wIC5O0jhDdxZV6pbX4oLCzkpZdeonPnzoSGhpKQkMDKlSstjaVEfXPd4sWLyc7O5pprrnFKXKJ2vtvxHV///rXDx5HS1QMdOXKEQYMG0ahRI9599102btzIpEmTiI2NZdu2bYwdO7bC/lrrGvvjAFV+UtmyZQsXXnhhhW0xMTGEhYWxZcsWRo0aVf8nI4TwCK7OK7VRWFjIxRdfzObNm5k8eTIdO3bknXfeYeTIkfzvf/8jOjq63se2p765bt68ebRp04bzzjvP0nhE3fy05ydLjuMXhY96sup1Ct++5G3GxY8DYOa6mYz/enyV++rJuvT7+Jnx/LL/F7v7je03lpmjZtYzWnj22WfJyMjgjz/+oHnz5lx66aW88sorPPLIIzz++OOEhIRU2H/OnDncfPPNNR5Xa213e3Z2Nk2bNq20vVmzZmRnZ9frOQjh6ySvFMdfRV6pjZdffpk1a9awbt06unbtCpiOxTExMXz22Wfcfvvt9T62PfXJdSdOnOCrr75i3LhxKCVr3rqLrcjG6r2rLTmWXxQ+3kRrzZw5cxg7dizNmzcv3d6kSRNycnKYMGFCpceMGjWKtWvXOnRee3/QWmv5QxfCB7grr1SnqKiIl19+mRtvvJGOHTtSWFgImFzUuXNndu/eXekxOTk57N+/v8Zjd+vWrcr76prrvvrqK44fPy6Xudxsw4ENHM8/ToemHdjBDoeO5ReFT/lPVNUZFz+u9FNaTdaNW+dISFXaunUrWVlZJCUlVdhus9m4++67CQ8Pr/SYyMhImjRpUu9zNmvWjCNHjlTanpOTY/fTkRBC8oqjNmzYQFZWFjNmzGDGjBmV7r/44osrbfv0008rXZKzp6pWqPrkunnz5tG5c2cSEhJqPK9wnlV7VgEwqN0gKXx8zY4d5hfavn370m2rVq1i586d9O3b1+5jHG2S7tatG1u2bKmwbc+ePeTm5lb7yUkI4R3ckVdqUtJys2zZMrsFVvlYS9x6663ceuut9Tof1D3X5eTksGjRIiZOnFjvcwprlPTvObfducxlrkPHksLHwwQGBgJm9AWYpFLyR1dVgnG0SXrEiBG89NJLHDt2jMaNGwMwf/58QkNDueACuwtjCyG8iDvySk3OOOMMAJo2bUp8fLzTzlNeXXPdF198walTp+Qylwe4b8B99I3uy0WdLnL4WFL4eJj4+HhCQkKYOHEikyZNYv78+Rw+fJguXbowb948evfuTWxsbIXHREVFERUVVe9zTpgwgVdffZXLL7+chx56iO3btzNlyhTuu+++0mGf77//Pv/4xz/Ytm1b6SexEydOsHDhQgD27dvH0aNH+eyzzwAYOXIkYWFh9Y5JCGEdd+SVmvJDz5496du3L9dee23p6LKsrCx+/vlnevTowU033VTvc1elNrmuvHnz5hEXF0f37t0tj0XUTXzreOJbW1Qga6296hYfH6+rsnnz5irv8ybz58/XsbGxOjg4WPfr109v375dz549W4eHh+sXXnjBKefctGmTHjJkiA4JCdGtWrXSjz32mC4sLCy9/7333tOA3rFjR+m2HTt2aMDurfx+9vjK70rUDEjVHpA7qrpVl1O09p33qqvzSm3yw549e/S1116rzzjjDB0SEqI7dOigr7vuOr1161bL4ylRU64rkZWVpYOCgvTUqVNrfWxfea94OkdzitIODEV0h4SEBJ2ammr3vt9++00qcy8hvyv/oZRap7X22J6h1eUUkPeqqD15rzjH/637PzKOZ3Bt72vpFNnJ4Zxi+dS7SqkPlVL7lVJHlVK/K6Wq7ImmlLpXKZWhlMpRSr2rlGpodTxCCO8mOUUI/zYjdQZPJD/BziM7LTmeM9YcmArEaq0jgEuBZ5RSlS7MKaWGAw8DQ4FYoCPwpBPiEUJ4N8kpQvipw3mHSctIo0FgAwa1G2TJMS0vfLTWm7TWp0p+LL51srPrTcCs4v2zgaeBMVbHI4TwbpJThPBfP+76EY1mQNsBhAaHWnJMp6wyqZSaoZQ6AWwB9gML7ezWE0gr93MaEK2UqjSMQCk1TimVqpRKzcrKckbIQggPJjlFCP/0/Y7vARgSO8SyYzql8NFa3wY0Bs4DPgdO2dmtEZBT7ueS7xvbOd5MrXWC1jqhRYsWVocrhPBwklOE8E/f7/SSwgdAa23TWq8E2gL/tLPLcaD8xAkl3x9zVkxCCO8lOUUI/5KVm8WGAxtoGNiQ/m37W3ZcV0xgGIT96/GbgDjgk+Kf44BMrfUhF8QkhPBeklOE8AO5Bblc1/s6inQRIUEhlh3X0sJHKdUSuBD4GsgDhgHXANfa2f19YLZSai7mmv1jwGwr4xFCeDfJKUL4r9imsXx4+YeWH9fqS10a0wS9F8gGpgH3aK0XKKVilFLHlVIxAFrrxcCLwPfAruLbZIvjEUJ4N8kpQghLWdrio7XOAuyuaqm13o3pfFh+23RgupUxCCF8h+QUIfxTVm4WP+76kaEdh9I0pKmlx3Za52YhhBBCiPr4+vevueLTK7jhixssP7YUPoI//viD8ePHExcXR2BgIImJie4OSQjh5SSvCEcs2bYEgOGdhlt+bFeM6hIebtOmTSxcuJABAwaQn5/v7nCEED5A8oqoL1uRjW+3fQvAxZ0vtvz40uIjGDVqFHv27OHTTz+lZ8+e7g5HCOEDJK+I+lr751qyT2bTqVknOkd2tvz4Uvh4qMOHDzN+/HiaN29OZGQkDz30EACDBw/mtddes/RcAQHyNhDCH0heEd5gyR/Ou8wFcqnLIx05coRBgwbRqFEj3n33XTZu3MikSZOIjY1l27ZtjB07tsL+WmtsNluNxw0Kkl+3EP5K8orwFou3LQacc5kL/KXFRylzK2/UKLPtq6/Kts2cabaNG1e27c8/zbbWrSs+Pj7ebF+3rmzblClm25QpDoX77LPPkpGRweLFi7n00kt59NFHad68OY888ggPPPAAISEVZ7CcM2cOwcHBNd6EEBaSvCJ5RViuwFaArchGw8CGDOlg3fpc5Ump7mG01syZM4exY8fSvHnz0u1NmjQhJyeHCRMmVHrMqFGjWLt2rSvDFAKAlBRITobERBg40N3RiKpIXhGerqgIsrIgOjqYf/f6mUUZeWxYF+qUvOIfhY/WlbeV/0RWYty4ip/KwHwis/f48p/ISkyZ4vCnsq1bt5KVlUVSUlKF7Tabjbvvvpvw8PBKj4mMjKRJkyYOnVeIukpJgaFDIT8fGjSA5cv9rPiRvCKEZebNg+uvh5Yt4eBBgFCmTXVOXvGPS11eZMeOHQC0b9++dNuqVavYuXMnffv2tfsYaZIW7pCcbIoem818TU52d0SiKpJXhKf76CN47/18Lro8A5tNOzWv+EeLjxcJDAwEzOgLME3UEydOLP3eHmmSFu6QmGhaekpafGR+Os8leUV4ssOHYcUKGPt8Mh8kT0YFJaOKGjotr0jh42Hi4+MJCQlh4sSJTJo0ifnz53P48GG6dOnCvHnz6N27N7GxsRUeExUVRVRUVL3PeeLECRYuXAjAvn37OHr0KJ999hkAI0eOJCwsrN7HFr7l9D49y5dLHx9vIHlFeKqUFHjpJUhIgG93LYB2qzln/Bxsa8bx6qtOyitaa6+6xcfH66ps3ry5yvu8yfz583VsbKwODg7W/fr109u3b9ezZ8/W4eHh+oUXXrD8fDt27NCYVbAr3Xbs2GH5+bT2nd+VP1m1SuvQUK0DA83XVatq9zggVXtA7qjqVl1O0dp33qv+kFfczVfeK65SklNA6wYNinTzO0dppqDfWbhWJyRU/ThHc4q0+HigK6+8kiuvvLLCtg4dOnDTTTc55XyxsbFVNncLUcJenx5p5fEekleEpynJKQCFNji4uSdtRv7C38+N567NUFgIzpgmSjo3CyFqpaRPT2Cg9OkRQjguMRECAsw0VQFBBRCbzGXdLiMiQnHGGfD77845r7T4CCFqRfr0CCGsNHAgnHUWdOmiWRl9E7siVjP6zKcA6NsX0tKgRw/rzyuFjxCi1gYOlIJHCGGNoiLYuhVmfvwn33y6mOYBzUmMTQQgLs4UPtdcY/15Lb3UpZRqqJSapZTapZQ6ppT6VSk1oop9xyilbEqp4+VuiVbGI4TwbpJThPBdW7ZA8+YQ17ENmQ9k8v1N3xMcaOaG6tsX1q93znmtbvEJAvYAFwC7gZHAJ0qp3lrrnXb2T9FaD7YyAK016vT1c9zg+HE4dgwaN4ZGjdwdjWeRDo+iDtyeU0DyiqiZ5LW6W70aBgww3zcIbECvlr1K7ytp8XEGS1t8tNa5WuspWuudWusirfXXwA4g3srzVCU4OJi8vDxXnKpax4+bTln79pmvx4+7OyLPkpeXJzO+ilpxd04BySuidiSv1V1KCvQ8J4tjp45Vuq9dOzh5EjIzrT+vU0d1KaWiga7Apip2OUspdVAp9btS6nGllEMtUC1btmTfvn2cOHHCrdX3sWPm2iWYr8cq/079ktaaEydOsG/fPlq2bOnucIQXcnVOAckronqS1+pv9WpIb/IM0dOimbN+ToX7lCrr4Gw1p3VuVkoFA3OBOVrrLXZ2+RHoBewCegLzgUJgqp1jjQPGAcTExFR5zoiICAD+/PNPCgoKHHsCDjh1Cg4dMmsQKmXmIThyxG3heJTg4GCio6NLf1dC1JY7cgpIXhE1k7xWdzk5sH2njYNZn5BXmEePFpWHb5Vc7rroImvP7ZTCRykVAHwA5AN32NtHa7293I8blFJPAQ9iJ0lprWcCMwESEhKq/cgVERHhEW++8lP7V7EGoBCiltyZU0DyihBWW7sWOly4nE25GXRq1omE1gmV9omJgT17rD+35YWPMj0AZwHRwEitdW0/ImnA/b0HLSLDfoWwhuSUMpJXhK9YvRoK+8wCYEzfMXYHDzRv7pyRXc7o4/Mm0B0YpbWuskegUmpE8fV6lFLdgMeBBU6IRwjh3SSnCOFjflh7iG3B/0WhuCnO/rIpUVFw8KD157Z6Hp/2wHigL5BRbi6N65RSMcXfl1xQHwqkK6VygYXA58BzVsYjhPBuklOE8D1aw6qjH1Go8xneeTjtmrSzu19UlOnXZjVLL3VprXdRfdNyo3L7PgA8YOX5hRC+RXKKEL5n+3ZQrTYA8I++/6hyv+bNnVP4yCKlQgghhHCZX36BYXkz2XrHVi4989Iq93PWpS5Zq0sIIYQQLvPLL9CvH3SN6lrtfk2aQG4uFBaa6RusIi0+QgghhHCJIyeP8N3WNfTrV/O+AQHQrBkcPmxtDFL4CCGEEMIlZv3yLj/HDeDzvDtrtb8zLndJ4SOEEEIIp7MV2fh3yusAXNYnqVaPccbILil8hBBCCOF0C/+3kD3HdxB6sgN/6fKXWj3GGSO7alX4KKXeUkpppVRrO/edqZTKV0r929rQhBBCCOErXvv5NQDODb6DwIDAWj3GnZe6Uoq/nmPnvleAo8AUKwISQgghhG/5df+vLN2+lEBbODf2vrnWj3Pnpa7VxV8rFD5Kqb8AI4AntNbZVgbmLVJSYOpU89VfyWsghHXk70leA1/03EoziXrDDRO4oH+zWj/OGZe6ajUyXmu9VSl1mHKFj1IqGJgObATetjYs75CSAkOHQn4+NGgAy5f73wKC8hoIYR35e5LXwBdprekS2YXIkOboX+6jnf0VKuyKioItW6yNpy6dm1cDCapsCdW7ga7APVprm7VheYfkZPPHabOZr8nJ7o7I9eQ1EMI68vckr4EvUkrx3NDneKf7XhLObI2dhdir5O5RXauBJsCZSqmWmJWP/6u1Xm5tSN4jMdF8IgkMNF8TE90dkevJayCEdeTvSV4DX7ZhfcNaTVxYntsudRUr38H5fKAhcL+14XiXgQNNM2xysvnj9MfmWHkNjJQUeQ2E4+TvSV6D8nwhrzyy7BFim8byj7P+wdq1wYwZU7fHO2NUV10KnzVAEXALMBh4SWu93dpwvM/Agd77hrSKv78G0iehssxMWLYMVq50dyTex9//nkBeA/CNvLI5azMvrnqRABXA0A7DSE3txOuv1+0Ybr3UpbU+BmzGtPYcAJ61NhQhvJP0STALCS5aBPffD336QLdu8J//QO/e7o5MCO/kC3nlse8eo0gXMbbfWEJPdqKwEGJi6naMyEg4cgSKiqyLq67rnf4M9AIeKS6EhPB7JX0SSj6Z+UOfBJsN1q2DpUtNy05qqlltOSkJZs6EhISy1ZRvv929sQrhjbw9r6zZu4YvtnxBWHAYj5//OD9/B2efTZ06NoPJI40ameInMtKa2Gpd+BQPX08EUoE51pxeCO/nD30StIZt20yRs3QpfP89tG5tCp0HH4TzzzfJSQhhDW/OK1prHlj6AAD39L+HMxqfQWqq+UBUHyWXu1xe+AAPAB2A67TW2prTC+EbfLFPwqFDJvGWtOrk58OwYfDXv8Lrr8MZZ7g7QiF8m7fmlbkb5rJy90pahrfkwXMfBGDtWrjttvodr3lz08G5Sxdr4qu2j49SKlIpdY1SairwNDBda726mv0bKqVmKaV2KaWOKaV+VUqNqGb/e5VSGUqpHKXUu0qphvV/KkIIR+TlmSLnoYcgPh46doQPPoBeveCbb2DvXpgzB66/3nVFj+QUIbzP3A1zAXhh2As0DWmK1ljS4mOVmlp8hgMfYTozvwI8XIvj7QEuAHYDI4FPlFK9tdY7y++olBpefLwLgT+BL4Ana3EOIYQFiopg/fqyFp3Vq01n5KQk+Pe/oX9/CA52d5SSU4TwNl9d8xWfbvqUq3pdBcCuXdCwobk8Xh9Wz+VTbeGjtf4Y+Li2B9Na51JxsdKvlVI7gHhg52m73wTM0lpvAlBKPQ3MRZKUEE6zc2dZP53ly80nqaQkuOMO+OwzaNLE3RFWJDlFiJpt2wZvvQULFkBBgZn8MTER7rrLjLJ0taCAIK7pfU3pz4609oD1c/nUZebmOlNKRWOWtdhk5+6eQFq5n9OAaKVUlJ3jjFNKpSqlUrOyspwTrBA+KDsbPv8c/vlPc328f3/TMfnii+HXX2HrVtNfZ/Rozyt67JGcIkSZkydh3DgYMMD8/Omn5gPNV19Bhw4wYoT5287JcX4sBbYC7l9yP/uP7a9039q1jhc+Vrb4OK3wKR4FNheYo7W2t8RYI6D8r6Pk+8an76i1nqm1TtBaJ7Ro0cL6YIXwEadOmVEgjz1mipyYGDO8vHNn06Kzfz/MnQs330ydFgr0BJJThCizZ48ZTXnkCOzYAS+9BHFxpm9e9+4waZLZ3q6d6SC93cnTDU9dOZXpq6cz6uNRnD7+ydEWH5de6qovpVQA8AGQD9xRxW7HgYhyP5d8L/MDCVFLWsPGjWX9dFauNJMHJiXB88/DoEHm2rp7YtPYLFq/WHKKEGV27oTBg82lrAcfrHpunAYNTIvuG2/AueeaD0Vnnml9PGkZaTz949MATLtoGqpcQIWFpvA555z6H9/qS12WFz7Fq7fPAqKBkVrrgip23QTEAZ8U/xwHZGqtLZ6cWgjfsm9fWT+dZcvM/DnDhsEtt8CHH1o310VdnCo8xW8HfyMtI430zHTSMtNIy0xj0nmTHD625BQhymRnw8iRpuC5++7aPeb22yEkBEaNgjVroFkz6+I5nn+cqz67isKiQm4/+3YSYxMr3J+eDm3bmuKlvlw9qqs+3gS6A8O01nnV7Pc+MFspNRfYDzwGzHZCPEJ4taNHzSe1ZcvMLTPTrOEzbBg89ZRp2nYVrTWZuZm0atSqdNvoeaNZ+L+FFBYVVtp/y0F7V6TqTHKKEJhL2ZddZvro1bboKXHLLbBhA1x1FSxcWDazuiO01tz2zW1sPbSVni168mLSi5X2WbHCtE45wqMvdSml2gPjgVNARrnmrvHACsxaXz201ru11ouVUi8C3wOhwH+AyVbGI4Q3KiiAn382LTpLl0Jamumvk5Rk5tXp29eM2nA2e6046ZnpZJ3IIvOBTFqGtwSgYWBDbEU2ukZ1JS46jj7RfYiLjiOuVRztItrxNm/XOwbJKUKUefhh01ozbVr9Hj9tGvzlL/DEE/Dcc47HM3v9bD5I/4Cw4DA++fsnhAWHVdpnxQpTrDkiMtKDCx+t9S6gupU4Kkxqr7WeDky3MgYhvI3WsGVL2eWrH34wrThJSTB5svm0FFY5nwBmBWdHp7TXWpNxPIO8wjw6NjPNR2v2rmHwe4PttuI0adiEXUd2lRY+r454ldmXzbab9BwlOUUIY+lSM0AhLQ0C6jksKSgI3n/fDHG//HL7HY7rklOOnDyCQvHGyDfo0aJHpfu1Nv0O61uolQgPhxMnHDtGeU7p3CyEqF5GRsXlIAIDTaFz7bUwaxbUZqBRSoq55FWyiOHy5TUnqpJWnPTMdNOSc8B8zTqRxeXdL+c/V/4HgE6Rney24vSJ7kNMk5gKnRfLX/YSQljv0CEzEnP2bMf78EVHw/Tp5nipqRUHP9Q1p9w78F6GdBhC31Z97d7/xx9mEtT27R2LOTwccnMdO0Z5UvgI4QK5ufDjj2WFzp495hNVUpIZdtq5c91XLU5ONgnKZjNfk5MrJqmM4xmkZaQxsN1AIhqaAU43/vdGPtn0SaVjNWnYhJCgkNKfm4c15/ijx53SiiOEqJs77oC//93067PCtdfC/Pnw7LOmn2CJmnIKmM7MB08cJLZpLECVRQ+Y1p7Bg+ue205XMoN8QYE1s8lL4SOEE9hssG5dWT+d1FSz/lVSEvzf/5nvHe1cmJhoPpXl52uCgjV5bRZz/5LlpB9IJz0znQO5BwBYdsMyhnYcCkBcdBzrM9ZXaMGJi46r1IoDSNHjJLt3mz5c27aZT9s33WTtKBvhW5YuNcvJbLI3ZWc9KWVmeu7Tx6y917Wr2V6WU8zXxMSKjztZeJLR80aTnpnOV9d8xYC2A6o9z4oVcN551sRc0urTtKnjx5LCRwgLaG3+kZX00/n+e2jTxnxCmzjRTDTWqFHNx6lJ5vFM0jLTyCvIY/TA0SxfDt98e4Jn9wzj6R2rYUfZvhENI+gT3adCQfPI4Ed49LxHHQ9E1MvHH5u5VwYPhk6dzAi9p56CG280n77Dw90dofAkJ0+aFc1fe63qfn711bq1WZD4vvvg66/NtoEDzeUte318CmwFXP3Z1Xy34ztaNWpF87DmNZ5jxQq45x5r4g0LM/18pPARwo0OHoTvviu7fHXqlGnRuewyM2mYoyuYb8/ezk+7fyodTZWWmVbaitM5sjOju41m4EAYODCcH94LolWjK+jTsg9xreKqbMU5/WfhGkVFcO+9ZpX7ZcvMDLsl9u0zxfHFF5v7IyKqPo7wLy++CL16wSWXOOf4d99tWqAXLTLLWwDFOaXifvm2fG744gYWbF1As5BmfHv9t3SO7FztsTMyTI7s1cuaWMPCrOvnI4WPELWUl2euWZe06mzbZppxk5LMP7Xu3et3LbukFSc9M52LO19Mr5YmU8xNn8sTyU9U2LekFeesVmehtS4tZFbcvMLh5yec59VXzcRxa9dWvqzVpo2ZpuDOO03H0sWLHZvsTfiGnTvh3/82a+o5S4MG8MorplVm6FDz8+nyCvL4+6d/55v/fUNEwwgWX7+Y3tG9azz2smWmpbu+I9BOZ+XILil8hKhCURGsX1/WT2f1anNNPCnJ/CPr37/uHe201nyQ/kGFeXFKWnEAAlRAaeEzOGYwf+v+t9I5cfpE96F9k/bSauNltmyBZ54x75+q+vIEBJhWwnvvNRPMffutdf8whHd65BFzWTQmxrnnGTnSvPdmzKh8WUprzV/n/5Ul25YQFRrFkuuXEN86vlbH/eoruPRS6+KUFh8hnGTnzrIWneXLzbDyYcNMAvrPf2q/gnn5Vpz9x/bz8vCXAXOp6eFlD7P/eNkKxiWtOHHRcfQ7o1/p9iEdhjCkwxArn55wscJC03n56afNyL3qKGXmO7nwQrPg5EMPuSZG4XlSUkz/mHfecf65lDLvtwsvrNzRXinFNb2uIT0znaU3LKVny561OmZ+vineX33VujilxUcIi2Rnm47IJf10jh41hc6IEeafUG1XME/PTOeDtA9K16gq34qjUDw15CnCG5ieq7effTtFusgUO63ipBXHh732mumzM2FC7fYPCjLrrZ19tulc2r+/U8MTHkhr0+HYlZ3de/UyfROfe84UQZnHM4luFA3ATX1v4rJul9EkpJaf+jBTd3TrZuYMsoq0+AhRT6dOmU9TJa06v/1mVi0eNgz++U/o3bvqfjqZxzMrXKIa3mk41/W5DoAd2TuYllI2PWlJK05JZ2ONLr1v0vmOL9wpPF9+vimev/mmbn2/YmLgzTfhuuvM2kqhoc6LUXieTz4x750bbnDteZ98Enr2LuTEwMnM2fJvUm5JKe3LU5eiB+DLL82CqFaSFh8haklr2LixrJ/OTz+ZTyJJSfD88zBoUMWZS083dcVUvt/5faVWHIDAgMDSwuecNufwZOKTpf1xpBVHfPQR9Oxp1larq8svNy0/06bB449bHppXOXIEfvnF/NPLz4czzzQDCXyxD1R+Pjz6qBlp5erndzR4K+F33syMDSkEqABW711dq07Mp9PaFD4lQ+StUjKc3QpS+Aifs3dv2Urmy5aZTwpJSXDrrTB3bsUp3zOPZ5K217TgpGems+HABlbevLL0stQPu35g6falADRu0LjCxH8D25WN+Tyj8Rk8cUHFEVjCfxUVmUsG//53/Y8xfbqZ6PKmm5zfwdXT2GwwZ44Z7ZaaCmedZS4ZBgaaifwOHTLD/x98EPr1q/l43uLtt81kghde6LpzFtgK+Nfqf/FE8hOc5CQBua359/kfMTb+gnodb+NGU7T1rF13oFqzctkKKXyE1zt61Ey4VdJP58ABMzQzKcl0Ku3QoeL+m7M2c++Se0nLSCMzN7PS8TYe2Ej/tqZzxQODHmBCwgTiouOIbRorrTiiVhYtMkODhw6t/zFiY80Q9wceMJc//MWSJeY5R0aar0OHVp6878AB0yJ26aXm8vTrr5sJIb3Z0aOmX8+SJa4756/7f+X6L65nc9ZmAMb0HcPAnOnMnNKM20bWr9Xpyy/N78XqVCktPsKvFRSYOVFK+umkpZlOoElJJhm26XqADQdMK87k9WmkLUljQJsBvD3qbQBCg0L5dtu3QFkrTklLTsmw8RLDOlq0OI7wK9OmmUkJHU3+EydCjx6ms+j551sTm6cqLISHHzajJ//1r+r/ebZsaToA3367KXoGDDBTBowbZ/0/XFeZNg0uuqji5JbOFhUWxY7sHXRq1onXRrzGiC4j0Bree9Os4j5mTN2OZ7PBu++alnWrSYuP8Ctam7lQSvrp/Pij+XQ3ZFgBj08O4PzBgYSFwWPfPcYlybPI+Dqj0jECVWDp97FNY/nvVf818+I0bU+A8sHOAsJtdu0ynZKvuMLxY4WFmVbLRx81w5u99Z96TbKz4corzfepqbWfwLFhQ7j/fjMXzQ03wA8/wHvvVd9vzxPt3w9vvGH6MjnTz/t+5sP0D3ll+CsEBgQS0ySGpTcsJaF1Ag2DzIumlLlEe9ll5laXJSK+/NIUpQOqX8KrXsLCTH8vK0jhIzxSRoaZR6fk8hXhB+g9LJ0WI9O5+Lo0th5J47WszVzVcyVhYecAZlr1jOMZNG7QmN7RvU0LTnF/nPKd9JRSjO422k3PTPi6efPgb3+zZhVpMCtpP/+8mdG5ZFkBX3LkiGmtHTjQzCJcn8V7u3c3heGNN5pWky++qNiXz9M99ZRpXWnf3vpj59vyWbBlAa+vfZ0fd/0IQELrBG6MuxGAc2POrfSYc86B0aNNUTlrVu3P9corZhJOZ5AWH+FzcnNNS87ipQV8uzKLjP+1ZsgQOOfCTBZ270vWyQz2ARwovhX74/AfnNPGFD539b+LCQkTiG0aW6EVJyUFXvuw8qJ7QjjDxx+bSzVWCQw0/xgfe8x06PWlVp+jR81zOvdc85o58txCQ2H+fHN58LzzzPxcLVtaFmoFKSn2F/Ksj61b4bPPTKu2ldIz0/kw/UPeT3u/tC9jRMMIJsRPqNUl/BdfNP2nFi82v6OapKaa1s7LL3c0cvs8to+PUuoOYAzQG/hYaz2miv3GALOAvHKbL9FaJ1sZj/BchYWwdNUBPvkhnR9/T2P3yXQatk/jZJPNdL6mN1l3riMoCIp0C557PpdGDRpVGFEVFx1Hr5a9aNywcekx20a0rXSelBTTOTI/33Q2Xb5cih9v4m055bffICvL/OO10uWXm8nlPv/ctCb5gvx806rQt6/jRU+JgADTVyYszMzN9d130LzmRcTrxOqc8uijphO3o+uzaW3mCisZgHHd59ex8cBGAHq26Mn4+PHc1PcmIhrWbhXcxo1Na8/NN5tLtzXNWv/KK2aG+/q02NWGJ7f4/Ak8AwwHapp2K0VrPdji8wsPVGArYMvBrZw40IpfVjZn6VJYdOJJTg6cYnYonsq/sHj/oIanCAzUgCJABbDlji20atSqXn1xkpNNgrLZzNfkZCl8vIxX5ZSPPzZ9VQIDa963LpQyfX0eegj++lffmMPmrrvMP9cZM6xvxXrySTMIIinJtPzUpZ9KTazMKSkp8PPPZlBGfezO2c0PO39g2Y5lfLvtW5besLR0rb/bEm5jw4ENXN/nega2HVivEalDh5pO5tddZy4fVnX5duVK0y1hxoz6PY/asLLFB6215TdMoppdzf1jgJX1OXa86euqK7jkErPtyy/Ltr39ttk2dmzZtn37zLYzzqj4+H79zPbU1LJtkyebbZMnl21LTTXb+vWr+PgzzjDb9+0r2zZ2rNn29ttl27780my75JKKj/eh53Tg+AG9bNuy0ufUfmqcDpgcrJmCXhTVR2vQ3937pX79xw91o+ca6edv7Kg16E2XDdY/7f5JHz151NLntGqV1qGhWgcGah3KCb2KAS79Pa1apfVzz2m9asEBj/o9OfKcStXy9wSkaj/IKUVFWnfpovWaNZVfA621w7/XolvN7/WXCZ7xe3XkOc2YoXWPHlqfusl579WiW8fqu+7S+oILtD653fNyis1W9+dku/UWfdfbH+ve136ku98ySmvQ+xqhmWJur615zfL3Xn6+1ovamedke7Pyczox9BJ9xhlaL1pUvN2J771b+lqTU9zZx+cspdRB4DDwATBVa11Yw2OEhyiwFbD10Fby9q3l7OJtJ05oYl/pxAnbsdIFGnadSgMF7cI70r0HsAKGDIHBg67kn4OvIeD/3oH3x9OjRXdoN6j4Uccsi3PgQNMUnZwMiS9fycBDqy07dk1SMjuWNYkHR7GcAQxkl8vO74fcmlPWrzetAGefXeOu9VLygf2bhdBXe29fn3XrYMoUM4t6gxeddx6lzOWXq66Cu++Gtyw6rlU5Zd48uPa0bUW6iFOFJwkFvt32LcuWruDYqWO8yVkArM7sxKt3XAqFDSBwNCkMpWvQr0xLepbhnYfTs0VP4D1Hnl4lwcHmsiHvmZapS66s2HF8bSrcO6l2/YAcdfKkNcdRpniyllLqGaCtrvp6fEdAA7uAnsB84AOt9dQq9h8HjAOIiYmJ37VL/nnYY2WHu/KOnDxC6p+pFdap2py1mXxbPg0CGjI5+DjfLQtizRoIGjOCRlE5xLeNI6l3HP1aV+6L4y+mTjXLDdhs5tLH00/DI4+4OyrXU0qt01onOHgMj88pzzwDhw+bGZedpajI9Il5/nkzhNvZrM4pOTlmFubp081QaVc4edKM9DrnHNP/xx3ybfkczjvMoROHOJB7gD3ZB3joxrP45M2unHcefLzhY55IfoLdObvJt+VXeGxwQDC5j+YSHBjM1Kkw6bEidFEAgYGaJ5/STHrUNdc9jxwxfZG++AKuv95MIrl4sZm2YeZM5xfimzeb/m2//eZ4TnFLi4/Wenu5HzcopZ4CHgTsJimt9UxgJkBCQoL1lZoPsKLDXUkrTlpGGl2iupSOlvr696+54YvKK+YF5HSkwbE+7AjL4a67okhMhIiIRRY8G9+QmGh+FyW/k8REd0fkuzwhpyxeDE84edWSgACYNMkU0SNGOPefjdWdeLU2q9RffLHrip58Wz460MbcT4tIHGKjXZcirr/Bhk3baBDYgKYhTUv325G9g8KiQgqKCiiwFVBQVEC+LZ98Wz7xZ8QTFWZ6H6/eu5pf9v9CXkEeJwpOcKLgBLkFueTm59IstBnTLppW/Hw1nV7txIHcA+QWVO6VG3fuvznvvK5mXzR/HP4DgJbhLenYrCNdIrvQJbILPVr0KF3kODERQhoGFP9OFBcOcV2zX9Om8M47MHmyKXR69zYtaq1aueb8vrhIqenJKuqtPh3uVuxaQeqfqaRlppGWmVbaigNw5zl3ck6bc8jOhoMbzyI6fyB5O+Mo2t+HgR3i+Ou5vRh1SwRtKw+kEsUqNIknSqdqF3NpTsnOhvR018yufMUV5p/P9987d02n2uaUAlsB2SezOXLyCEdOHqFrVNfSguLHXT+yas8qcvNzWZt2gp+DT/CXoXlc/dlJIkMjmfGXst6wQ98fyuG8wxTYCigsKqxwe3jww9xxzh0ALNiygBu+uAGbtlGki7AVma9FugiN5tDEQ0SGmmsxl3x0Selae1wF92TAPS+ZH//W/W98duVnAOw9updub3Sr8rX49vpvSeqUBMB/t/yXF356we5+sU1jSwsfpRTH8o+RW5BLoAokMjSSqLAoIgJbsv6nFtw+Nrb0cSM6j2DTbZto36R96TqB9nhCTmnXzhTerhYW5qGjupRSQcXHDAQClVIhQOHp19mVUiOAX7TWmUqpbsDjwKdWxuJvqmpdKLAV8Puh30nLTGND5gaevvBpggLMr/3h5Q+zas+qCseJbdKBtkFx7FgVT/8ppllx0KCePJi0iqSrTJXvrX0L3GHgQCl4HOEtOWX5chg8GEJCnH+uwECztMNzzzmv8CnSRfQ+J5ug4KZoFEHBmsREM1Rt79G9XD7/crJOZHHoxCGO5Vfsk7fk+iVc1Oki8/0fS3hu5XNld3aCD80Ia9pFtKtQ+KRnpnPwxEG78eSczCn93qZtlc55euwlGgY1JCQohAAVQIAKQNsCOZEbQNMmgRUuv4cEhdA1qiuBKpDgwGCCA4IJCgiiYVBDggOCaRLSpHTfAW0HMCF+AqHBoYQGhRLeIJzw4HAaNWhU2ipUYv349TRu2JhGDRqZ82szSmryQBhbrkhuFtqMZqHNqnxO5flrTrGyxcfSPj5KqSnA5NM2Pwm8C2wGemitdyulpgE3AI2ATOBD4GmtdUFN50hISNCpqamWxexLUlLgqyVHOd76G462XFKpFQdg822b6d6iOwAvr3qZ/x3+g+aFfcj5PY4tP/RmzYrGdOtmhoGWzKbqbdO/C8/iyPV4b8kpt94KffqYIdquUFAAnTubxUv796/fMQqLCglUgaXDnF9JeYXF2xazI3sHu3J2mbyxZwDsTKT/4JOsnvIKAAdPHKTFSy1KjxOgAmgWYv5xN2nYhJcvepkLYi8ATOGzbPtyvvgknE7twrl0RCihwaGEBIXQNKQpI7uUdVT6Zb9ZryE4IJjgQFN4BAUEEagCaRrStLRQKbAVkFeYR4AKIFAFmq8Bgaji6S9qGrY9a5bpI7V6teNz59TVggVmSoK0NMmrdVVUZOYIKiyEwEDH+vg4pXOzM0nhYxLW1oNbSzsbd2jagfEJ4wHzqSnurbgK+3do2oG4Vmb5hlv73QpH25Yu8LlsmZlLY9gwU+gMGeJdU70Lz2dF52ZncjSnaG2a/7/7Drp2tTCwGrzxBnz7rflnWpOs3Cx+zfiV9RnrSc9MZ8OBDWw5uIVNt22ic6SZSOvmBTcze/3s0sdEhUbRrkk72kW0o3+b/kw6fxJgWlTW7F1Di/AWRIVG0SSkSbVzbM2aZWJds8a6ZTwcNXGimT/n229NC7krHD1qWszfe8+5lyh9WXi46VTdqJEUPn7h002fsvCPhaRlmFacU7ZTpfdd0P4CksckA6aT3t2L7jazG7cyI6o4FUFyclmhc+CA6bRYUux06OCe5yScz1kj/erC1wufjRvN5Ytt21x7GTgvDzp2hCVLTGtTCa11aavH74d+56IPLmJXjv1Ra4uvW8zwzsMBWLtvLZm5mXRo2oHYprHV9jWprX37zCi05csrxuhuNpsZIdSkCcye7Zrf2y23mPO8847zz+VM7swpLVrApk0QHe2Fo7pEZYVFhaYvTkZaaWfjVy9+lS5RXQD4bsd3FT6NdWzWkT7RfejTsg/925a1dTcIbMCrw99kzRr4dhY8uNR0uuzf3xQ6H35ohpP6wsyvonqyXIdrLFninjW0QkPhvvvg6efyufOF1Xy34zuSdybTPKx5aafdthFt2Xt0L+HB4fRt1ZezWp1FXKs4erfsTY8WPSr0czm7jbUTEGkNt98O//ynZxU9YPpJffSRaeGePNmsheZMCxaYzuhpac49j7O5O6dY1c9HCh83Opx3mPu/vZ/0zHQ2HdhUoRUHYN3+daWFzzW9rzGFTvFK4+XXW9HadEJeutTcfvwROnUyrTlTpphOl6E1TfYvfI4s1+Eay5bBuHGuPefeo3v5cuuXLG+1kCWdk/lsdtlwl6jQqNJWn7DgMLbcsYUOTTsQGGDxOho1+PxzswDn/PkuPW2thYXBV1+Zv4mYGNNPyxkOHDDD+D/5xHQr8GbuzilWjeySwseJyrfilPTHiWgYwbwr5gHQqEEjPkz/kMIiM0ClQ9MOpcVNXHQc57UvW+nw/Pbnc377smEAGRkm4ZbcAgNNoXPddfDuu6ZJUPg3mUfI+Ww2WLUKPvjAuefRWlNQVECDQNMhZf7G+Tyw9AFzZzA0OdWDm84bxpAOQzgv5rwKHXxL+vC4Una26eg9f75nd+Jt2RIWLTJ/G40bm1merVRQYI55883WL1zrDu7OKdLi42HKX1efs34Or/78qt1WnGYhzUr3bRDYgPcve5+YJjGVWnFOl5trWnJK+uns2WPedElJZkKzzp1lmLmoyBPm/PB1aWnQtq31K4CX2HpwK3M3zOWTTZ9wTa9rmJxoBrhd0vUSUvamMLLLSAZFD+e8uDbccTN06eKcOOpq4kTT72mwFyxD3bWrmXzyootMy/ill1p37HvuMa0U7pj3xhncnVOkxcdN7LXipGemM334dK7seSUAuQW5pUMzY5vGEhdtRlSVtOaUd03va+yex2Yza9qUjL5KTYX4eNNP5//+z3wfJL89UQN/nfPDVX780fpJC7Pzsvlow0fMSZvD2j/Xlm5fvmN5aeFzZvMzS/vxANx5p5nX5z1rl2mqlx9+MIXExo3ujqT2+vSBr782y4DMmGEmiXTUm2+afj0pKaZF3le4M6dIi48LHDt1rLTzn9aaQe8O4tf9v1ZqxQHYkLmhtPC5rNtlpi9Oy94VJr6qjtZmVEhJP53vvzefJJOSzKen886DRo2se25CCMetWGFGB1nlX6v/xSPLH+FkoVmNsXGDxlzR4wqu7X0tibGJVT7urrtMa8/WrXDmmdbFU1cnT5r+Tq+9ZkZMeZOEBDO8/S9/MV0J7rij/sd6+2149lnTMuJtr4MnCwuTwscy9lpx0jLTyCvI49DEQyilUEqRm5/LKdup0lac3i17l86P0ymyU+nxWjduTevGrWs878GDptmwpFWnoMC06Fx+uZn34owznPmshRCO0NoUPv/6V/2PcbLwJDknc4huFA1Al8gunCw8ybCOw/hH339wWbfLCA2ueWRC06Zw//1mhNK8efWPx1HPPAO9erluLS6r9e0LK1eaUXpbt8KLL9Z9YMjLL8Prr5uWr06dat5f1J5c6qqnw3mHsRXZaBFuev9+sukTbvziRrutOGHBYWSdyKJleEsAPrvyM6LDo2vdinO6vDzzR1XST2fbNtNMnpQE994L3btLPx0hvMXWrabpvV27uj/2cN5h3vj5DV5f+zqJsYnMv8IMfbq488Vsv2s7HZrVfXKtO+80rT6//mqmrHC1tDSzeOX69a4/t5U6dDCzOk+YAGefDXPnQlxczY87eNC0vK1fby6B1ud9Iaonl7pqUNKKk56ZXtaKk5HGvmP7eHTwozw79FkA2jRuU9qKUzKaqqQ/TqfIThVmJO0aVbdpWYuKTBIqGXm1erW5lpyUBK++aubW8ZSZTIUQdbNiRd1H6mQcz2Daqmm8ve5tjucfB2B79nYKbAUEBwYTGBBYr6IHzD+FRx+Fxx6Db76p1yHqraDAjFx64QVoXXNjt8dr1sy0nH34YdlErw8/bH8+opwcs++UKXDttWaCwrAwl4fsF6TFp5xDJw6x8cBGzm9/funIqgHvDGDd/nWV9g0LDiu9fg5wTptzOPLQkXq34pxu586yFp3ly82w8qQk80ngP/+BiKoHbgkhvEhdOjZn5WYxdeVU3kx9szT/DO80nInnTmRI7JAa15eqrbFjzaWWklE3rjJtmsl1Y8a47pzOphTccAOMHg1vvQXDh5v83b8/tG8PR46Y0bXff28m9fv8cxlI4Gx+2+JzsvAk8zfOL+2HU9KKA7Dn3j20jWgLQPcW3Tl44iBxreLo07JP6RIOnZp1qjCRV3BgME0C61/0ZGebNXpK+ukcO2Y+IYwYYZKBNHcK4ZtWrDCtK7Vx5OQRXl3zKjZt47Jul/H4+Y/T74x+lsfUsKHpl3L33WZUqCtGfm7aBNOnm5GnvnipPiLCDDC5/37YvNmsObZ3r7mseO65Zt40Wd/QNcLCzP9cR3ld4bPpwCau/s/VFbaFBYfRq2UvsvOySwuf2aNnlxY4KSmQ/CW0TYSuDlbkp06ZCctKLl/99pt58w8bZqZm79Wr5uUgPGH9JCFE/X35JWRlwaFD9u8vsBXw3y3/5YoeV6CUoktUF14f+Tr92/TnrDOs74BTPqdccYUZkj1zJtx2m+WnquDUKTNp6vPPm1YQXxYYaBYZ7d3b3ZH4r/Bws/6bo7yu8GkQ1ICLz7yYPi1NC06f6D6VWnGACkWPI2uLaA0bNpS16Pz0k+mEnJRk/tgHDarbzKTuXutECOGYlBT4+9/N3/CwYZX/hr/d9i13L76bLQe38PmVn/PX7n8FYELCBKfFc3pOefVVs+2qqyAqyimnBeCJJ0xn4H/8w3nnEKKE3/bx6d2yNwuuXlDr/euztsjevWWFzrJlZirzpCSzlstHH5mObyXq2nrj7rVOvIm0jAlPlJxsOvNCxb/hXUd2ce+Se/liyxeAWSqiPiucW5FTHnnEFD0PP2wmPHWG5GSzVEdamvdc4pKc4t38to9PXdVmbZGcHPPHsHy5KXaysuDCC02x88wz5hONPfVpvXH3WifeQlrGhKdKTDT/6JUy783B5xcybdW/mJw8mRMFJwgPDueJC57gngH3lK6tVVtW5pRnnjHDsBctMn0OrfTnn+YS13vvec+6gJJTvJ/ftvjUlb21RQoKTAe1klmS09NhwABT6Hz4oZkDo6Z+OlC/1ht3r3XiLaRlTHiqc86BkBC47z6zxMEa9ToPLnkQgCt7Xsn0i6bTJqJNvY5tZU5p0sQUJtdfb3KcVZe88vPNpb4JE8xIJ28hOcX7SYtPHQwYYJLAsmVmPZsffzQzag4bBk8+aRbSq+vsnFD/1htZP6lm0jImPNVvv0GbNmULT/YtGM/Xv3/NfQPvY2SXkQ4d2+qcMmSIueR1221mrhlHL0lpbQq+qCizOLI3kZzi/TyyxUcpdQcwBugNfKy1HlPNvvcCDwGhwH+Af2qtK0+fXE8ZGWUjr5YuNcM6k5LMp59337WmeVZab5xHXlsBnpVTSvz3p83kjZ7E0VNziGgYQWhwKMtuXGbJsZ3xvn/2WTMI48UX4aGHHDvW88+b2FaurF2ruCeRnOL9PLXF50/gGWA4JvnYpZQaDjwMXFj8mC+AJ4u31UturlkbpaTQ2bvXfNpJSjKfTDp3dk4HPGm9cR55bQVuzCmn01rz+s+vM2X/RGyNTvL0D0/z0kUvWXX4Ula/70NDzcrjAwdCbKxpAaqPN980sxKvWGHWBvNGklO8m0e2+GitPwdQSiUAbavZ9SZgltZ6U/H+TwNzqUOSKiw0E3SVjLxatw769TOFzjvvQHy8aybvEkI4jytzSnUOnTjEzQtu5qvfvwIFl7S5mccveNyKQ7tEmzbw1Vfm8n5kpMmTtaW1WW39xRdNNwFfWJJCeCdPbfGprZ5A+THpaUC0UipKa11pSjCl1DhgHEBk5JlcfrmZJrxtW/MH/NBDZs2cRo1cE7zwLjKE1S/UO6fExMRUe+CUPSlc+dmV7D26lyYNm5A3/x0+W31Fnebv8gRxcWbZnCuvNPPv1GZyw1OnzH5r15qWnqpGuPobySnu4ZEtPnXQCMgp93PJ942BSklKaz0TmAnQvHmCvvxyMzNpq1ZOj1N4ORnC6jfqnVMSEhJ0VQf9/dDvnD/7fAqLChnQdgD3tvuYaZ/Hel3RU+L8880krKNGwc8/w1NPgb26T2tYssR8qOzUycxWLx8sDckp7uPthc9xoPxynSXfH6vpgbGxpoOyELUhQ1j9Rr1zSnW6RnVlbL+xhASF8MKwF3jt38Gcc44jR3S/Tp1g9WrT6fmss+DSS+GCC6BjR/M3snatWd390CEzF9Dll3vPBIWuIDnFfYKDzdIhRUWOHcddhc8mIA74pPjnOCDTXpO0EI6QIax+w7KcsjtnN7n5uXRv0R2AN0a+Ubp6+i+/mE/73i4iAl54wbTovPOO6Se5fbvpF3nOOWZBzlGjpJ+kPZJT3CsszEw67Airh7MHFR8zEAhUSoUAhVrrwtN2fR+YrZSaC+wHHgNmWxmLqDtfvG4tQ1i9m6tzSsqeFEbPG01Ewwh+HvszkaGRpUUPmEEUEyfW88l4oMhI5z8fX8srklPcKzzcwwofTLKZXO7n64EnlVLvApuBHlrr3VrrxUqpF4HvKZtzY3Klo3kxb/tj9+Xr1jKE1au5LKfM2ziPMf8dwynbKeJaxVW6//hx2LXLLFLsDt6WU8B384rkFPcJC3P8GFYPZ58CTKni7gpd47TW04HpVp7fU3jjH7tctxaeyFU5ZeqKqTz63aMATIifwGsjXyMooGJ6TEuDXr1MPwNX88acApJXhPXC677ubyVeNvemd7D3x+7pSq5bBwbKdWvhX3bn7ObR7x5FoZh+0XRm/GVGpaIHTP+efv3cECDemVNA8oqwnse1+AjDGzu/yXVr4a+ycrNoGNiQuZfP5W89/lblfuvWwbnnujCwcrwxp4DkFWE9K1p8lNZVTmHhkRISEnRqaqq7w6iRN16PF8IZlFLrtNYJ7o6jKm27tdUfL/mY89qfV+1+ffqY1c7j410U2GkkpwgBixfDiBGO5RQpfIQQTuXphU9tckpenlmRPDsbr528UAhf4WhOkT4+QghRgw0b4MwzpegRwhdI4SOEEDX45Rf3XeISQlhLCh8hhKjBunXuG9ElhLCWFD5CCFGDX34x61oJIbyfFD5CCFGNggL47TczqksI4f2k8BFCiGps3QoxMdbMHyKEcD8pfIQQohrr10Pfvu6OQghhFSl8hBCiGlL4COFbpPDxcikpMHWq+SqEsN769RBXebF2nyU5Rfg6WavLi3nris1CeAut/avFR3KK8AfS4uPFvHXFZiG8xZ9/mpXFW7VydySuITlF+AMpfLxYyYrNgYHetWKzEN6ipLVHKXdH4hqSU4Q/kEtdXmzgQNMULSs2C+Ec/ta/R3KK8AeWt/gopSKVUl8opXKVUruUUtdWsd8YpZRNKXW83C3R6nh83cCB8MgjkqCE73JnTvGn/j0lJKcIX+eMS11vAPlANHAd8KZSqmcV+6ZorRuVuyU7IR4hhHdzW05JS/O/wkcIX2dp4aOUCgf+BjyutT6utV4JfAncYOV5hBD+wZ055dgx2LsXunZ19pmEEK5kdYtPV8Cmtf693LY0oKpPZ2cppQ4qpX5XSj2ulLLb50gpNU4plaqUSs3KyrI4ZCGEB3NbTtmwAXr0gCDpCSmET7G68GkE5Jy2LQdobGffH4FeQEvMJ7prgAftHVRrPVNrnaC1TmjRooWF4QohPJzbcopc5hLCN1ld+BwHIk7bFgEcO31HrfV2rfUOrXWR1noD8BRwhcXxCCG8m9tySlqaf43oEsJfWF34/A4EKaW6lNsWB2yqxWM14CezZQghasltOUUKHyF8k6WFj9Y6F/gceEopFa6UOhcYDXxw+r5KqRFKqeji77sBjwMLrIxHCOHd3JVTiopMH58+feofuxDCMzljOPttQChwAPgY+KfWepNSKqZ4Xo2Y4v2GAulKqVxgISa5PeeEeIQQ3s3lOWXbNoiKgqZNHQ9eCOFZLB+voLU+DFxmZ/tuTEfFkp8fAB6w+vxCCN/ijpwil7mE8F2yVpcQQpxGCh8hfJcUPkIIcRopfITwXVL4CCHEaaTwEcJ3SeEjhBDlHDkChw9Dp07ujkQI4QxeX/ikpMDUqearEEI46qOPoFkzWLPG3ZEIIZzBq1ehSUmBoUMhPx8aNIDly2HgQHdHJYTwVikpcM89UFhocovkFCF8j1e3+CQnm6LHZjNfk5PdHZEQwpslJ5uiR2vJKUL4Kq8ufBITTUtPYKD5mpjo7oiEEN6sJIcEBEhOEcJXefWlroEDTVN0crJJUNIkLYRwxNlnQ3AwPPIIDB8uOUUIX+TVhQ+YxCTJSQhhha1boX17mDLF3ZEIIZzFqy91CSGElWT+HiF8nxQ+QghRbP166NvX3VEIIZxJCh8hhCgmLT5C+D4pfIQQAjOE/ddfpcVHCF8nhY8QQgAZGWZOsDZt3B2JEMKZpPARQgjMZa6+fUEpd0cihHAmKXyEEALTsVn69wjh+6TwEUIIylp8hBC+zfLCRykVqZT6QimVq5TapZS6tpp971VKZSilcpRS7yqlGlodjxDCu7kqp0iLjxD+wRktPm8A+UA0cB3wplKq5+k7KaWGAw8DQ4FYoCPwpBPiEUJ4N6fnlNxc2LULune3KmQhhKeytPBRSoUDfwMe11of11qvBL4EbrCz+03ALK31Jq11NvA0MMbKeIQQ3s1VOSU93RQ9DRpYFLgQwmNZvVZXV8Cmtf693LY04AI7+/YEFpy2X7RSKkprfaj8jkqpccC44h9PKaU2WhizlZoDB90dRDU8OT5Pjg08Oz5Pjg3gTAce65KcMmiQySkeOqLLk3+/Elv9eXJ8nhwbOJZTLC98GgE5p23LARrXYt+S7xsDFZKU1nomMBNAKZWqtU6wJFqLeXJs4NnxeXJs4NnxeXJsYOJz4OF+nVPAs+OT2OrPk+Pz5NjA4ZxieR+f40DEadsigGO12Lfke3v7CiH8k+QUIYSlrC58fgeClFJdym2LAzbZ2XdT8X3l98s8vUlaCOHXJKcIISxlaeGjtc4FPgeeUkqFK6XOBUYDH9jZ/X3gFqVUD6VUM+AxYHYtTjPTqnidwJNjA8+Oz5NjA8+Oz5NjAwfik5wCeHZ8Elv9eXJ8nhwbOBif0lpbFYg5oFKRwLtAEua6+sNa64+UUjHAZqCH1np38b73AQ8BocB/gAla61OWBiSE8GqSU4QQVrK88BFCCCGE8FSyZIUQQggh/IYUPkIIIYTwGx5X+Hj6Wl+1jU8pNUYpZVNKHS93S3RybHcopVKVUqeUUrNr2Nelr11tY3PT69ZQKTWr+Pd5TCn1q1JqRDX7u+y1q0ts7njtis/7oVJqv1LqqFLqd6XUrdXs67F/s54en+SU+sUmOcWx+Nz0+jk3p2itPeoGfAzMx0xGNhgzCVlPO/sNBzIxs7U2A5KB5z0ovjHAShe/dpcDlwFvArOr2c/lr10dYnPH6xYOTMGs7xQAXIKZ+yXW3a9dHWNz+WtXfN6eQMPi77sBGUC8u1+7cueVnFL/2CSn1C82j80p9YjPHa+fU3OKy55IHX4Z+UDXcts+sPdEgI+A58r9PBTI8KD4XP5mKXfuZ2pIBC5/7eoQm9tet9PiSAf+5kmvXS1ic/trh5lKfj9wpSe8dpJTLItTcorjcXpsTqkhPre+fs7IKZ52qauqdXkqrcRcvC3ttP2ilVJRHhIfwFlKqYPFTXWPK6WsXiKkvtzx2tWFW183pVQ05ndtb5I8t752NcQGbnrtlFIzlFIngC2YJLXQzm6e/jfr6fGB5JT6kpxSDU/MK87MKZ5W+Fi1Lo+z1CW+H4FeQEvM6tLXAA86Mba6cMdrV1tufd2UUsHAXGCO1nqLnV3c9trVIja3vXZa69swr8F5mAkH7c2d4+l/s54en+SU+pGcUg1PzSvOzCmeVvh4+ro8tY5Pa71da71Da12ktd4APAVc4cTY6sJj1zRy5+umlArAXGbIB+6oYje3vHa1ic3d7zmttU1rvRJoC/zTzi4e/TdrZ1+Pis/dv98aSE6xw5NzCnh+XnFWTvG0wsfT1+WpS3yn04BySlR1501rGrnkdVNKKWAWEI25zl1Qxa4uf+3qENvp3PWeCwI62dnu6X+znh7f6SSn1I/f55Q6xnc6d7zvrM0p7uqwVE1HpnmYUQ7hwLlUPcLhYkxP7x6Y3tzf4ZoRGLWNbwQQXfx9N2AjMNnJsQUBIcBUTBUfAgR5wmtXh9hc/roVn+stYDXQqIb93PHa1TY2d7znWgJXY5qcAzGjLHKB0Z7w2hWfV3JK/WOTnFL/+Dw2p9QxPpe+fq7IKU59Yev5pCOB/xY/0d3AtcXbYzDNWjHl9r0PM5TtKPAexcPfPCE+YFpxbLnAdkzzYLCTY5uCqcbL36Z4wmtX29jc9Lq1L47nZHEsJbfr3P3a1SU2N712LYAfgCPFr8cGYKy9vwl3vO+Kzyk5pf6xSU6pX2wem1PqGp+rXz9X5BRZq0sIIYQQfsPT+vgIIYQQQjiNFD5CCCGE8BtS+AghhBDCb0jhI4QQQgi/IYWPEEIIIfyGFD5CCCGE8BtS+AghhBDCb0jhI4QQQgi/IYWPEEIIIfyGFD7CJZRSoUqpvUqp3Uqphqfd945SyqaUutpd8QkhvI/kFVEfUvgIl9Ba5wGTgXbAbSXblVJTgVuAO7XW89wUnhDCC0leEfUha3UJl1FKBQJpmNV3OwK3Aq9gVvp9yp2xCSG8k+QVUVdS+AiXUkpdAnwFLAcuBF7XWt/l3qiEEN5M8oqoC7nUJVxKa/018AswFJgP3H36Pkqp25VSPyulTiqlkl0cohDCy0heEXUR5O4AhH9RSl0J9C3+8Zi23+S4H3geOBsY6KLQhBBeSvKKqAspfITLKKUuAj4AvgAKgH8opV7RWv9Wfj+t9efF+8e4PkohhDeRvCLqSi51CZdQSvUHPgd+Aq4DHgOKgKnujEsI4b0kr4j6kMJHOJ1SqjvwDfA7cJnW+pTWehswCxitlDrXrQEKIbyO5BVRX1L4CKcqblb+FsgBRmitj5a7+ykgD3jRHbEJIbyT5BXhCOnjI5xKa70bM7mYvfv2A2GujUgI4e0krwhHSOEjPI5SKgjz3gwCApRSIUCR1jrfvZEJIbyV5BVRQgof4Ykew0xDXyIP+AFIdEs0QghfIHlFADJzsxBCCCH8iHRuFkIIIYTfkMJHCCGEEH5DCh8hhBBC+A0pfIQQQgjhN6TwEUIIIYTfkMJHCCGEEH5DCh8hhBBC+I3/B4kXc6vvVxn0AAAAAElFTkSuQmCC)]

Lasso regression subgradient vector:
$$
g(\Theta,J) = \nabla_{\Theta}MSE(\Theta)+\alpha\left| \
begin{matrix}
sin(\Theta_1)\
sin(\Theta_2)\
\vdots\
sin(\Theta_n)
\end{matrix}
\right|\ \ \ where sign(\Theta_i) =
\begin{cases}
-1\ if \Theta_i<0\
0\ if \Theta_i = 0\
+1 \ If \Theta_i>0

\end{cases}
$$

from sklearn.linear_model import Lasso
lasso_reg = Lasso(alpha=0.1)
lasso_reg.fit(X, y)
lasso_reg.predict([[1.5]])

elastic network

from sklearn.linear_model import ElasticNet
elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5, random_state=42)
elastic_net.fit(X, y)
elastic_net.predict([[1.5]])

stop early

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-GFuvg0wm-1658889211622)(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABEYElEQVR4nO3dd3hUVfrA8e+bBEIJoYZA6ApSBQSsKCCgKAgWFAurgCIqYMdeALGsruju/tZVRBAUV7EAKq6yiIAiNQhBKaJI7016SXl/f5yZzEyYhASSmUnyfp7nPHPvuWfmnrmEvDnnnnuOqCrGGGNMpIkKdwWMMcaYYCxAGWOMiUgWoIwxxkQkC1DGGGMikgUoY4wxESkm3BXIT1WqVNG6deuGuxrGGGPyYPHixbtUNSFrfpEKUHXr1iU5OTnc1TDGGJMHIrI+WL518RljjIlIFqCMMcZEJAtQxhhjIpIFKGOMMRHJApQxxpiIVKRG8Rlj8l9GRgabNm3i0KFD4a6KKYRKlChB1apViY+Pz/N7LUAZY3K0a9cuRISGDRsSFWWdLib3VJUjR46wefNmgDwHKftp83jtNbj2WmjcGFasCHdtjIkcf/75J4mJiRacTJ6JCGXKlKFGjRrs2LEjz++3FpTH9OnwzTdue8UKaNIkvPUxJlKkp6dTokSJcFfDFGKlS5cmNTU1z++zP4k8Gjf2ba9cGb56GBOJRCTcVTCF2Kn+/FiA8rAAZYwxkcUClEejRr7tVavCVw9jjDGOBSgP/xbUqlWQkRG+uhhjwm/YsGE0a9Ys2/1gBg8eTIcOHfL93MWVBSiPKlVcAjhyBDZsCG99jDGnpnv37nTu3DnosZUrVyIiTJ8+Pc+fO2TIEGbPnn261Quwbt06ROSEVRgK4lwFoUOHDgwePLjAPt8ClJ+srShjTOHTv39/vvvuO9atW3fCsTFjxlCnTh06deqU58+Ni4ujcuXK+VDDyDpXJLMA5cf/PpQNlDCmcOrWrRuJiYm8++67Afmpqam8//773H777agqd9xxB/Xq1aN06dI0aNCAV155hYwc+vazdrulp6czZMgQKlasSMWKFXnggQdIT08PeM8333zDJZdcQsWKFalUqRJdunRhpd8vl3r16gFw7rnnIiKZ3YNZz5WRkcGIESOoVasWsbGxnH322Xz++eeZx70tsc8++4zLLruMMmXK0KRJk5O2FL///nsuuOAC4uLiKF++POeffz6//PJL5vG5c+fSvn37zGeZ7rnnHvbv3w9A3759mT17Nm+88QYigogE/aPgdFiA8mMj+Yw5OZHwpdyIiYmhT58+jBs3LiDgfPnll+zatYt+/fqRkZFBjRo1+Pjjj1m5ciUvvPACL7744glBLScjR45k9OjRjBo1innz5pGens4HH3wQUObQoUM88MADLFy4kFmzZlG+fHm6d+/O8ePHAVi4cCHgAtnWrVuZNGlS0HP94x//4G9/+xsvv/wyP//8M9deey3XXXcdS5cuDSj31FNPcd9995GSksK5557LTTfdxMGDB4N+ZlpaGldffTUXX3wxKSkpLFiwgPvvv5/o6GgAfv75Zy6//HJ69OhBSkoKkyZNYunSpdx+++2Zdbrwwgvp168fW7duZevWrdSqVSvX1y9XVLXIpNatW+vp+PprVXDpkktO66OMKTJWrFgRsO/9PxKOlFurV69WQKdNm5aZ17VrV73iiiuyfc9jjz2mnTp1ytwfOnSoNm3aNNv96tWr6/PPP5+5n56erg0aNND27dtne46DBw9qVFSU/vDDD6qqunbtWgV00aJFAeWynispKUmHDx8eUKZ9+/bau3fvgM956623Mo9v2rRJgcxzZbV7924FdNasWUGP33rrrXr77bcH5C1ZskQB3b59e2YdBg0alO339Zf158gfkKxBfqdbC8qPdfEZUzQ0aNCAdu3aMXbsWAC2bNnCtGnT6N+/f2aZt956izZt2pCQkEBcXByvv/46G3I5Omrfvn1s3bqVCy+8MDMvKiqK888/P6DcmjVruOWWWzjzzDOJj48nMTGRjIyMXJ8HYP/+/WzZsoW2bdsG5F988cWsyDIvW/PmzTO3k5KSALKdYqhSpUr07duXLl260K1bN1577TU2btyYeXzx4sVMmDCBuLi4zOStw5o1a3Jd/9NhAcpP7dpQpozb3rXLJWNMoHC2ofKif//+TJkyhT179jBu3DgqVapEjx49AJg4cSIPPPAAffv2Zdq0aSxdupSBAwdmdr3ll+7du7Nz505GjRrFggULWLJkCTExMad0nmCzMWTN85+Synssp/tq7777LgsWLKBdu3Z88cUXnHXWWUybNi3zff3792fp0qWZKSUlhd9++42WLVvmuf6nwgKUn6goaNjQt2+tKGMKr+uvv55SpUoxYcIExo4dy2233Zb5C3zOnDmcf/75DB48mFatWlG/fv08tQrKly9P9erVmT9/fmaeqmbeUwLYvXs3K1eu5Mknn6Rz5840btyYAwcOkJaWllmmZMmSACcMrvAXHx9PUlISc+bMCcifM2cOTfJh0tAWLVrw2GOPMWvWLDp06MD48eMBaNWqFcuXL6d+/fonpNKlS2fWP6e6ny4LUFnYjBLGFA2lS5fmlltuYdiwYaxZs4Y77rgj89hZZ53FTz/9xNdff81vv/3GiBEj8vzc0f33388rr7zCp59+yq+//soDDzzA1q1bM49XrFiRKlWqMHr0aH7//Xdmz57N3XffTUyMb47uqlWrUrp0aaZNm8b27dvZt29f0HM98sgjvPrqq3z44YesXr2aZ599lh9++IGHH344j1fFZ+3atTz++OPMnTuX9evXM3PmTJYtW5YZ9B577DEWLlzI3XffzZIlS/j999+ZOnUqd911V+Zn1K1bl4ULF7Ju3Tp27dqVY2vtVFiAysJG8hlTdPTv35+9e/dy0UUX0djvP/ddd91Fr169uOWWWzj33HNZt25dnn/ZP/zww/Tr14/+/ftz/vnnk5GRQe/evTOPR0VFMXHiRJYtW0azZs0YNGgQI0aMIDY2NrNMTEwM//znP3nnnXdISkri6quvDnqu++67j0ceeYRHH32UZs2aMXnyZD777LPT6morU6YMq1ev5oYbbuCss86iT58+9O7dm8ceewxw97O+//571q1bR/v27WnRogVPPPEEiYmJmZ8xZMgQSpYsSZMmTUhISMjTvbXcEM1rx24Ea9OmjWZ9IjuvPv0UbrjBbV9xBXz9dT5UzJhCbOXKlQG/3I05FTn9HInIYlVtkzXfWlBZ2GwSxhgTGSxAZVG/vhssAbB+PRw+HN76GGNMcWUBKovYWDjzTLetCr/+Gt76GGNMcRXSACUiE0Rkq4jsF5HVItI/m3J9RSRdRA76pQ6hqqd18xljTPiFugX1ElBXVeOBHsDzItI6m7LzVDXOL80KVSVtRgljjAm/kAYoVV2uqse8u550ZijrkBv+LagsM4kYY4wJkZDfgxKRf4vIYWAVsBX4bzZFzxGRXZ6uwGdEJCZYIREZICLJIpK8c+fOfKnj2Wf7trNMFmyMMSZEQh6gVHUgUA64BJgEHAtS7HugGVAV6AncDDySzee9raptVLVNQkJCvtSxaVPwPuy9Zg1k83C3McaYAhSWUXyqmq6qc4CawD1Bjv+hqmtVNUNVfwaeA64PVf1KlQrs5lu2LFRnNsYY4xXuYeYx5O4elAK5XK7sNHlWizznHF/WkiUhObMxJoL17duXq666Kk/v6dChA4MHDy6gGhV9IQtQIlJVRG4SkTgRiRaRLriuu++ClL1SRBI9242AZ4DPs5bLV4cOQZ06kJQE6ekWoIwppLzLj2eX+vbte0qf+49//IMJEybk6T2TJk3ipZdeOqXzhdKpBN9QCDrwoIAorjvvLVxgXA88oKqfi0htYAXQRFU3AJ2AcSISB2wHJgAvFmjtypZ1a0ofOgSrVnHOOU0zD1mAMqbw8J9RfOrUqdx5550Bed6lIrxSU1MD1lHKTvny5fNcl0qVKuX5PcYnZC0oVd2pqu1VtYKqxqvq2ao62nNsg+dZpw2e/SGqmqiqZVX1DFV9VlVTC7ySrT2PZC1ejP8kwcuXw7FgQzmMMRGnWrVqmalChQoBeUePHqVChQp8+OGHdOzYkdKlSzNq1Ch2797NzTffTM2aNSldujRNmzbl3XffDfjcrK2MDh06MHDgQJ588kmqVKlC1apVGTJkSMCSE1m7+OrWrcvzzz/PXXfdRXx8PDVr1uRvf/tbwHlWr15N+/btKVWqFA0bNuS///0vcXFxjBs3Ltvv/PPPP9OpUyfi4+MpV64cLVq0YObMmZnHV6xYQbdu3ShXrhxVq1bl5ptvZtu2bQAMGzaM8ePH89VXX2W2MmfNmpXXy14gwn0PKrK08Uymm5xM+fJwxhluNy3NnocyJoBI9untt33l3n4757L+WrfOXbl88MQTTzBw4EBWrFjBNddcw9GjR2nVqhVTp05l+fLl3H///dx1113MmDEjx8/54IMPiImJYe7cufzrX//i73//OxMnTszxPa+//jpnn302P/30E4899hiPPvoo8+bNA9wqttdeey0xMTHMnz+fcePGMXz4cI6d5C/kW265herVq7Nw4UKWLFnCsGHDKFWqFOBalO3ataNZs2YsXLiQb7/9loMHD9KjRw8yMjIYMmQIvXr1onPnzmzdupWtW7dy0UUX5eFqFpxQdvFFPr8WFLiBEn/84bKWLAkcOGGMKbzuvfderr8+cGDwI4/4nmQZMGAA3333HR9++CGdOnXK9nOaNGnCc889B7hFEEePHs2MGTO4+eabs33P5Zdfntmquvfee/nnP//JjBkzuPDCC5k+fTq//vor//vf/6hRowbgAlrbtm1z/D7r169nyJAhNPJMg1O/fv3MY2+++SYtWrTg5Zdfzsx77733qFSpEsnJyZx33nmULl2a2NhYqlWrluN5Qs1aUP68AWrJEkhLs4ESxmRHNfs0YICv3IABOZf1t3hx7srlgzZtApceSk9P54UXXqB58+ZUrlyZuLg4Jk2adNIF+Jo3bx6wn5SUxI4dO075PatWrSIpKSkzOAGce+65REXl/Kv6oYceon///nTs2JEXXniBVX6TiC5evJjvv/+euLi4zFSrVi2APC1zHw4WoPxVrgz16sGRI7BqFa1a+Q4tWhS+ahlj8lfZsmUD9l999VVGjhzJI488wowZM1i6dCnXXHMNx48fz/Fzsg6uEJGTLnue03tUFTmFLs1hw4ZldlfOnTuX5s2bM3bsWMB1G3br1o2lS5cGpN9++y0iR+75sy6+rEaOhLg4qFeP86r7spcscQMl/FZrNsYUEXPmzKF79+7ceuutgAsUq1evzhxkESqNGzdm8+bNbNmyhaSkJACSk5NPGvQAGjRoQIMGDbjvvvu45557eOedd7j99ttp1aoVH3/8MXXq1Ml2tGLJkiVJT0/P1++SH6wFldW118Jll0HZslSuDA0auOzjx62bz5ii6qyzzmLGjBnMmTOHVatWMXjwYNauXRvyelx22WU0bNiQPn36kJKSwvz583nooYeIiYnJtmV15MgRBg0axKxZs1i3bh0LFixgzpw5NGnSBIBBgwaxb98+brzxRhYsWMAff/zBt99+y4ABAzhw4ADgRhf+8ssv/Prrr+zatYvU1IIfNJ0bFqBO4oILfNvz54evHsaYgvP0009z3nnnceWVV9KuXTvKli1L7969Q16PqKgoJk+ezLFjxzjvvPPo06cPTz31FCKSOSovq+joaPbu3UufPn1o2LAh1157LRdeeCGvvfYa4O5x/fjjj0RFRXHFFVfQtGlTBg0aRGxsLLGeLqE777yTxo0b06ZNGxISEvjxxx9D9p1zIloANyDDpU2bNpqcnHx6H6IKr7zimksTJvDvt2MYNMgduvFG+Oij06+nMYXJypUraew/OaUJqZSUFFq2bElycjKtW2e3fF7ky+nnSEQWq2qbrPl2DyorERg92k1j/uSTXHCBb8SNtaCMMQVt8uTJlC1blgYNGrBu3ToeeughWrRoQSv/UVvFhHXxBeP9KyU5mbPPBu/MKOvXg+fha2OMKRAHDhxg8ODBNGnShN69e9O4cWOmTZt2SqP7CjsLUMF4n5FYvJgSJXzxCmDBgvBUyRhTPNx2222sXr2aI0eOsGXLFv7zn/+QmJgY7mqFhQWoYPxaUGADJYwxJhwsQAXj7etNSYHU1IAANWdOeKpkTDgVpcFUJvRO9efHAlQwFSpA/fruydzly7n4Yt+hhQvdRBPGFBfR0dER81yMKZyOHDmSqyVNsrIAlZ1u3dxDu0BiInjmYOT4cevmM8VLhQoV2L59e65mMzDGn6py+PBhNm/eTNWqVfP8fhtmnp2//z1gt3178M6/OHs2XHpp6KtkTDhUqVKFTZs28euvv4a7KqYQKlGiBImJicTHx+f5vRagcql9exg1ym3Pnh3euhgTSlFRUdSuXTvc1TDFkHXx5WTvXpgxA44fp317X/b8+bbCrjHGFDQLUDm58ELo3Bl++YWkJDduAuDoUVt+wxhjCpoFqJx4n4fyRCP/VpR18xljTMGyAJWTc891r57pI/wD1HffhaE+xhhTjFiAyknbtu7V83Rup06+Q3PmwOHDYaiTMcYUExagcnLOOVC2LPz2G2zbRlISNG3qDh0/Dt9/H97qGWNMUWYBKicxMb6J+DwLeF1+ue/w//4XhjoZY0wxEdIAJSITRGSriOwXkdUi0j+Hsg+KyDYR2SciY0UkNpR1zeSd5+iXXwALUMYYEyohXVFXRJoCv6vqMRFpBMwCuqnq4izlugDvAR2BLcBkYL6qPp7T5+fLirpZbd8OGRlQvTrg7jtVrOi6+AA2bYIaNfL3lMYYU5xkt6JuSFtQqrpcVb2PuKonnRmkaB9gjKf8XmAE0Dc0tcwiMTEzOAGUKQOXXOI7PH16GOpkjDHFQMjvQYnIv0XkMLAK2Ar8N0ixpkCK334KkCgilYN83gARSRaR5J07dxZInTN5Jsu87DJf1rRpBXtKY4wprkIeoFR1IFAOuASYBASbNCgO2Oe3790uF+Tz3lbVNqraJiEhIb+r63z4ITRoACNGAHDFFb5DX38NthKBMcbkv7CM4lPVdFWdA9QE7glS5CDgP/Wtd/tAQdctqNhY+P33zOehmjcH79yZ+/bBDz+EpVbGGFOkhXuYeQzB70EtB1r47bcAtqvq7pDUKivvA7vz5kFaGiLQvbvv8JdfhqVWxhhTpIUsQIlIVRG5SUTiRCTaM1LvZiDYpEHvAXeISBMRqQg8DYwLVV1PkJjouvgOHYKffgKgRw/f4c8/B1sR2xhj8lcoW1CK687bBOwFXgUeUNXPRaS2iBwUkdoAqvoN8AowE1jvSUNDWNcTeVconDEDcPPylfPcEVu7FlasCFO9jDGmiApZgFLVnaraXlUrqGq8qp6tqqM9xzaoapyqbvAr/5qqJnrK9vMbnh4e3on4PAEqNha6dPEdtm4+Y4zJX+G+B1V4eFtQP/7oFoQisJvviy/CUCdjjCnCbMn33EpIgNdfh2bNIDoagK5dISrKPR41f76bdCIxMcz1NMaYIsJaUHnxwANuhd0SJQCoXNk3wE/VDZYwxhiTPyxAnabrrvNtf/JJ+OphjDFFjQWovHrrLbjmGvjzTwCuv9536LvvYMeOsNTKGGOKHAtQefXBB64vb/ZsAGrW9K3IkZEBkyaFsW7GGFOEWIDKqyzDzQF69fIdnjgxxPUxxpgiygJUXnkDlN86G9dfDyJue/Zs2LYtDPUyxpgixgJUXl1wgZtCYtUqWLcOcMtFtWvnDqvCZ5+Fr3rGGFNUWIDKqxIlfOu+f/11ZvaNN/qKWDefMcacPgtQp6JrV/f61VeZWddd5x7aBbf8xvr1YaiXMcYUIRagTsWVV7pVC6+5JjMrMTFwpd333w99tYwxpiixAHUqqld33Xv9+wdk9+nj237vPVuCwxhjTocFqHx0zTUQ71n797ff3PqGxhhjTo0FqFOVkeHGlL/wQmZTqXTpwGeixo0LT9WMMaYosAB1qlTdyIinn3bNJQ//br6JE+HIkTDUzRhjigALUKcqOtoNlICA0Xxt28KZZ7rt/ftthnNjjDlVFqBOx1VXudcpUzKzROC223xFxo4NbZWMMaaosAB1Orp2dQ/uzpkDO3dmZvfp45v6aPp0WLMmTPUzxphCzALU6Shf3s3Nl5ERsOZ7nTruUSmvt98OQ92MMaaQswB1uq691r1OnhyQfffdvu1334Vjx0JYJ2OMKQIsQJ2uq6+G2rWhUaOA7K5doVYtt71z5wnxyxhjzElYgDpdiYluVvNXXw3Ijo4OnGhi1KjQVssYYwq7kAUoEYkVkTEisl5EDojIEhG5MpuyfUUkXUQO+qUOoaprnnlHRGRxxx0uUAHMmgUrV4auSsYYU9iFsgUVA2wE2gPlgWeAj0Wkbjbl56lqnF+aFZpqnqKdO92Y8qNHM7Nq1IAePXxF/vWvMNTLGGMKqZAFKFU9pKrDVHWdqmao6lRgLdA6VHUoUFde6ZpM//tfQPa99/q2x42DvXtDWy1jjCmschWgRORFESnjt99VREr77ceLyHt5ObGIJAJnAcuzKXKOiOwSkdUi8oyIxGTzOQNEJFlEknf6PYsUcjfc4F4//DAgu0MHaN7cbR8+DO+8E9pqGWNMYSWaizUhRCQdqK6qOzz7+4GWqvqHZz8R2KKq0bk6qUgJ4GtgjareFeT4GYAC64GmwETgfVV9KafPbdOmjSYnJ+emCvlv3TqoVw/KlIEdO6Bs2cxDY8e6xhW4AX9r1kBM0HBrjDHFj4gsVtU2WfNz28WXdRRA8FEBuatIFPA+cBwYHKyMqv6hqms9XYE/A88B15/qOUOibl244ALXTPryy4BDt9wCVaq47Q0bAmZGMsYYk42QDjMXEQHGAIlAT1VNzeVbldMIiiFz883u9aOPArJLlYJ77vHtv/56COtkjDGFVKifg3oTaAx0V9VsF6IQkSs93YaISCPciL/Inxf8hhsgKsqttvvnnwGH7rnHTdsHMHeum77PGGNM9vJyJ+RuETno9747RGS3Z7/cyd4sInWAu4BjwDbxPTt0F/ADsAJooqobgE7AOBGJA7YDE4AX81DX8KheHTp2dHPz7dgBFSoEHPrLX9y0RwAvvRSwSocxxpgscjtIYh2umy1HqlovH+p0ysI6SMIrLS3bERC//gqNG2cuwMvSpdCiReiqZowxkei0Bkmoal1VrXeylP/VLoRyGJ7XsCH07Onb/+tfQ1AfY4wppGwuvoKgCvPmwbRpJxx64gnf9scfw++/h7BexhhTiOT2Qd0WInJplrzeIvKHiOwQkbdEpGTBVLEQmj4dLroIHnjA15/n0aoVXH65287IgFdeCX31jDGmMMhtC+p54GLvjog0Ad4FfgM+BHoDj+V77QqrSy91s5yvWgULF55w+MknfdvjxsGmTaGrmjHGFBa5DVCtgOl++zcBK1S1i6reDzwA3JjPdSu8SpSA3r3d9vjxJxxu1w4uvNBtp6a6EX3GGGMC5TZAVQY2++23A/ynS5gF1M6nOhUNffq41w8/DJjhHNzqHM8+69sfPdrNlGTy17hx4xCRzBQdHU2NGjXo1asXv/76a4Gcc9asWQwbNoyMjIwC+fz8VLduXfr27Zvn9w0bNgzJZokZY/JTbgPUTqAGgIhE42YgX+B3vCQQ+f8jQ6l5c3fD6c8/4fMTnzHu0gXatnXbqakwYkRoq1ecfPLJJ8ybN4/vv/+el156iSVLltCpUyf27duX7+eaNWsWw4cPLxQByphIl9sANQsY6pnE9WFP3ky/402AdflXrSKiXz/3Onr0CYdE4IUXfPvjx8Pq1SGqVzHTsmVLLrjgAtq2bcttt93Gm2++yebNm5k7d264q2aMyUFuA9QzQAPgd+AF4FFVPeR3/FZgRj7XrfD7y1+gVi045xw3ZC+L9u2hc2e3nZ4Ow4eHuH7FVHx8PACpqYFTQaakpNCjRw8qVqxI6dKladu2LT/88ENAmUWLFnHZZZdRuXJlypQpwxlnnMHAgQMB1/U13POPWKJEicyuxZyICE8//TQjR46kTp06lC1blm7durFjxw527NhBr169KF++PLVq1eLll18+4f0LFy6kc+fOxMXFUbZsWTp16sTCIANz/vGPf1C3bl1KlSpFmzZtTvheXmvXrqV3794kJCQQGxtLy5YtmTx5co7fwZgCo6q5SrjpjVoASUGOtQAq5/azCiq1bt1aI056eo6H581TdWPRVUVUf/45RPUqBt59910FdNWqVZqamqpHjx7VFStWaKdOnbRq1aq6b9++zLKLFy/WMmXKaNu2bfWTTz7Rr776Srt3764lS5bU5ORkVVU9cOCAVqxYUbt06aJffPGFzpw5U99991298847VVV148aNescddyigc+bM0Xnz5um8efNyrCOgtWvX1q5du+rUqVN1zJgxWq5cOe3SpYtedNFFOmLECJ0+fboOGDBAAf3qq68y35uSkqKlSpXSVq1a6SeffKKffvqptmnTRkuVKqVLly7NLPfOO+8ooH379tWvv/5a/+///k9r1Kih8fHx2qdPn8xyGzZs0ISEBG3atKm+//77+s0332i/fv1URPTzzz/PLDd06FB1vzqMyR9AsgaLO8EyC2uKyACVC1dd5QtS3buHuzZFhzdAZU1JSUm6cOHCgLIdO3bURo0a6bFjxzLz0tLStFGjRnr11VerquqiRYsU0JSUlGzP6f3lnZqamqs6AtqgQYOA8g8++KACOmLEiMy81NRUTUhI0L59+2bm9ezZU8uXL6979+7NzNu3b59WrFhRr732WlVVTU9P15o1a2qXLl0CzvvRRx8pEBCgbr/9dq1SpYru2rUroGznzp21RYsWJ3xHY/JLdgEqtw/qPpSblD9tuiJo/354803473+DHh4xwt2TAreU1MyZQYuZUzR58mQWLVrEwoULmTJlCk2aNKFr166sXLkSgCNHjjB79mxuuOEGoqKiSEtLIy0tDVWlc+fOfP/99wA0aNCAChUqcNdddzFhwgQ2btyYL/W77LLLiPGbIqtRo0YAdOnSJTMvJiaG+vXrB5zz+++/56qrrqKC36TE8fHx9OjRg9mzZwOwadMmNm3aRK9evQLO2bNnz4BzAnzzzTd07dqV8uXLZ16DtLQ0unTpQkpKCvv378+X72tMbuV2NvNXgV3AQbJfl0mB1/KjUkXOxIkwcKAbtte16wmHW7aEW2+F995z+w8/DMnJbuUOc/qaNWtG/fr1M/cvv/xyatWqxbBhw5g4cSJ79uwhPT2dESNGMCKb4ZQZGRmUL1+emTNnMmLECAYOHMiBAwdo2rQpw4cPp6f/JIt5VLFixYD9kiVLZpt/1O+RhT179lC9evUTPq9atWrs3bsXgK1btwKQmJgYUCYmJobKlSsH5O3YsYP33nuP97w/iFns3r078/6dMaGQ2wCVjBup9xUwRlVtNaO8uPlmF3V+/BF+/hnOPvuEIi+8AJ98AkeOwJIl8P77vkepTP4qXbo0Z5xxBsuWLQOgQoUKREVFMWjQIG677bag74ny/LXQsmVLPvvsM9LS0khOTuall16iV69epKSk0KxZs5B9B4BKlSqxbdu2E/K3bdtGpUqVADID2Pbt2wPKpKWlsXv37oC8ypUrc8kll/DYY8EnhUlKSsqPahuTa7mdzfw84HxgLzBJRH4VkUe9iwqak4iLA+8vvrfeClqkZk0YMsS3/9RTbvV4k/8OHz7MmjVrSEhIAKBs2bJccsklpKSk0KpVK9q0aXNCyiomJoYLLriAESNGkJGRkdldGBsbC7huw4LWvn17vvrqKw4cOJCZd+DAAb788kvat28PQM2aNalVqxYff/xxwHu9QdbfFVdcwbJly2jatGnQa+D9bsaESq47kVR1uao+hHtg9ymgA7BORD4XEfvJPZm773av7713wmq7Xo8+CtWque3Nm2HkyNBUrahbunQp8+fPZ968eUyaNImrrrqKPXv2cO+992aWee2111i8eDFdunTho48+Yvbs2Xz22Wc89dRTPP744wBMnTqVHj16MHbsWGbOnMnUqVMZMmQI5cqV40LP3FVNmjQBYOTIkSxYsICCXJ/smWee4ciRI3Tq1InPPvuMSZMm0blzZw4fPsyznqlKoqKiGDp0KNOmTaNfv35MmzaNN954g4cffviE7rrnnnuOffv20a5dO8aPH8/s2bOZMmUKzz//PLfffnuBfQ9jshVs5ERuEnA57gHeNKDCqX5OfqaIH8XXsaMbqve3v2VbZPRo34i+MmVUN2wIYf2KmGCj+BISEvTSSy/Vb7755oTyK1as0BtvvFETEhK0ZMmSWqNGDe3evXvm0O5Vq1Zpr169tG7duhobG6tVqlTRK6+8UufPn5/5GWlpaTpw4EBNSEhQETnpaDdAn3rqqaD1/u233wLy27dvr23btg3Imz9/vnbq1EnLli2rZcqU0Y4dO+qCBQtOOM/f//53rV27tsbGxmrr1q31hx9+0Dp16gSM4lP1DZVPSkrSEiVKaLVq1bRz5876/vvvZ5axUXwmv5HNKL5crajrJSJ1gdsB792R94Cxqro2H2LlaYuIFXVz8tVXcNVV7uHdP/4IurhherqbIclze4SePeHTT0NcT2OMCaHTWlFXRG4RkRnACqAhcBdQV1WfiZTgVChceSX06gVBZgTwio6Gf/3Lt//ZZ0HXPTTGmCIvVy0oEckANgD/wQ03D0pVwzrMPOJbUHnQp49v2Hn9+vDLL2D3qI0xRVF2LajcBqh1uD78nKiqnnFq1csfhS5Aqfqe0M1i+3Zo2BC8E26PGAFPPx3CuhljTIicVhefqtZV1Xo5JaB9vte6qNqyBe66yz2dm43ExMDZzl94wd22MsaY4uK05yoQkWoi8i/AFovILRF49134z3/g99+zLXb33W4idHBrHg4Y4BpdxhhTHOR2kEQFEflARHaKyBYRuU+cocAfuId47UGJ3Kpe3bWeVOGvf822WHQ0jBrlm/JoxgwYMyZEdTQhkZGRwfbt2wNS1mVAjCmuctuCehG3zPt4YA/wOvAFrlvvSlU9V1U/zOkDRCRWRMaIyHoROSAiS0TkyhzKPygi20Rkn4iMLXIPAz/+uIs848fDhg3ZFjv3XHjIbxrehx92D/GaomHMmDHUqFGDevXqUa9ePWrVqpXtVEPGFDe5DVDdgH6qOgTogZswdo2qdlTV2bn8jBhgIy6olcctgvix59mqACLSBXgc6ATUBc4AitZyfg0awE03QVoavPJKjkWHD3cj+cBNjH7PPdbVV1QcOXKEmJgYjhw5wpEjR0hNTeXgwYPhrpYxESG3ASoJ9wwUqvoHcBQ4cR3zHKjqIVUdpqrrVDVDVacCa4HWQYr3wU1Ku1xV9wIjgL55OV+h8OST7vWdd8Az63QwZcoEdu19+SV8mGN71RhjCr/cBqgowL9jPB04ralMPRPNngUsD3K4KZDit58CJIpI5awFRWSAiCSLSPLOnTtPp0qh17QpXHcdHDsGX3+dY9F27VzLyWvQIMin5YiMMSYi5TZACTBBRL4QkS+AUsBo775ffu4+TKQE8AEwXlVXBSkSB+zz2/dul8taUFXfVtU2qtrGOzt1ofLii5CSArmYjPPll6FuXbf955/uYd6MjAKtnTHGhE1uA9R4YAuw25Mm4O4n7c6STkpEooD3gePA4GyKHQT8p1r2bh8IUrZwa9gQmjfPVdFy5dw6Ud5RfTNnwuuvF2DdjDEmjHK1YKGq9suPk4mIAGOARKCrqmY3nnY50ALwLmLTAtiuqrkKgoXWzJnQuLFvzY0gLr4YnnjC9xDvk09C587QokWI6miMMSES6kXF3wQaA91VNacV3d4D7hCRJiJSEXgaGBeC+oXPc89Bx47w/PMnLTp0KHjX0Dt+HG65BQ4dKuD6GWNMiIUsQIlIHdws6C2BbSJy0JN6i0htz3ZtAFX9BngFmAms96ShoaprWPTs6WaYePttWJvzBPElSsCECVC6tNtfscINmjDGmKIkZAFKVderqqhqKVWN80sfqOoGz/YGv/KvqWqiqsaraj9VPRaquoZF06bwl79AaqprIp1Ew4aBy3KMH+9mTzLGmKIi1F18JifDh/uaR8uDjb4P1K+fG8nnNXCgb6FDY4wp7CxARZJ69Xwzwj711EmLi8Abb7jGF7gJZW+4wc02YYwxhZ0FqEjz9NNQtix8/rkb1XcSZcvCJ5+4V4DVq11PoT0fZYwp7CxARZpq1dw9qEcfhdbBZoE6UePGMNpv4qkvv4Rnnimg+hljTIjk6jkoE2KPPJLnt9x8MyxZAn/7m9t/8UX3/O+NN+Zz3YwxJkSsBRXpDh+GXM5u/dJLcMUVvv1+/eCnnwqoXsYYU8AsQEWy6dPdePIRI3JVPDrazXJ+1llu/8gR6NYN1q0ruCoaY0xBsQAVySpUgE2b4O9/h99+y/VbvvgCypd3+9u2wZVXwp49BVVJY4wpGBagItm557p+uuPH87RKYcOGbhBgyZJuf9UquPpqNwzdGGMKCwtQke6VV6BSJZgxI0+rFLZvD++959ufM8cNP09PL4A6GmNMAbAAFemqVPENzXvwQdi7N9dvvfFGGDnSt//ZZ27OPlsu3hhTGFiAKgz69oVLLoEdO9xaG3nw4INw//2+/VGjXJ4FKWNMpLMAVRhERcGbb0KDBtC9e57eKgKvvQa9e/vy/vEPePxxC1LGmMhmD+oWFk2bwsqVbix5HkVFwbhxcOwYfPqpy3vlFbdcx7Bh+VpLY4zJN9aCKkz8g9OGDdmXCyImBv7zH+jRw5c3fLgLUNaSMsZEIgtQhdFjj8GZZ8LcuXl6W4kS8PHHgbNNDB8ODz9sQcoYE3ksQBVG0dGQluYGTxw+nKe3xsbCpEnQpYsv7/XX4c47bQi6MSayWIAqjIYOdfekfvstV+tGZVW6tHuQt2dPX96YMXDLLe6ZYGOMiQQWoAqj2Fi3xnt0tJsG6ZtvTukjPvoocEXejz920yL9+We+1dQYY06ZBajCqnVreP55t92nj5t0L49iYmDsWLj3Xl/ed9/BRRfB2rX5VE9jjDlFFqAKs0cfhY4d3QO8Q4ac0kdERbnnoryxDtxo9vPPh/nz86mexhhzCixAFWZRUfD+++4p3NdfP+WPEXG3sv7zH98Eszt3wqWXujxjjAkHC1CFXVISTJgACQmn/VE33+zmpK1c2e0fPepi33332eAJY0zoWYAqStLS3JO3W7ee8kdcfLHr2mvUyJf3f//nWlNbtpx+FY0xJrdCGqBEZLCIJIvIMREZl0O5viKSLiIH/VKHkFW0sHrySffk7Q03nFaTp359WLAgcBj63LnQqpVrYRljTCiEugW1BXgeGJuLsvNUNc4vzSrYqhUBDz8MNWrAjz+e8qAJr/h4+OQTN2dflOenZPt26NwZHnnEzetnjDEFKaQBSlUnqeoUYHcoz1tsJCa62WBLlHD9cuPGndbHibhgNH164C2uV1+FCy5wo/2MMaagRPI9qHNEZJeIrBaRZ0Qk6MzrIjLA022YvHPnzlDXMfJccAH8619ue8AAmDXrtD+yY0f4+Wf3EK/X0qWuy++112yKJGNMwYjUAPU90AyoCvQEbgYeCVZQVd9W1Taq2iYhH0ayFQkDBsADD0BqKlx3nZsS6TQlJsJXX7mGWWysyzt61PUqXnihC2DGGJOfIjJAqeofqrpWVTNU9WfgOeD6cNerUHn1Vbe2xvnnu+iSD0Rg8GBIToYWLXz5ixa51tQzz7igZYwx+SEiA1QQCki4K1GoREfDhx/Cl1+6EQ/5qFkzF5Sef973YG9amttv0gSmTLHlO4wxpy/Uw8xjRKQUEA1Ei0ipYPeWRORKEUn0bDcCngE+D2Vdi4QyZdyEe+CG3f31r/n2xG2JEm72iaVL3dx9XmvXwrXXuuU8bBCFMUVDWhps3Oiekfz0Uzc92iOPuBUQXnih4M4b6iXfnwaG+u3/BRguImOBFUATVd0AdALGiUgcsB2YALwY4roWLbff7uYtWroUPvjglJaOD6ZxY/jhBxg1Cp5+GvbscfnTp8PZZ8Mdd7iuv5o18+V0xph8dvgwbN4MmzZl/7p9O2RkBH9/hw6ntOpProQ0QKnqMGBYNofj/MoNAU7vQR4T6OGHXXffxIlQsSL8+9/uplI+iIqCe+6BXr3g2WfhrbfcD3N6Orz9tlsZZPBgePxxqFIlX05ZaB07doxly5Zl7q9fv/6EMjt37mTRokWZ+w0bNiQ+n7tpTfFw5IgLMBs2uBbQhg0u+QegvXtP7xybN+dPXYMRLUI3C9q0aaPJycnhrkbkmj3b9b0dO+Yixj//mW9Byt+yZW4Q4cyZgfnlyrn8++4rvoFqypQpXHfddQEB58CBA2T4/XkaHx+PeP5dDh06xAsvvMCjjz4a8rqayJaR4VbZyRp8/Pfz88mbxEQ3D0DNmoGvdetC+/an99kislhV25yQbwGqmPn6a7jmGncv6p573DNTUfl/K1IVvv3Wzb6U9Z+kdGnX9ffww+6HuzhJTU2lRo0a5PaZvdKlS7NhwwaqFNeIXozt23di8MnaCkpNPf3zlCjhAo03ZQ1ANWtC9eq+AVEFwQKU8fnmGxekjh1zXX333FNgp1J1o/qefhpWrAg8Fh0NN94IDz4IbU740Sy6Ro8ezYMPPsihQ4dyLBcbG8ugQYMYOXJkiGpmQuX4cdfFll3w2bgR9u8//fNER7sAU7s21Krle61Vyxd8qlQpkL9R88QClAk0bZq7QfSf//ievC1A6eluSfmXX4aUlBOPt2kDAwe6gFWmTIFXJ6xy24qy1lPhpOrWEM0p+Gzblj+PYlSpEhh8sm5Xq5Zv46EKlAUok7NDh9z9qAKODqrwv/+5QJX1HhVAhQpuBfs+faBlywK5RRYRTtaKstZT5Dp4MOfgs3Fj/kymXLp09oHH2woqKn/MWYAy2Tt+HLp3d0Hqyy/dKL8QWLTITZ308cfB/0M3aQK33uqetahdOyRVCpmTtaKs9RQeaWlu3bPsgs+GDac/6g1cl1r16tkHn9q13cKhRfUPtKwsQJnsrV8Pl1zi/gc2buyC1Jlnhuz0u3a5idffegvWrAlepm1b9wDw1Ve79aqKguxaUbGxsQwcOJDXXnstTDUrmlTdz5p/S8c/8Gzc6IJTds/75EWFCjkHn6QkNzjBOBagTM42boSuXeGXX6BSJfjsM/cEXghlZLgFEd97DyZNcg8QBtOsmQtUV1/t5gAsDH3swWTXirLW06k5cCD7wONN+TFXZMmSvi42b+DJOgihXLnTP09xYgHKnNz+/a4/7auv3BRJb7zhZkYPg4MH3ei/CRPcrBTZ/VVbqZJbDqRzZ5fOOKNwdYtkbUXZvafgjh93w6qDBR7v9p9/5s+5qlXL+d5P1arhH/VW1FiAMrmTnu6mfHj1Vbc/dSp06xbWKu3c6aoxZYobYJHTX8F16sDFF7slQC68EJo3901HGImytqKKW+tJ1QWWzZuDpy1b3OuOHfkz6q18+cDWT9btmjVDMqjVZGEByuTNu++6Nd+//DKi+tAOHXJByhustm3LuXyZMnDuuW4Ye4sWLjVqVLAPHeaVtxWVlpZWpFpPhw+7Ody2bs05AB05kj/ni431BZzsApDNGBWZLECZvFP19Zdt3QoLF7obPxFC1T38++23Ls2a5boGT6ZECTdCsHlz99qggUv164dn2K63FXXgwAE2btwY0a2nY8dc0Nm27eSvBw7k33n9R71lF4ASEgpX967xsQBlTl16urvBM2sWDBoEr7wSkQ9gpKbCTz/BvHm+tHFj3j6jRg0XrOrVC5zqxZsqVSqYX4KTJ09m3bp1PPjgg/n/4dk4ehR2784+7doVuL9zZ/7d5/FXtmzgVDvBUrVqkd1Va06PBShz6lR9C8Ckpbnf4OPGBS4EFaE2b3Zr2KSk+FKQCcRzLTbW/aVeubJ7it//tWJFiItzv3CzvpYt61puMTEu+W/HxLheVFX3t4B3Jnj/14wMN1DgyBEXWI4c8SX//cOHXctl3z435mX//uy386trLTsxMW6C0erVfYEmKenE4BMfby2f4s4ClDl9P/0Et90Gy5e7PpchQ2D4cChVKtw1y5N9+9yM68uWwerV8NtvLq1d6wKCyV50tAs6iYmuVZN12/+1YkULPCZ3LECZ/HHsGAwb5rr5MjLcjZzFi4tE/0tqKqxb54LVpk0npo0bc3ePq7CIjnYtP/9WoH/KmpeQ4Lo4bYi1yW/ZBajC/1vFhFZsLLz0khss0bevmxW9CAQncN1u3gET2Tl4MPD+jP/rvn1ulOHBgy75bx8+7HpHU1PdqzelpgYumRAd7QJAsNeYGDc/mzeVKhV8Oz7epfLlfdvB8sqWtRaOiXCqWmRS69at9XQB6i6Lz1VXXaWAfvHFF5l5o0aNUkDvvPPOzLzNmzcroNWrVw94f6tWrRTQ5OTkzLyhQ4cqoEOHDs3MS05OVkBbtWoV8P7q1asroJs3b87Mu/POOxXQUaNGZeZ98cUXCuhVV10Vku9UEnTx3LmZeROvv17vA33u6acL7XcK179TRkbR+06q9p2Kw3fKD0CyBvmdXjT+9DVhcRxQ7wNFBw/S9euv6QXseeMN98BRz57hrF6hYi0ZY05k96BM/lB10z08+iisWuXyLrzQratxySXhrZsxJqJldw/Kbnea/CHiluz4+Wd48003Ydm8edCuHVx6qXt60xhj8sAClMlfMTFw993w++8wdKi7K79tmxsSZowxeWAByhSMcuXccPT162HiRN98fhs2uMnxxo/Pn2VHjTFFVkgDlIgMFpFkETkmIuNOUvZBEdkmIvtEZKyI2BzDhVH58u5ZKa/RoyE52Q1Rr13bPehr3X/GmCBC3YLaAjwPjM2pkIh0AR4HOgF1gTOA4QVdORMCTz3lZkpv3tytoTBsmJvkrmdPNz25McZ4hDRAqeokVZ0C7D5J0T7AGFVdrqp7gRFA3wKungmFUqVc62npUpg50z3wm5HhltAdN85XrgiNLjXGnJpIvQfVFEjx208BEkWkctaCIjLA022YnHXpbBPBRNyS8lOmuDmEXnwRBg/2Hf/0U7ee+8svu0nyjDHFTqQGqDhgn9++d7tc1oKq+raqtlHVNgkJCSGpnMlnSUnwxBOBs6NPmQJLlrjVfc84A84/H0aOdLO7GmOKhUgNUAcB/7Uvvdv5uASaiWhjxsDkyXDTTW7tqYUL3ezpDRvCjTeGu3bGmBCI1AC1HGjht98C2K6qJ7t3ZYqKUqXcRLQffuhWyvv4Y+jd202n3cLvR+Onn9wDwq+/7hZ7ysgIW5WNMfkrpHPxiUiM55zRQLSIlALSVDUtS9H3gHEi8gGwFXgaGBfKupoIUqYM3HCDS2lpgc9P/e9/boqlqVPdfuXKbuaKSy+Ftm3daEGb6M6YQimkc/GJyDBgaJbs4bhh5yuAJqq6wVP2IeAxoDTwGXC3qub4ZKfNxVcMbd0K06fDd9/BjBlu4SavatVgyxZfgJoxA84+203DZIyJGLZgoSn6VGHNGhesZs92S7r+61/u2P79UKGCK1OnDpxzDrRs6boLW7Z0edbSMiYsLECZ4u2PP+COO2DRIreSYFZTp0K3bm77p59cQGvc2LW2LHAZU6BsRV1TvJ1xhnswOC3NDVVfutSllBQ3nN1/4MVrr8EHH7jtChWgUSNfOu88d3/LGFPgLECZ4iUmBpo0cemWW1xe1l6Ehg3hggtg5Ur480+YP98lcDNfeAPUzp2u1XXGGVCvnktnnAF160KNGm79dWPMKbMAZUzWLrxnnnFJ1c0XuGqVLzVp4iu3Zo3rMly0KPjn/vij7+HjTz+FFSvcvIM1arjXpCTXQrMuRGOCsgBlTHZEIDHRpfbtTzx+9tkwZ467v7V2rXv94w83ddPmzVC9uq/sp5+6ZUeyKlECrrwSPv/c7aemwtNPu3tf3nMnJrr9SpWgZMmC+a7GRCALUMacqrJl3bNWbdueeCwjI7BldNNNcOaZbhi8N23dCgcOBJbbuRNeeSX7c06Z4roZwT3EPGWKC1yVKwe+Vq3qpocyphCzAGVMQYjKMknLNde4lNWRI3D0qG+/VCk3ce727S7t2OF73bPHdQl6LV7sZtgIpk4dWLfOt1+9umudxce7Nbq8KT7ezdBxxRWu3Lp1MG+e71i5chAX50tlyliXpAkZC1DGhFPp0oGDKSpVchPnBqMaOKCjb19o08YFrt27A1+rVAl8365dbgTj7iCzhZ17ri9Aff899OmTfX0PHnQtR4Cbb3b31cqW9QUw7/bFF7vAB26gydSpLrh5v69/qlXLBWZjsrAAZUxhIRLYemnWzKXcvO/gQfds1759Lvlvn3uur2ytWtCrl+/YoUPuvQcPwuHDgcH0119h2bLg50xL8wWotWvh1luzr9/8+b7uyEGDYOzY4IGsWTN3zGvAAIiN9R2PjfWljh19A1rWr4flywOP+yf/h7QzMk5s/ZqwsQBlTHEQGwsJCS7lxDuPYW5MnuxaR94AdvCgL6A1bOgrV66cG9J/+LDr0vQm736838IFhw+7Ls+jR2Hv3sDz+QfntDQYPTr7uo0e7QtQX33lAl92/CcYbt3aBd1ggax3bxjqmalt1Sq4/343yCVYevZZF+zB3SdcujR4uerVoUcPV07V1TW7z6xVy82O4r1Ohw+7/Oho9/hETIzbLkJdsBagjDGnpk4dl06mfn3fg88nM2YMvPFG8EAWGxtY9u23A8sdO+ZLTZv6ytWsCV26BB73Jgj8hZ6a6gKW9zP97drl2969201UnJ377gsMUOPHBy93/vm+AJWa6mbmz86YMXD77W577Fi4997g5WJjA+9rtmvnWrveIOYNZDEx8Je/wJNPunK//AL33BNYzj+NHAm1a7uy777rWr733pu7VvwpsgBljIkcUVHuXlWZMjmXi4mBO+/M3Wf26OELAifzyy++GfOzJv+WXpMm8PXXLqhkTcePu6Dodc01LpAHK1u3rq+cKnTtmv1nVvZbULxkSXe/Mi3Nl9LTXcragtq1yw2yCcY//88/3WMT2Rk2zLf93XcwYQJcd12BBiibi88YY4oKVRekYvzaHnv2uACXnh4Y0NLSXJdhUpIrt2+f697MWsb7vssv9wXpmTNdq6xbN19L8TTYZLHGGGMiUnYByoarGGOMiUgWoIwxxkQkC1DGGGMikgUoY4wxEckClDHGmIhkAcoYY0xEsgBljDEmIlmAMsYYE5GK1IO6IrITWH+aH1MF2HXSUkWfXQcfuxY+di0cuw4++XEt6qjqCTMZF6kAlR9EJDnYE83FjV0HH7sWPnYtHLsOPgV5LayLzxhjTESyAGWMMSYiWYA60dvhrkCEsOvgY9fCx66FY9fBp8Cuhd2DMsYYE5GsBWWMMSYiWYAyxhgTkSxAGWOMiUgWoDxEpJKITBaRQyKyXkRuCXedCoKIDBaRZBE5JiLjshzrJCKrROSwiMwUkTp+x0REXhaR3Z70iohIyL9APhKRWBEZ4/n3PiAiS0TkSr/jxeZ6iMgEEdkqIvtFZLWI9Pc7Vmyug5eINBCRoyIywS+vWF0HEZnluQYHPelXv2OhuRaqaskNFPkQmAjEARcD+4Cm4a5XAXzP64BrgDeBcX75VTzf+QagFPA3YL7f8buAX4GaQA1gBXB3uL/PaV6LssAwoC7uj7WrgAOe/WJ1PYCmQKxnuxGwDWhd3K6D3/f6H/ADMMGzX+yuAzAL6B8kP2TXIuwXIRKS5xfVceAsv7z3gb+Gu24F+J2fzxKgBgBzs1yTI0Ajz/5cYIDf8Tv8fyiLSgKWAT2L8/UAGgJbgV7F8ToANwEf4/548Qao4ngdsgtQIbsW1sXnnAWkq+pqv7wU3F+VxUVT3HcGQFUPAWvwXYOA4xTB6yMiibifheUUw+shIv8WkcPAKlyA+i/F7DqISDzwHPBwlkPF6jr4eUlEdonIjyLSwZMXsmthAcqJwzVZ/e0DyoWhLuFysmuQ9fg+IK6w97N7iUgJ4ANgvKquohheD1UdiPt+lwCTgGMUv+swAhijqhuz5Be36wDwGHAGrpvubeBLETmTEF4LC1DOQSA+S1487n5EcXGya5D1eDxwUD1t+MJMRKJwXbrHgcGe7GJ5PVQ1XVXn4O4f3EMxug4i0hLoDLwe5HCxuQ5eqrpAVQ+o6jFVHQ/8CHQlhNfCApSzGogRkQZ+eS1wXT3FxXLcdwZARMoCZ+K7BgHHKSLXx/NX3RggEeipqqmeQ8XyeviJwfd9i8t16IAbILNBRLYBQ4CeIvITxes6ZEcBIZTXItw34iIlAR/hRvKVBdpSdEfxxeBG3ryEazWU8uQleL5zT0/eywSOzLkbWIlr7id5fuAK9Sglz/d6C5gPxGXJLzbXA6iKGxgQB0QDXYBDwNXF7DqUAar5pVeBTz3XoNhcB8/3qeD5OfD+fujt+ZloGMprEfYLESkJqARM8fwjbABuCXedCuh7DsP9JeSfhnmOdcbdID+CG8FT1+99ArwC7PGkV/DM5VhYE1DH8/2P4rolvKl3cboenl84s4E/gf3Az8CdfseLxXUIcl2G4RnFV9yug+dnYhGu2+5P3B9xl4X6WthkscYYYyKS3YMyxhgTkSxAGWOMiUgWoIwxxkQkC1DGGGMikgUoY4wxEckClDHGmIhkAcqYIkpEVESuD3c9jDlVFqCMKQAiMs4TILKm+eGumzGFRUy4K2BMEfYtcGuWvOPhqIgxhZG1oIwpOMdUdVuWtAcyu98Gi8hXnmWz14vIX/zfLCJni8i3InJERPZ4WmXls5TpIyI/i8gxEdkuIuOy1KGSiHwiIodE5I+s5zAmklmAMiZ8hgNfAC1x6+28JyJtAESkDPANbm7A84BrgYuAsd43i8hdwCjgXaA5bimErLNGPwt8jptReiIwVkTqFNg3MiYf2Vx8xhQAT0vmL7iJaP29oaqPiYgC76jqnX7v+RbYpqp/EZE7cbNp11TVA57jHYCZQANV/V1ENuEmM308mzoo8FdVfcKzH4ObDHaAqk7Iv29rTMGwe1DGFJzvgQFZ8v70256X5dg8oJtnuzGwzBucPOYCGUATEdmPW85gxknqsMy7oappIrITt7yGMRHPApQxBeewqv5+iu8V3FIgwXgXjsuN1Cz7inXtm0LCflCNCZ8Lguyv9GyvAFqISDm/4xfh/s+uVNXtwGagU4HX0pgwsRaUMQUnVkSqZclLV9Wdnu3rRGQRbsG363HB5nzPsQ9wgyjeE5FngYq4ARGT/FplLwCvi8h24CvcirCdVHVkQX0hY0LJApQxBaczsDVL3magpmd7GG7Z7H8CO4F+qroIQFUPi0gX4O/AQtxgi8+B+70fpKpvishx4GHcstt7gP8W0HcxJuRsFJ8xYeAZYXeDqn4a7roYE6nsHpQxxpiIZAHKGGNMRLIuPmOMMRHJWlDGGGMikgUoY4wxEckClDHGmIhkAcoYY0xEsgBljDEmIv0/+yhYT8ErZVIAAAAASUVORK5CYII=)]

np.random.seed(42)
m = 100
x = 6 * np.random.rand(m, 1) - 3
y = 2 + x + 0.5 * x**2 + np.random.randn(m, 1)

x_train, x_val, y_train, y_val = train_test_split(x[:50], y[:50].ravel(), test_size=0.5, random_state=10)
poly_scaler = Pipeline([('poly_features',PolynomialFeatures(degree=90,include_bias=False)),('std_scaler',StandardScaler())])
x_train_poly_scaled = poly_scaler.fit_transform(x_train)
x_val_poly_scaled = poly_scaler.transform(x_val)
sgd_reg = SGDRegressor(max_iter=1,tol=-np.infty,warm_start=True,penalty=None,learning_rate='constant',eta0=0.0005)
minimum_val_error = float('inf')
best_epoch = None
best_model = None
for epoch in range(1000):
    sgd_reg.fit(x_train_poly_scaled,y_train)
    y_val_predict = sgd_reg.predict(x_val_poly_scaled)
    val_error = mean_squared_error(y_val, y_val_predict)
    if val_error < minimum_val_error:
        minimum_val_error = val_error
        best_epoch = epoch
        best_model = deepcopy(sgd_reg)

#画图
sgd_reg = SGDRegressor(max_iter=1, tol=-np.infty, warm_start=True,
                       penalty=None, learning_rate="constant", eta0=0.0005, random_state=42)

n_epochs = 500
train_errors, val_errors = [], []
for epoch in range(n_epochs):
    sgd_reg.fit(x_train_poly_scaled, y_train)
    y_train_predict = sgd_reg.predict(x_train_poly_scaled)
    y_val_predict = sgd_reg.predict(x_val_poly_scaled)
    train_errors.append(mean_squared_error(y_train, y_train_predict))
    val_errors.append(mean_squared_error(y_val, y_val_predict))

best_epoch = np.argmin(val_errors)
best_val_rmse = np.sqrt(val_errors[best_epoch])

plt.annotate('Best model',
             xy=(best_epoch, best_val_rmse),
             xytext=(best_epoch, best_val_rmse + 1),
             ha="center",
             arrowprops=dict(facecolor='black', shrink=0.05),
             fontsize=16,
            )

best_val_rmse -= 0.03  # just to make the graph look better
plt.plot([0, n_epochs], [best_val_rmse, best_val_rmse], "k:", linewidth=2)
plt.plot(np.sqrt(val_errors), "b-", linewidth=3, label="Validation set")
plt.plot(np.sqrt(train_errors), "r--", linewidth=2, label="Training set")
plt.legend(loc="upper right", fontsize=14)
plt.xlabel("Epoch", fontsize=14)
plt.ylabel("RMSE", fontsize=14)

plt.show()

Answers to questions 1, 2, and 4
The purpose of early termination is to prevent overfitting. If we only need to return the parameters that make the validation error the lowest, we can obtain a model with a lower error in the validation set.
Write picture description here
Figure 1. Learning curve (horizontal axis is training rounds, vertical axis is negative log-likelihood)
Notes to Figure 1:
The blue curve indicates how the loss on the training set changes with the training rounds.
The red curve shows how the loss on the test set changes with the training rounds.
Note: The red curve is the curve obtained by testing once after each training epoch.
It can be seen from the figure that the test error gradually decreases in the first few epochs, but after training to a certain epoch, the test error increases slightly again. This shows that overfitting has occurred at this time.

Overfitting is something we don't want to see, we can use early stopping (early stopping)
to prevent overfitting from happening.

Early termination means that the training is stopped before the test error starts to rise, even if the training has not yet converged (that is, the training error has not reached the minimum value).

First of all, we need to save the current model (network structure and weights), train num_batch times (ie one epoch), and get a new model. Use the test set as the input of the new model for testing. If we find that the test error is larger than the test error obtained last time, we will not terminate the test immediately, but continue to train and test for several epochs. If the test error still does not decrease, then we consider that the test is in Stop the last time the lowest test error was reached. The specific algorithm can be found in "deep learning"

Answer to Question 3
This question is to answer why early stopping works as a regularizer .
First, we expand the loss function in the neighborhood of ω∗ω∗ with Taylor expansion (only expanded to the second order), then we have

J′(ω)=J(ω∗)+1/2(ω−ω∗)TH(ω−ω∗)J′(ω)=J(ω∗)+1/2(ω−ω∗)TH (ω−ω∗)

where H is the Hessian matrix. The reason why there is no first-order derivative information here is because ω∗ω∗ is the optimal solution, and in the neighborhood of ω∗ω∗, the gradient can be approximately considered to be 0.
Finding the gradient of J′(ω)J′(ω), we get

∇ωJ′(ω)=H(ω−ω∗)∇ωJ′(ω)=H(ω−ω∗)

We initialize the parameter vector ω(0)ω(0) to origin 0. According to the gradient descent method, the following formula can be obtained:

ω(τ)=ω(τ–1)−α∇ωĴ (ω(τ−1))ω(τ)=ω(τ–1)−α∇ωĴ(ω(τ−1))

ω(τ)=ω(τ−1)−αH(ω(τ−1)−ω∗)ω(τ)=ω(τ−1)−αH(ω(τ−1)−ω∗)

ω(τ)−ω∗=(I−αH)(ω(τ−1)−ω∗)ω(τ)−ω∗=(I−αH)(ω(τ−1)−ω∗)

Decompose H into eigenvalues: H=QTπQH=QTπQ, where Q is an orthonormal matrix and π is a diagonal matrix.
So

ω(τ)−ω∗=QT(I−απ)Q(ω(τ−1)−ω∗)ω(τ)−ω∗=QT(I−απ)Q(ω(τ−1)−ω ∗)

Q(ω((τ))−ω∗)=(I−αp)Q(ω((τ−1))−ω∗)Q(ω((τ))−ω∗)=(I−αp) Q(ω((τ−1))−ω∗)

where αα is small enough to ensure that |1−απi|<1|1−απi|<1.

Qω(τ)=(I−(I−ap)τ)Qω∗Qω(τ)=(I−(I−ap)τ)Qω∗

When analyzing the L2 regular term, Qω̃ = (I−(π+εI)−1ε)Qω∗Qω̃=(I−(π+εI)−1ε)Qω∗ From the comparison of the above two formulas, we can see that if the following
formula Established:

(π+εI)−1ε=(I−απ)τ(π+εI)−1ε=(I−απ)τ

Then L2 regularization and early termination can be considered equivalent. Further, there are

ε/(πi+ε)=(1−απι)τ/(πι+ε)=(1−απι)τ

Taking the logarithm on both sides, we can know:

log(ε/(πi+ε))=τlog(1−αpi)log⁡(ε/(πi+ε))=τlog(1−αpi)

As a simple approximation:

−log(1+πi/ε)=τlog(1−αpi)−log⁡(1+πi/ε)=τlog⁡(1−αpi)

−πi/ε=−ατπι−πι/ε=−ατπι

(Comparing the first non-constant term of the Taylor expansion, the Taylor expansion is unique)
Thus

t=1/yr=1/yr

The above derivation shows that early termination can play the role of regularization.

logistic regression

estimated probability

The estimated probability of the logistic regression model
p ^ = h Θ ( x ) = σ ( x T Θ ) is denoted as σ ( . ) in logic, which is a sigmoid function (that is, an S-type function), and outputs a value between 0 and 1 The number \widehat{p} = h_{\Theta}(x) = \sigma(x^T\Theta)\\ logic is recorded as \sigma(.), which is a sigmoid function (that is, an S-type function), and outputs an intermediate a number between 0 and 1p ​=hTh(x)=s ( xT Θ)The logic is denoted as σ ( . ) ,Is a s i g m o i d function (i.e. S -type function), outputting a number between 0 and 1 Logical
function
σ ( t ) = 1 1 + exp ( − t ) \sigma(t) = \ frac{1}{1+exp(-t)}s ( t )=1+exp(−t)1​
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-AhRbsEOJ-1658889211623)(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAoAAAADQCAYAAACX3ND9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwSUlEQVR4nO3daXgUVfr38e9NwqJsgigqEhFlFEUFxXHHUlGWv4M4OMq4sKiDKKigKCIKKiMyDuMDOq64oKIOLuiIyiIqIDoqIFFBaVSEALKFJSQhG8l5XlTS6YQsHeikk/Tvc111dZ+q09V3V046d05VnWPOOUREREQkdtSJdgAiIiIiUrWUAIqIiIjEGCWAIiIiIjFGCaCIiIhIjFECKCIiIhJj4qMdQDhatGjh2rRpE+0wRCRGBQIBAI477rgoRyIisWrp0qXJzrlDIrW/GpEAtmnThiVLlkQ7DBGJUZ7nATB//vyoxiEiscvM1kZyfzoFLCIiIhJjlACKiIiIxBglgCIiIiIxRgmgiIiISIxRAigiIiISY5QAioiIiMQYJYAiIiIiMUYJoIiIiEiMCWsgaDMbCgwATgLecM4NKKPucGAkcADwDnCzcy4rf1tz4AXgEiAZGOWce30/4t9LVlYW27dvJzU1ldzc3EjuWqTKxMXF0bhxY5o3b079+vWjHY6IiNQy4c4E8jvwd6AbfmJXIjPrBtwDXJj/mneBB/PXATwJZAMtgY7Ah2b2nXNuxb4EX1xWVhZJSUk0a9aMNm3aULduXcwsErsWqTLOOXJycti1axdJSUkkJCQoCRQRkYgK6xSwc26Gc+49YFs5VfsDLzjnVjjndgDj8HsOMbOGQB/gfudcmnNuEfA+cF157x8IBJg6dSoAOTk5eJ7HtGnTANi9ezee5zF9+nS2b99OkyZN2LZtG+np6ZgZOTk5BAIBdu7cGXx9IBAgJSUFgOzsbAKBALt27QL8JDIQCJCamgpAZmYmgUCAtLQ0ADIyMggEAqSnpwffPxAIsHv3bgDS09MJBAJkZGQAkJaWRiAQIDMzE4DU1FQCgQBZWVkA7Nq1i0AgQHZ2NgApKSkEAgFycnIA2LlzZ5Hyjh07CAQC7NmzB4Dt27cTCASCvZ3btm0jEAiQl5cHQHJycnAeU4CtW7cWKW/ZsoVVq1YFy5s3b+bnn38Oljdt2sQvv/wSLG/cuJFff/01WP79999ZvXp1sLxhwwZ+++23YHn9+vWsWbMmWF63bh1r1xbOZpOUlERSUlKwvHbtWtatWxcsr1mzhvXr1wfLv/32Gxs2bAiWV69eze+//x4s//rrr2zcuDFY/uWXX9i0aVOw/PPPP7N58+ZgedWqVWzZsiVYDgQCbN26tUg5OTkZgLy8PAKBANu2+b8Gubm5BAIBtm/fDsCePXsIBALs2LEDYJ/bXlpaGvXq1aNRo0ZkZmby/fffA7B8+XI8z2Px4sUAJCYm4nkeiYmJACxevBjP81i+fDkAX375JZ7nBX/eCxYswPO84M9r3rx5eJ4XPN6zZ8/G87zg8Zo5cyae5wU//4wZM/A8Lxj/9OnT8Twv2PanTZuG53nBtjp16tTgFGoAU6ZMoWvXrsHyU089RY8ePYLlyZMn06tXr2B54sSJ9OnTJ1ieMGECffv2DZbHjRvHtddeGyyPGTOGgQMHBsujRo1i0KBBwfKIESMYMmRIsDxs2DCGDRsWLA8ZMoQRI0YEy4MGDWLUqFHB8sqVKxkzZkywfO211zJu3LhguW/fvkyYMCFY7tOnDxMnTgyWe/XqxeTJk4PlHj168NRTTwXLXbt2ZcqUKcGy53lhfe+B/73heR4zZswA/N97z/OYOXMm4P8ee57H7NmzAf/30PM85s2bB/i/R57nsWDBAsBv957n8eWXXwJqe9Fqe7m5kJkJ1157M3fdNZ6tW2HjRrj88tu4444n+Pln+Okn6N79Lm677UWWLoVvvgHPu5dbbnmDBQvg00/hrLPGMHjwe3z0EcycCaee+iCDBs3inXfgrbegQ4e/c+ONn/Laa/Dqq3D88Y9w442LePFFePbZPRx33D+5/vqveeYZmDQpm3btJjNw4Lc8/jhMmJDBscf+mwEDvudf/4IHH0znmGOeZsCAH3n0UbjvvlTatn2OgQNXMX483H33To4++nkGDvyVceNg+PDttGnzIgMHruGBB2Do0GTatJnKwIHrGDMGbrppC0cd9QrXX/87990HN964iaOOepUbbtjMvffCgAEbSUh4jRtv3Mo998B1160nIeF1/va3bYwcCVdfnURCwuvcdNNORo6Eq65aQ+vWbzB48C7uuguuuOI3Wrf+DzffnM6IEXD55b/SuvV0hgzZzYgR0KvXz7RuPZ2hQ7O480649NIArVtP59Zbc7jjDujR4yeOPPJNhg3L5Y47oFu3FRx55JvccQfccQdcfPEPJCS8w/DhMHw4XHjhd7Rp826w7HnLaNv2v8Fyly5LOPbYDxg2DIYNg3POWcwf/vARIc0lYiI9F/CJwH9Dyt8BLc3sYCAByHXOrSq2/fySdmRmg4BBQNi9H6mpqbRu3Tr4B1ekpouPjyc+vkZM2S1Sa+XlGdnZB7BuHaSmwvr1R7BpUx0++ADS0yExsRNbtvyBf/wDMjLgs8+6k5a2h0GD/PI331xHdrZxySWQlQWrVg1jz566zJ0L2dmwefM48vLiefFFv5yR8SR5eXEU/j/xNACF/088DsD/+38F5X8yZw488URBeTwLFsDTTxeUH+Krr+DZZwvKY1m2DAr/37iPFSvghRcKyqMIBArK8cBdrFoFL70EUA+4nV9+Af//kwOAoRT2CzQEbqawX6AxMIjCfoGDgBvzXwvQHLg+pNwCGBBSPhTol//eAIcB1/HiiwXlw4FrQmI/Eria558vKCcAV/PccwXlNkCbkGNxNHA0zzxTUD4GOIbC/83aAe148smC8nHAcfz73wXl9kB7Cv+3OxE4MeRncxJwEpMmFZRPAU4JKXcCOoWUOwOE7O/0YuXIMedc+JXN/g4cWdo1gGb2KzDEOTc7v1wX/5Tv0UBr4C3n3GEh9f8GXOOc88p6386dO7slS5aUG99PP/3E8ccfr9O+Ums451i5ciXt27ePdigxraA3af78+VGNQ/adc7BzJyQn+8u2bUWf79zpLzt2FD7fudNP+PI7GqucGdSt6y/x8YWPoUtcXOFjwRJarlOn6LaCcp06pS9mRR9Lel6wFC+HLgWfIZxtoeWC52WtK+mxrOfFj+u+1g9n/f7WLa3+8OG21DnXuWJ7Kl2kuxbSgCYh5YLnqSVsK9ieGskAlPxJbaL2LFK+Xbtg3bqiy8aNsHkzbNrkL5s3+71r+8IMGjWCxo2hSRP/ecOGhcuBBxY+HnDA3kuDBlC/fuFj6FKvXtGlbt3Cx7i4yB4nqdmGD4/s/iKdAK7A7998M798CrDZObfNzDKBeDNr55z7OWR7RG4AERGR2mnPHlizBn79tXBZvdp/XLvWTwDD0bgxHHIItGgBBx9c+HjwwdC8ORx0kL80a+Y/Nm3qJ3wHHuj3dInUJuEOAxOfXzcOiDOzBsAe59yeYlVfAaaa2WvARuA+YCqAcy7dzGYAD5nZjfh3AV8GnB2BzyEiIjXcnj2wahWsWAE//li4BAKQf39HiQ44AFq3LlyOPBJatYLDDoOWLQsfDzyw6j6LSHUXbg/gfcDYkPK1wINm9iLwI3CCcy7JOTfbzB4FPqNwHMDQ190CvAhswb+j+OZIDQEjIiI1R3a2n+gtXQrffus/fv+9f9drSVq3hmOPhbZt4ZhjCpc2bfzeO10tIVIxYSWAzrkHgAdK2dyoWN3HgMdK2c92oHfY0YmISK2wcyd88QV8/jksWgSLF5d8TV6bNtChA5x4Ipxwgr8cf7x/3Z2IRI7GlxARkYhLS/PHoJs7FxYuhOXL/TtxQ7VrB6edBqee6j926uRffycilU8JoEiIhQsXMnHiRJYuXcrvv//OSy+9xIABA6Idlki155x/SnfWLJg92+/pC71ur149OP10OPdcOO88OPtsJXsi0aQEUCREWloaHTp0oF+/fvTr1y/a4YhUa87B11/Dm2/C22/7w68UqFMHzjwTuneHCy7wk78DSp1IVESqmhJAkRA9e/akZ8+eAOr5EymBc/71e2++6U8jFjKTIy1b+glf9+5w8cX+8CoiUj1pZCMpol+/fhx66KHBuY7DsXTpUsyMFwrn4hGRWmbjRhg/3r8T94wz4F//8pO/I4/0B6j93//g99/96cH69lXyJ1LdKQGUoCVLljBt2jTuueceGjZsuNf2xx57DDPj9ddfL7L+tNNOo3fv3tx3332kpaVVVbgiUslyc/1r+v78Z38YltGj/QGYjzgCbr/dv6t37Vp47DH/dK8GSxapOfTrKkH33nsvTZo04eabby5x+7fffgv4CV9xo0aNYtOmTTz++OOVGqOIVL4dO/zevrZtoWdPePddf/3ll/sJYVISTJrk38ihpE+kZtKvrgCwatUq5s2bx5VXXskBpVypvXTpUho1akS7du322vbHP/6R448/nmeffZbc3NzKDldEKsH69XDnnZCQ4Pf2JSXB0Uf7yeC6dTBjhn99n+aoFan5lADWYmlpaTz00EN06tSJxo0bY2YlLlu2bOHFF1/EOcdVV121135GjhyJmbFy5UrS0tKIi4sLvnbatGnBen379iUpKYl58+ZV5ccUkf30008wcKDf4/fYY/4YfhdfDHPmwC+/wKhRcPjh0Y5SRCJJdwHXUlu2bOH8889n5cqVnHzyyQwePJisrCzeeustNm3aRN26dUlISKBFixYceuihzJs3j7i4OM4888y99nXaaafRv39/Xn75Zc4++2wuvvji4Lbzzz8/+Pycc84B4OOPP6Zbt26V/yErQVpaGr/88gsAeXl5JCUlkZiYSPPmzUlISIhydCKRFQj4PX3vvOOX69SBq66Cu+/2B2cWkdpLCWAtdfXVV7Ny5UruvvtuJkyYgOVPlHnXXXfRrl07cnNz+eqrr2jRogXp6ekkJibSvn37Em/+uPLKK9m5cycvv/wy/fv3Z9CgQSW+5+mnnw74gymXZ9KkSezcuTPsz9OxY0d69+4ddv19tWTJEi644IJgeezYsYwdO5b+/fszderUSn9/kaqwcSM8+CA8/7x/o0f9+n4P4IgR/vy6IlL7xUwCWFMmCi8+VdK++Pjjj/nkk08455xzeOSRR4LJH0Dr1q0577zzmDdvHomJiXTt2pUNGzaQm5vL4WWc4ym4AeTUMroFmjZtSoMGDUgKHRisFJMmTWLt2rVhf6b+/ftXSQLoeR4uEj8EkWpo1y745z/907y7d/s9fn/7G4wdC61aRTs6EalKMZMAxpKC6/KGDx9OnRJu0WvatCngn+IE2LZtGwDNypiX6dtvv6Vu3bqcdNJJZb538+bN2bx5c7kxrlmzptw6IhIZeXkwZQrcdx8kJ/vrevf2b+5o3z6qoYlIlMRMAhhLnTqff/45derUoXv37iVuX79+PQDHHnssQPCu38zMzBLr79mzhx9++IETTjiB+vXrl/neGRkZpd5FLCJV7/vvYfBgf6BmgHPOgUcf9YdwEZHYFTMJYKzIy8tj7dq1HHrooSVez7d582YWL17M0UcfTdu2bQE49NBDgcKewOJ+/PFHMjMzyzz9W/DeO3fu5Oijjy43zkhdA2g15dx+OXTaWSItPd2/zu+xx/zr/A4/3B+77y9/qTmXxIhI5VECWMsUJESpqank5eXtdQr40UcfJS8vj5tuuim47vDDD+eQQw4hEAiUuM/ExEQAOnXqVOZ7BwIBnHN07Nix3DgjdQ2gEieRvX3wAQwd6s/SYeY///vfIf/qDxERjQNY25gZp5xyCunp6bzxxhtFtr399ttMmjSJ448/nttvv73Ia7p06UJycnJwCJRQBT2DTZo0KfO9v/rqK4Aid9GWZs2aNTjnwl5q0x24a9as4b///W+0w5BaKCUFrrsO/vQnP/nr2BG++gqeeELJn4gUpR7AWmjMmDH8+c9/ZuDAgcyePZvWrVuzePFi5s2bR7t27fjoo49o0KBBkdf06dOHd955hzlz5gSvDSxQMPXb6NGjWb58OQ0bNuTEE0/kL3/5S5F6c+fOJS4ujssuu6xyP2ANN2fOHJKTk3WcJKI+/9xP/tauhQMO8Hv8brsN4vUtLyIlUA9gLdS7d2/effddOnfuzIwZM5g0aRJbt27l4Ycf5ttvvy3xGr0+ffrQsmVLXnnllb22denShSeeeIKGDRvyxBNP8OCDDwZPCxdISUnhvffe49JLL6V169aV9dFqvAULFjBq1CimTp1Kx44dSUlJiXZIUsPl5Ph393qen/yddhosWwZ33KHkT0RKpwSwlrrsssv48ssvSU9PZ/fu3SQmJnLvvffSqFGjEuvXq1eP22+/nW+++YZly5bttX3o0KEEAgEyMzNxzvHwww8X2f7KK6+QmZnJnXfeWSmfpywLFy6kV69etGrVCjOr1qeLzz//fE4++WTmzp1LYmJicEgekX3x88/+Xb0PP+yPdDBqFHz5JRx3XLQjE5HqTgmgBA0fPpyEhATGjBlToddlZGTwyCOP0KdPH84777xKiq50aWlpdOjQgcmTJ+/TEDQDBgzggQceiFg8HTp0KHFZt24d4F8D2KZNm4i9n8SmN97wr/FbvBgSEmD+fH9cv3r1oh2ZiNQEOkEgQQ0aNODVV1/ls88+Iz09vcRhZEqyZs0aBg0axIABAyo3wFL07NmTnj17AlR6DBs2bGDUqFF8+OGH5ObmctFFF/HUU0/RsmXLYJ3ly5eX+vr169dz2GGH1Zrha6Tq7dkDI0f6w7sA9O0LTz8NBx0U1bBEpIZRD6AU0aVLF8aOHRt28gfQvn17HnjggVrfq/Xbb79x6qmn0qpVKxYtWsT8+fNJTk5m8ODBYe9j3bp1HHHEEZUYpdRmW7fCJZf4yV98PDz5JLz+upI/Eam4sBJAM2tuZu+aWbqZrTWzq0up94yZpYUsWWaWGrJ9vpllhmwveeA5kWpo8ODB3HDDDTzyyCO0b9+ejh07cv/99/PJJ5+EvY8TTjiBtWvXctJJJ/HDDz9UYrRS2yxd6t/g8dln0LKl/3jLLRrUWUT2TbingJ8EsoGWQEfgQzP7zjm3IrSSc24wEOwOMbOpQF6xfQ11zj2/rwGL7K/x48czfvz4YDkrKwszY+LEicF1s2bNKnI9Y1JSEnPnzuXzzz/n8ccfD67Pzc3lwAMPDPu9mzZtytKlS/fzE0iseflluOkmyMqCM8+Et9+GVq2iHZWI1GTlJoBm1hDoA3RwzqUBi8zsfeA64J4wXndphGIViYjBgwdz5ZVXBssjR46kVatW3HbbbcF1rYr9dU1MTKRJkyYlJm/1dNW9VJK8PBg9GiZM8Ms33QSTJ0M5U3KLiJQrnB7APwC5zrlVIeu+A84v53V9gK3AwmLrHzGzCUAAGO2cm1/Si81sEDAIICEhIYwwRcLTvHlzmjdvHiw3btyY5s2b7zUAdqi6deuSnp7OYYcdVupQOiKRlJ0N118Pr70GcXH+9X4hMziKiOyXcK4BbAQUH602BWhczuv6A6+4opO1jgTaAq2A54CZZnZMSS92zj3nnOvsnOt8yCGHhBGmxKq0tDQSExNJTEwkLy+PpKQkEhMTSUpKith7nHnmmTRr1ozrrruOZcuW8euvv/Lxxx8zZMgQ8vKKX+Ugsn927oTu3f3kr1Ejf25fJX8iEknhJIBpQPFJYJsAqSXUBcDMWuP3EBaZVsI597VzLtU5l+Wcexn4AuhZsZBFilqyZAmdOnWiU6dOZGRkMHbsWDp16lTh8QzL0qxZM2bNmkVKSgoXXHABHTt2ZMSIERx55JHUqaOb6SVy1q2Dc8/1b/I47DBYuNBPBkVEIimcU8CrgHgza+ec+zl/3SnAijJe0w/40jm3upx9O0D3sMl+8TyPoh3NFRPuzCGdO3fm008/3ef3ESnPd99Bz57w++/Qvj3MmgVHHRXtqESkNiq368I5lw7MAB4ys4Zmdg5wGfBqGS/rB0wNXWFmB5lZNzNrYGbxZnYN0AWYs8/Ri4jUEosWQZcufvLXpQt88YWSPxGpPOGeu7oFOADYArwB3OycW2FmCfnj+QXv0jCzs4AjgbeK7aMu8Hf8G0OSgVuB3s45jQUoIjFt3jzo1g127YIrroA5c6BZs2hHJSK1WVjjADrntgO9S1ifhH+TSOi6/wF7TSPhnNsKnL5PUYqI1FIffgh9+vhj/A0YAM8/79/1KyJSmXT1uohIlLz9NvTu7Sd/t9wCL7yg5E9EqoYSQBGRKHj1VbjqKtizB0aMgH//G3RDuYhUlVr3dbM/d4OKVDdqz7XTc89B//7+TB9jx8Kjj2pOXxGpWuHOBVwjxMXFkZOTo6m5pNbIyckhTucEa5Xnny8c1Pkf/4C7745uPCISm2pVD2Djxo3ZtWtXtMMQiZhdu3bRuHF5k+5ITfHKKzBokP/8sceU/IlI9NSqBLB58+bs2LGD5ORksrOzdfpMaiTnHNnZ2SQnJ7Njx44i8xZLzfWf/8DAgeAcTJgAw4dHOyIRiWW16hRw/fr1SUhIYPv27axZs4bc3NxohySyT+Li4mjcuDEJCQnUr18/2uHIfnrnHbj2Wv+av4cegpEjox2RiMS6WpUAgp8EHn744Rx++OHRDkVEhJkzoW9fyM2F0aPh/vujHZGISC07BSwiUp3MmePP7FEw1Mu4cdGOSETEpwRQRKQSLFoEl18O2dlw220a6kVEqhclgCIiEfbdd3DppZCRATfcAJMmKfkTkepFCaCISAT98gt06wYpKf4cv88+q+RPRKofJYAiIhGyYQNcfDFs3gxdu8Jrr2luXxGpnpQAiohEwPbtfs/fmjXwxz/Cu++CRvARkepKCaCIyH5KS4OePWHFCjjhBPjoI2jUKNpRiYiUTgmgiMh+yM72h3r5+ms46iiYOxcOPjjaUYmIlE0JoIjIPsrL8+/ynTMHDjkEPv4YWrWKdlQiIuVTAigiso9GjYJp06BhQ/+0b7t20Y5IRCQ8SgBFRPbBpEn+4M7x8TBjBnTuHO2IRETCpwRQRKSC/vMfGD7cf/7SS3DJJdGNR0SkopQAiohUwCefQL9+/vN//hOuvTa68YiI7AslgCIiYVq2zJ/fNyfH7wG8885oRyQism/CSgDNrLmZvWtm6Wa21syuLqXeADPLNbO0kMWr6H5ERKqbzMzD6NkTUlPhr3+FiRM1xZuI1FzxYdZ7EsgGWgIdgQ/N7Dvn3IoS6v7POXduBPYjIlIt5OQ04fvvHyUjAy66CKZOhTo6fyIiNVi5X2Fm1hDoA9zvnEtzzi0C3geuq8gbRWo/IiJVKSMDli8fT0ZGAiefDO+8A/XqRTsqEZH9E87/sH8Acp1zq0LWfQecWEr9TmaWbGarzOx+MyvoZazofoICgQBTp04FICcnB8/zmDZtGgC7d+/G8zymT58OQEpKCp7nMWPGDACSk5PxPI+ZM2cCsGnTJjzPY/bs2QCsW7cOz/OYN28eAKtXr8bzPBYsWBB8b8/z+PLLLwFYvnw5nuexePFiABITE/E8j8TERAAWL16M53ksX74cgC+//BLP8wgEAgAsWLAAz/NYvXo1APPmzcPzPNatWwfA7Nmz8TyPTZs2ATBz5kw8zyM5ORmAGTNm4HkeKSkpAEyfPh3P89i9ezcA06ZNw/M8cnJyAJg6dSqe5wWP5ZQpU+jatWuw/NRTT9GjR49gefLkyfTq1StYnjhxIn369AmWJ0yYQN++fYPlcePGcW3IVfBjxoxh4MCBwfKoUaMYNGhQsDxixAiGDBkSLA8bNoxhw4YFy0OGDGHEiBHB8qBBgxg1alSwPHDgQMaMGRMsX3vttYwbNy5Y7tu3LxMmTAiW+/Tpw8SJE4PlXr16MXny5GC5R48ePPXUU8Fy165dmTJlSrDseZ7aXgy3vZEj7+Wvf4VduzpQp856LrzwnzRt6m9X21PbK6DvPbW9qmp7kRTOKeBGQEqxdSlA4xLqLgQ6AGvxE7vpwB7gkQruBzMbBAwCqK8Z1UWkijkHH3/ci2XLID5+F02b/pXGjS+IdlgiIhFhzrmyK5h1Ar5wzh0Ysu5OwHPO/amc1/YF7nLOnbY/++ncubNbsmRJ+Z9GRCRCxo+H0aOhfn1o3/5Wmjb9gfnz50c7LBGJUWa21DkXsSHnwzkFvAqIN7PQSY5OAcK5ccMBBffJ7c9+RESqzMsv+8mfGbz2GjRt+kO0QxIRiahyE0DnXDowA3jIzBqa2TnAZcCrxeuaWQ8za5n//HjgfuC/Fd2PiEi0zJoFN9zgP588GUIuxxERqTXCHcjgFuAAYAvwBnCzc26FmSXkj/WXkF/vIuB7M0sHPsJP+MaXt58IfA4Rkf32zTdwxRWQmwujRsGtt0Y7IhGRyhHWOIDOue1A7xLWJ+Hf3FFQHgGMKF6vvP2IiETbqlXwf/8Hu3dD//7w8MPRjkhEpPJoKFMRiXmbNkG3bpCcDD16wJQpmuVDRGo3JYAiEtN27fKTvjVr4PTT4a23oG7daEclIlK5lACKSMzKyoI//xkSE6FdO/jwQ2jYMNpRiYhUPiWAIhKTcnPhmmvgk0/gsMNgzhw45JBoRyUiUjWUAIpIzHEObrrJn9e3aVN/6Jejj452VCIiVUcJoIjEnHvugRdegAMOgA8+gI4dox2RiEjVUgIoIjHl0Uf9JT7e7wE899xoRyQiUvWUAIpIzJgyBUaO9Id4eeUV/+5fEZFYpARQRGLC22/D4MH+8yefhL/+NbrxiIhEkxJAEan1PvoIrr4a8vJg3Di4+eZoRyQiEl1KAEWkVps71x/rLycH7rgDRo+OdkQiItGnBFBEaq3PPoPLLvMHfB46FCZO1BRvIiKgBFBEaqlFi+DSSyEzE/72N5g8WcmfiEgBJYAiUut8/TX07Am7d8OAAfDMM1BH33YiIkH6ShSRWmXpUujWDVJT/Rs/nn9eyZ+ISHH6WhSRWuObb+DiiyElBa64Al5+GeLioh2ViEj1owRQRGqFhQuha1fYsQMuvxxef92f7UNERPamBFBEary5c6F798LTvm++CXXrRjsqEZHqSwmgiNRo778Pf/oTZGTADTf4U7yp509EpGxKAEWkxpo+Hfr0gexsuPVWeO45XfMnIhIOJYAiUiO99JJ/unfPHhg50h/nT3f7ioiER1+XIlKjOOfP53v99f7cvg89BI88okGeRUQqQlfKiEiNkZMDt9zij+1nBo8/7k/xJiIiFRNWD6CZNTezd80s3czWmtnVpdTrb2ZLzWyXma03s0fNLD5k+3wzyzSztPwlEKkPIiK1W1oa9OrlJ38NGsCMGUr+RET2VbingJ8EsoGWwDXA02Z2Ygn1DgSGAS2AM4CLgBHF6gx1zjXKX47bp6hFJKZs2gTnnw+zZ0OLFvDZZ9C7d7SjEhGpuco9BWxmDYE+QAfnXBqwyMzeB64D7gmt65x7OqS4wcxeAy6IYLwiEmN+/NGf13ftWjjmGD8JPPbYaEclIlKzhdMD+Acg1zm3KmTdd0BJPYDFdQFWFFv3iJklm9kXZuaV9kIzG2RmS8xsydatW8N4KxGpbWbMgDPO8JO/M86A//1PyZ+ISCSEkwA2AlKKrUsBGpf1IjMbCHQGJoasHgm0BVoBzwEzzeyYkl7vnHvOOdfZOdf5kEMOCSNMEaktcnNh9Gh/jL+0NOjbFz79FPRVICISGeEkgGlAk2LrmgCppb3AzHoDE4AezrnkgvXOua+dc6nOuSzn3MvAF0DPCkctIrXWjh1w6aUwfrw/rt+//uXP63vggdGOTESk9ghnGJhVQLyZtXPO/Zy/7hT2PrULgJl1B6YA/+ec+6GcfTtAo3eJCADffw+XXw6rV/s3e0yfDhdeGO2oRERqn3J7AJ1z6cAM4CEza2hm5wCXAa8Wr2tmFwKvAX2cc98U23aQmXUzswZmFm9m1+BfIzgnEh9ERGou52DqVDjrLD/5O/VUWLJEyZ+ISGUJdxiYW4ADgC3AG8DNzrkVZpaQP55fQn69+4GmwEchY/3Nyt9WF/g7sBVIBm4FejvnNBagSAzbvh2uvBIGDoTdu6FfP1i0CI46KtqRiYjUXmHNBOKc2w70LmF9Ev5NIgXlUod8cc5tBU6veIgiUlt9+qmf8G3YAI0awb//7Zc1rZuISOXSXMAiUuWysuCuu6BrVz/5O+ssSEyE/v2V/ImIVAUlgCJSpZYtgzPPhIkT/bt8H3gAFi70B3kWEZGqEdYpYBGR/ZWaCmPGwOOPQ14etG0Lr73mJ4MiIlK11AMoIpXuvffghBNg0iS/PGyYf8pXyZ+ISHSoB1BEKk1SEtx6K7z/vl/u3BmefdYf5kVERKJHPYAiEnG7dsH990P79n7y17ixf+r3q6+U/ImIVAfqARSRiMnOhmeegXHjIDl/EsgrrvBP/bZqFdXQREQkhBJAEdlveXn+tG2jR8Nvv/nrzjkHHn0Uzj47urGJiMjelACKyD7bswfeeQcmTPBv6gD/tO+ECfCnP2lMPxGR6koJoIhUWEaGP3fvxIn+3L0ARxwBDz4IAwZAvL5ZRESqNX1Ni0jYduyAp57yb+jYssVfd8wx/qwe/ftDgwbRjU9ERMKjBFBEyuQcfPEFTJkCb74JmZn++lNPhZEjoU8fiIuLbowiIlIxSgBFpETbtsGrr8Jzz8FPPxWuv+QSv8fvoot0jZ+ISE2lBFBEgtLT4aOP/J6+mTMhK8tf37IlXH893HCD5uwVEakNlACKxLjdu2HWLD/p++ADvwx+71737jBoEFx6KdStG904RUQkcpQAisSgpCSYPdtf5s71e/4KnHkmXHmlP4Bz69bRi1FERCqPEkCRGLB7t38jR0HS9+OPRbf/8Y+FSd9RR0UnRhERqTpKAEVqoW3bYNEif/n8c1i61B+0uUDjxtC1q3+Kt3t3SEiIXqwiIlL1lACK1HBpaf4sHEuX+suSJUXv2gWoU8cftuXii/2E7+yzoV69qIQrIiLVgBJAkRpizx749Vf/9O2PP8KKFbBsGQQC/lh9oRo0gDPOgHPPhfPOg7POgiZNohO3iIhUP0oARaqR3FxYt85P9EKXQABWrYLs7L1fU7cudOgAp53m9/KddhqccgrUr1/18YuISM2gBFCkimRn+9OnbdoEGzf6iV7osn69/5iTU/o+2rSBE04oXE4+2U/+lOyJiEhFKAEU2QfZ2ZCSAjt3+vPj7tzp33iRnOwvBc+3boXNm/2kb9u28PZ9xBH+YMtt2/qPxxwD7dpB+/bQqFFlfioREYkVYSWAZtYceAG4BEgGRjnnXi+l7nBgJHAA8A5ws3Muq6L7EdlXzvm9aFlZRZfMTMjI2HvZvdsfB6/4kpYGqamFy65d/mNKSuFgyRVRpw4ceigcdpi/tG7tL0ceWfi8dWs48MDIHxMREZFQ4fYAPglkAy2BjsCHZvadc25FaCUz6wbcA1wI/A68CzyYvy7s/RSXlgYLF4YXaPGL4fe1fknrK1K3+PqKPK/IY8ESWi7teThLXl7J5dDH0p7n5hauK1hycwuX4uWCZc+eos8LygXPc3IKl+Ll7OySl4q2g4qKi4NmzeCggwqXgw+GFi0KHwueFyR8Bx/sv05ERCTazJXzl9LMGgI7gA7OuVX5614FNjjn7ilW93VgjXPu3vzyRcBrzrnDKrKfvWPo7GDJPn1AiU1mezDLoU6d7Pyl4HlWyLos4uKy8h8z8h8z85/7j/Hxu4mLK1zi4zOIj0+jTp0MzKL9KaWqJCYmAtCxY8eoxiEisWvBggVLnXOdI7W/cHoA/wDkFiRt+b4Dzi+h7onAf4vVa2lmBwMJFdgPZjYIGOQ/b0+TJt+FEWqBSHX/7L0fs9L2Hc57FtYpup+SXuuK1SteJ3R70X2V/BpXrG5J5bwSthWu9x9D17kidc3yMMstst1/zMvfVvS5WW5+/b3XhS6QS506uflJnb8UlOvUyclP9PaEPO4J+SwiIiJSXDgJYCMgpdi6FKBxGHULnjeu4H5wzj0HPAfQuXNnt2TJKWGEKiISeZ7nATB//vyoxiEiscsifNqpThh10oDiQ8g2AVLDqFvwPLWC+xERERGRShJOArgKiDezdiHrTgFKunFjRf620HqbnXPbKrgfEREREakk5SaAzrl0YAbwkJk1NLNzgMuAV0uo/gpwg5mdYGbNgPuAqfuwHxERERGpJOH0AALcgj+u3xbgDfyx/VaYWYKZpZlZAoBzbjbwKPAZsDZ/GVvefiLySUREREQkLGGNA+ic2w70LmF9Ev7NHaHrHgMeq8h+RERERKTqhNsDKCIiIiK1hBJAERERkRijBFBEREQkxigBFBEREYkxSgBFREREYowSQBEREZEYY865aMdQLjNLBQLRjqMaawEkRzuIak7HqGw6PuXTMSqbjk/ZdHzKp2NUtuOcc40jtbOwxgGsBgLOuc7RDqK6MrMlOj5l0zEqm45P+XSMyqbjUzYdn/LpGJXNzJZEcn86BSwiIiISY5QAioiIiMSYmpIAPhftAKo5HZ/y6RiVTcenfDpGZdPxKZuOT/l0jMoW0eNTI24CEREREZHIqSk9gCIiIiISIUoARURERGKMEkARERGRGFMtEkAzG2pmS8wsy8ymlrD9IjNbaWa7zewzMzuqjH01N7N3zSzdzNaa2dWVGnwUmFlasSXXzJ4ope6A/O2h9b2qjbhqmdl8M8sM+bxlDiJuZsPNbJOZpZjZi2ZWv6pijQYzq29mL+T/fqSa2TIz61FG/ZhoQxX57lCbKb3NxEp7KUlFvntisA3p71YxZeU+VZH3VIsEEPgd+DvwYvENZtYCmAHcDzQHlgDTy9jXk0A20BK4BnjazE6MdMDR5JxrVLDgf84M4K0yXvK/0Nc45+ZXSaDRNTTk8x5XWiUz6wbcA1wEtAHaAg9WTYhREw+sA84HmuL/br1pZm3KeE0stKGwvjvUZsJqM7HQXkpT7ndPLLYh/d0qUYm5T1XlPdUiAXTOzXDOvQdsK2Hzn4EVzrm3nHOZwAPAKWZ2fPGKZtYQ6APc75xLc84tAt4Hrqu04KPvCmAL8Hm0A6mh+gMvOOdWOOd2AOOAAdENqXI559Kdcw8459Y45/Kccx8AvwGnRTu2aKngd4fajNrM/oq5NlSM/m5RZu5TJXlPtUgAy3Ei8F1BwTmXDvyav764PwC5zrlVIeu+K6VubdEfeMWVPZ5PJzNLNrNVZna/mdWUKQD3xyP5n/mLck4dFGlf+c9bmtnBlRlcdWJmLfF/d1aUUa22t6GKfHeozZTfZmp7eylLON89sd6G9HerbFWS99SEBLARkFJsXQpQ0oTIFalb45lZAv4pmZfLqLYQ6AAciv9fwl+Buyo/uqgaiX9KpRX+wJkzzeyYUuoWbzMFz2tlmynOzOoCrwEvO+dWllItFtrQ/nzPqM0UFQvtpTThfvfEbBvS362wVEneU+kJYP5Fsa6UZVEYu0gDmhRb1wRI3c+61VIFj1c/YJFz7rfS9uecW+2c+y3/tM0PwEP43e81UjjHxzn3tXMu1TmX5Zx7GfgC6FnKLou3mYLnNabNFBduGzKzOsCr+NeODC1tf7WtDZVif75nanybCVc4bSZG2kuJKvDdE7NtiBj8u7UPqiTvqfQE0DnnOeeslOXcMHaxAjiloJB/vvsYSj71sAqIN7N2IetOKaVutVTB49WPsv+LKvEtAItMtFVvH9tTWZ+5SPvKf77ZOVfS9ag1QjjHyMwMeAH/ouE+zrmcirwFNbgNlaIi3x21rs2EYz/aTG1sL+Eq7bPHZBvKF3N/t/ZBleQ91eIUsJnFm1kDIA6IM7MGIef73wU6mFmf/DpjgO9LOvWQf558BvCQmTU0s3OAy/D/Y61VzOxs/NMMZd1FhZn1yL9eh/wLSO8H/lv5EUaHmR1kZt0K2pCZXQN0AeaU8pJXgBvM7AQzawbcB0ytonCj6WmgPfAn51xGWRVjoQ1V8LtDbaaMNhML7aUkFfzuick2pL9bRZWR+1RN3uOci/qCf4eLK7Y8ELK9K7AS/7bx+UCbkG33ArNCys2B94B0IAm4Otqfr5KO2bPAqyWsT8DvEk7IL08ENucfj9X4Xel1ox1/JR6XQ4DF+N3fO4GvgItLOz756+7IP0a7gJeA+tH+HJV8jI7K/x3LzD8WBcs1sdyGSvvuUJspu83Eansp4RiV+t2jNhT8zPq7VfRzP0ApuQ9VkPdY/otFREREJEZUi1PAIiIiIlJ1lACKiIiIxBglgCIiIiIxRgmgiIiISIxRAigiIiISY5QAioiIiMQYJYAiIiIiMUYJoIhIGfLnVv53tOMQEYkkJYAiIiIiMUYzgYiIlMLMpgL9i60+2jm3puqjERGJHCWAIiKlMLOmwCz8OTnvzV+91TmXG72oRET2X3y0AxARqa6ccylmlg3sds5tinY8IiKRomsARURERGKMEkARERGRGKMEUESkbNlAXLSDEBGJJCWAIiJlWwP80czamFkLM9P3pojUePoiExEp20T8XsAfga1AQnTDERHZfxoGRkRERCTGqAdQREREJMYoARQRERGJMUoARURERGKMEkARERGRGKMEUERERCTGKAEUERERiTFKAEVERERijBJAERERkRjz/wG3/B6/S/i5+QAAAABJRU5ErkJggg==)]

Logistic regression model predicts
y ^ = { 0 , if p ^ < 0.5 1 , if p ^ >= 0.5 Note that when t < 0, σ( t ) < 0.5 ; when t > 0, σ( t ) >= 0.5 . If x T Θ is a positive class, the predictive result of the logistic regression model is 1. If it is a negative class, the prediction is 0 \widehat{y} = \begin{cases} 0, if \widehat{p}<0.5\\ 1, if \widehat{p}>=0.5 \end{cases}\\ Note that when t<0, \sigma(t)<0.5; when t>0, \sigma(t)>=0.5.\\If x^ T\Theta is a positive class, and the prediction result of the logistic regression model is 1\\If it is a negative class, the prediction is 0y ​={ 0,ifp ​<0.51,ifp ​>=0.5​Note that when t<0 , σ ( t )<0.5;t>0 , σ ( t )>=0.5.if xT Θis the positive class, and the logistic regression model predicts1If it is a negative class, the prediction is 0

Training and Cost Function

The cost function c ( Θ ) = { − log ( p ^ ) for a single training instance, if y = 1 − log ( 1 − p ^ ) , if y = 0 c(\Theta) = \begin{cases} -log (
\widehat{p}),\ \ \ \ \ \ if y = 1\\ -log(1-\widehat{p}), if y = 0 \end{cases}c ( Θ )={ −log(p ​),      if y=1−log(1−p ​),if y=0​
Logistic regression cost function (log loss)
J ( Θ ) = − 1 m ∑ i = 1 m [ log ( p ^ ( i ) + ( 1 − yi ) log ( 1 − p ^ ( i ) ) ) ] J( \Theta) = -\frac{1}{m}\sum_{i=1}^{m}[log(\widehat{p}^{(i)}+(1-y^{i})log( 1-\widehat{p}^{(i)}))]J(Θ)=−m1​i=1∑m​[log(p ​(i)+(1−yi)log(1−p ​( i ) ))]
Free-scale thermodynamic
∂ ∂ Θ j J ( Θ ) = 1 m ∑ i = 1 m ( σ ( Θ T x ( i ) ) − y ( i ) ) xj ( i ) \frac{ . \partial}{\partial\Theta_j}J(\Theta) = \frac{1}{m}\sum_{i=1}^{m}(\sigma(\Theta^Tx^{(i)})- y^{(i)})x_j^{(i)}∂Θj​∂​J(Θ)=m1​i=1∑m​( p ( ThTx(i))−y(i))xj(i)​

decision boundary

'''决策边界'''
iris = datasets.load_iris()
x = iris['data'][:,3:]
y = (iris['target']==2).astype(np.int_)
log_reg = LogisticRegression()
log_reg.fit(x,y)
#画图展示模型估算出的概率
x_new = np.linspace(0,3,1000).reshape(-1,1)
y_proba = log_reg.predict_proba(x_new)
plt.plot(x_new,y_proba[:,1],'g-',label = 'Iris virginica')
plt.plot(x_new,y_proba[:,0],'b--',label = 'Not Iris virginica')
plt.show()

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-CfEOzzhj-1658889211624)(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXcAAAD7CAYAAACRxdTpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAw60lEQVR4nO3dd3hUVfrA8e+bHhICASJFQECR3hOlIyq6NnBFXAVRFMEFdBUVBZEqIrhiWX+ADUVBcS00OyIKoojAKr1Il95DEtJzfn+chIQQyAAzuVPez/PcZ+7ce2bue7nhnTNnzj1HjDEopZTyL0FOB6CUUsr9NLkrpZQf0uSulFJ+SJO7Ukr5IU3uSinlh0KcDgCgQoUKpkaNGk6HoZRSPmXFihWHjDFxRe3ziuReo0YNli9f7nQYSinlU0Rkx5n2abOMUkr5IU3uSinlhzS5K6WUH9LkrpRSfkiTu1JK+SGXkruIPCQiy0UkXUSmFlN2oIjsE5FEEXlHRMLdEqlSSimXuVpz3wOMAd45WyERuR4YDFwD1ABqAaMuID6llFLnwaV+7saYmQAiEg9UPUvRe4Epxpi1ueWfBT7AJnyPeOUVmD0bgoPzl6Ag+1irlt2fp1cvMObUsnnlu3WD9u1tuRUrYNYsuz0kBMLDISzMLuHh9n1Ccv/lliyB5OTTy4SFQWwsVKjgqTNXSnmzHJNDelY6qVmppGamFvmYkZ1BTHgMV9W4yu3Hd/dNTA2AOQWerwQqikh5Y8zhggVFpC/QF6B69ernfcCNG2HhwqL3NW586vNp0yAnp+iydevmJ/c//oDnnjvzMXv1yl8fOBCWLj1zuXfftetr18KVV0JUlF2io/PXo6Jg/HgbA8CPP8KmTfbDoeBSrhzExNgPHaWUZ+SYHBLTEjl44iCHThzi0IlDHEs7RmJaIsfTj+cvGcdPeZ6UnsSJzBMnk3d6drpLx2tZtSVLei9x+3m4O7lHA4kFnuetlwZOSe7GmDeBNwHi4+PPe8aQgQPhjjsgOzt/ycmxj6VLn1p26tRTyxUs365dfrnmzeHZZ+2+zEy7pKdDRoZ9DA7OL5uQYBN13r6MjPz1SpXyyyUnQ0qKXYoyYkT++gcfwNtvF12uYUNYvdquGwMPPmi/HVSubI938cVQs6ZdF3H5n1Epv5eRncHepL3sOr6LXcd3sTtp98nH/cn7TybzwycOk22y3XLM8OBwIkMjiQyJPO0xIiSC8JBw6lWo55ZjFebu5J4MxBR4nree5ObjnHT55XZxRc+erpVr1swurnjtNdfKJSRAYmJ+gs9b8pL+pZfml23f3ibuo0fhyBH7mLcU/sB4662ijxcRYb813Hmnfb5mDWzdCvXr2+Rf8ANKKX+RnJHMn4f/ZPORzfnLUfu4J2mPy+9TJrwMFUpVOLnERsZSJrwMMeExZ1xKh5WmVGipk8k7PCScIHHua7a7k/taoAnwce7zJsD+wk0ygSgoyDapxMQUX7ZnzzN/EBVsVgoOhjffhH37YO9eu+zaBdu2weHDthknzyefwOjRdj0iwjYB1a8PDRpAfDxcd935n5tSJS3H5LDlyBZW7V9llwOrWL1/NVuObjnja4IkiMrRlakaU5WqMVW5uPTFJ9crRVciLiqOCqUqUC6yHGHBYSV4Np7hUnIXkZDcssFAsIhEAFnGmKxCRd8HporIB8Be4BlgqvvCVQXb20uVgj59ii53/Lj9YTdP7do2ga9bZz8A/vjDLmC/+WzcmF924kTbNBUfD6Gh7j4Dpc5dYloiS3cv5dddv7Jk1xKW7lrK0bSjp5ULDQrlsnKXUbt8bS6LzX0sdxmXlbuMqjFVCQnyirESS4S4MkG2iIwERhTaPArbNXIdUN8YszO37GPAU0Ak8BnwT2PMWX9ZiI+PNzoqZMlJTIT1622iX70aypeHZ56x+/bts+33YH/obdcOrroKOnaEFi20OUeVjLSsNH7e+TPzt85n/rb5rNizAsOpuapydGWaVW5Go4sa0bhiYxpXbEyd8nUIDQ6cGomIrDDGxBe5z5Xk7mma3L3HX3/BmDG2B1LB2jxAXBx89ZWt0SvlbvuT9zNn4xxmbZjFj9t/JC0r7eS+0KBQmlduTquqrWhVrRWtqraiakxVJMB7DZwtuQfOdxTlkmrV4I037PqePTbJ//ADzJ8Pu3dDnTr5ZV991f7A27kzREY6E6/ybfuS9zFj9QxmbpjJzzt/PqV23qRiE66tdS2danWibfW2RIVFORip79Gau3KJMbBzJ1xyiX2elQUVK9rePDExcPvtcM89tqdPgFemVDHSstL4fOPnvLfyPb7Z/M3JbodhwWF0qtWJv9f9OzdffjMVoys6HKn302YZ5Xbp6baGP20aFLx0devCQw/ZRF/4PgMV2P5K/IuJyyby1v/e4kjqEQBCgkK4qfZN3NXwLm6ofQMx4S50J1MnaXJXHrV+PUyfbm8S25PblfjXX+0duUot+WsJL//6MjPXzzxZS29aqSm9mvSie6PuxEUVOQWocoEmd1UiMjPtmDwLFsDrr+dvnzgRbrzR3jylAsfinYsZ+eNIvt/2PWBr6bfXv51HrnyEllVbOhydf9Dkrhzzxx/2bt/gYLj3XjusQ5UqTkelPOnXXb8y7IdhzN86H4CY8BgGJAxgQMIALo652OHo/MvZkrsOQaU8qmxZm9QB3nnH3kw1atSZx9hRvmvX8V30mNmDVlNaMX/rfGLCYxjefjjbH9nO2GvGamIvYZrclUfVqGHb4jdsgK5d4cQJGDnS3hU7Y4bDwSm3SM1MZfTC0Vz+2uV8uPpDwoPDGdJ2CNse2caojqOIjYx1OsSApMldlYjLLoNPP7X95lu0sD+8zpvndFTqQi3asYgmrzdhxI8jSM1KpVv9bmx4aANjrxlLuchyxb+B8hi9iUmVqPbt4bff4L337M1PeQ4csEMX61j1viEpPYnB8wczafkkABrENWDSTZNof0l7hyNTeTS5qxIXFAT33Zf/PDUVrr7ajkX/3nunDmusvM/SXUu567O72HZsGyFBIQxtN5Sn2z3tFyMp+hOtJynHbdxoByybN8/OnvXll05HpIqSY3IYv3g8bd9ty7Zj22heuTn/6/s/Rl41UhO7F9LkrhzXtCmsWgXXXAMHD8LNN8PgwXYmLOUdDp84zA0f3MDg7weTlZPFYy0fY0nvJTSq2Mjp0NQZaHJXXqFKFVtzHz/e9okfP94m+aOnD9mtStiaA2tIeCuBeVvmUaFUBb7s/iUTrp+gtXUvp8ldeY2gIHjySfjuOzvG/DffwJw5xb9Oec6cDXNoNaUV245tI75KPL8/+Ds31r7R6bCUC/QHVeV1Ona0g5F9+GH+DVCq5E34ZQJPfPcEAN0bdeftW94mMlTHdvYVWnNXXqlGDXj66fzhg7duhdmznYwocBhjePK7J08m9ueveZ7pf5+uid3HaHJXXi8pCf72N7jtNpg82elo/FtWTha95/bm37/8m5CgED647QMGtx0c8DMe+SJN7srrRUfbfvHGQP/+8MorTkfknzKyM+j2STfe/eNdSoWW4vO7Pqd7o+5Oh6XOkyZ35fVEYMiQ/Fr7wIEwYYKzMfmbzOxM7vz0TmZvmE1sRCzze87nb5f9zemw1AXQH1SVz/jnP22PmgcfhCeegJwcGDTI6ah8X1ZOFj1m9mDWhlmUjSjL/Hvm07xyc6fDUhdIa+7Kp/TtC2+/bWvzTz5pZ3xS5y87J5ues3ryybpPiAmPYd7d8zSx+wmtuSuf07s3ZGTA8ePQUif0OW/GGP719b/4aM1HlA4rzbd3f0vCxQlOh6XcRJO78kn9+p363Jj8bpPKNeMWj2PS8kmEB4fzRfcvdOo7P6PNMsrnbdliJ+NevdrpSHzH+yvf5+kFTyMI02+brkP1+iFN7srnTZgAy5bZsWj27XM6Gu/33Zbv6D23NwCv/u1Vbq9/u8MRKU/Q5K583ksvQatWsHMn3HqrHR9eFW3zkc3c8ekdZOVkMaj1IB6+8mGnQ1Ieosld+byICJg1Cy65BJYuhfvvt23w6lRJ6Ul0+agLx9KO0aVOF8ZdO87pkJQHaXJXfqFiRfjiCyhdGj76SO9iLSzH5NBzVk/WHVxHvQr1eP/v7xMk+t/fn+nVVX6jYUOYOtWuDxpkZ3hS1rMLn2XOxjmUjSjLnDvnEBMe43RIysNcSu4iUk5EZolIiojsEJEiB5wQa4yI7BaRRBH5UUQauDdkpc7stttg6FA7VMHllzsdjXf4bst3jFo4iiAJYkbXGdQuX9vpkFQJcLWf+0QgA6gINAW+FJGVxpi1hcp1A+4H2gI7gDHANEBveVMlZswYpyPwHvuS93H3rLsxGEZ2GKnjxQSQYmvuIhIFdAWGGWOSjTGLgblAzyKK1wQWG2O2GmOygelAfXcGrNS52LgRpkxxOgpn5LWzH0g5QMcaHRnabqjTIakS5EqzzOVAtjFmU4FtK4Gimls+Ai4TkctFJBS4F/imqDcVkb4islxElh88ePBc41aqWAcPQny8HY/ml1+cjqbkjVs8jvlb5xNXKo7pt00nOCjY6ZBUCXIluUcDiYW2JQKliyi7F/gJ2AikYptpBhb1psaYN40x8caY+Li4ONcjVspFcXEwYIAdPbJHDzsWTaBY8tcShv8wHID3//4+VUpXcTgiVdJcSe7JQOGf1mOApCLKjgASgGpABDAKWCAipS4kSKXO1+jR0Lw5bN8ODz3kdDQlIyUjhXtm30O2yWZQ60Hazh6gXEnum4AQESn4E3sToPCPqXnb/2uM2WWMyTLGTAVi0XZ35ZCwMDvRdmQkTJtm+8D7uyHfD2Hzkc00vKghz3Z81ulwlEOKTe7GmBRgJjBaRKJEpA3QBdsLprBlQDcRqSgiQSLSEwgFNrszaKXORZ068PLLdn3AANi/39l4POn7rd/z2m+vERIUwvu3vk94SLjTISmHuHoTU38gEjgAzAD6GWPWikh1EUkWkeq55cZjf2z9AziGbW/vaow55s6glTpXfftCp05w4gT89pvT0XhGYloi98+9H4Dh7YfTrHIzhyNSThLjBYNwxMfHm+XLlzsdhvJzO3dCWpr/3tzUZ24f3v79bRKqJPBL718ICdLpGvydiKwwxsQXtU+vvgoY1asXX8ZXLdqxiLd/f5uw4DCm3jpVE7vSsWVU4DHGjkHjL71n0rPS6ft5XwCGtB1C/Tjtv6C05q4C0O7d0L+/Hff91lvh2mudjujCjFs8jo2HN1KnfB2GtB3idDjKS2jNXQWcqlVh2DC7PmAApKc7G8+F2HBoA2MXjwXgjZvf0N4x6iRN7iogPf441K0LmzbBiy86Hc35yTE5PPjFg2RkZ9C7WW861OjgdEjKi2hyVwEpLAwmTrTrY8bAtm3OxnM+pq2cxqIdi7go6iJe6PSC0+EoL6PJXQWsq6+G7t1t98iHH/atqfmOpx/nqflPAfDvTv+mXGQ5hyNS3kaTuwpoL74IMTG25n70qNPRuO7Zhc+yP2U/raq24u7GdzsdjvJC2ltGBbTKlWHBAmjUyDbV+IKNhzby6tJXEYT/3PAfnQtVFUmTuwp4LVo4HYHrjDE8+u2jZOZk0rtZb+KrFHlzolLaLKNUnv377Y1N3jyw2Jd/fsk3m78hJjyGsdeMdToc5cW05q5Urocfhk8+gcxMeOMNp6M5XUZ2BgO/tXPfjLpqFBdFXeRwRMqbac1dqVyjR0NICLz9Nqxc6XQ0p3tj+RtsPrKZuhXqMiBhgNPhKC+nyV2pXHXrQr9+dlq+xx7zrq6RiWmJjF40GoDx144nNDjU4YiUt9PkrlQBI0ZAbKztQfPFF05Hk++Fn1/g0IlDtKvejlsuv8XpcJQP0OSuVAHly+ePO/P005Cd7Ww8ALuO7+KlX18C7A1LIuJwRMoXaHJXqpD+/eGSS2DNGli0yOloYMQPI0jLSqNb/W5cWfVKp8NRPkJ7yyhVSHg4vPUWlC0LCQnOxrJ6/2qmrrSTb2jXR3UuNLkrVYROnZyOwBr8/WByTA4DEgZwWbnLnA5H+RBtllGqGD/9BImJJX/chdsX8tWfX1E6rDTD2g8r+QCUT9PkrtRZPPMMtG9f8mO+G2N45odnAHii9RPERcWVbADK52lyV+osbrzRPr70UskOSzBvyzwW71xM+cjyPNry0ZI7sPIbmtyVOovWraFzZzhxAsaPL5ljGmMY9oNthnmqzVPEhMeUzIGVX9HkrlQxRtsbQ5k8Gfbs8fzxPt/0Ocv2LKNiVEUGXKHDDKjzo8ldqWI0aQJdu9oZm8aN8+yxckzOyVr70+2eplRoKc8eUPktTe5KuWDkSBCxo0Xu2uW543y27jNW7V9F1Ziq9G3R13MHUn5P+7kr5YKGDeHee+3wBJGRnjlGdk42w38cDsAz7Z4hIiTCMwdSAUGTu1IueucdW3v3lA9Xf8iGQxuoWbYm9zW7z3MHUgFBm2WUclHBxO7u4YCzc7JPDuk7osMIwoJ9ZEJX5bVcSu4iUk5EZolIiojsEJHuZylbS0S+EJEkETkkIi+4L1ylnDd9OtSvD9u2ue89P177MZuPbKZWbC16NO7hvjdWAcvVmvtEIAOoCPQAJotIg8KFRCQM+A5YAFQCqgLT3ROqUt5h3jzYsAHGjHHP++WYHMYutoOCDW4zmJAgbS1VF67Y5C4iUUBXYJgxJtkYsxiYC/QsongvYI8x5iVjTIoxJs0Ys8qtESvlsGHDICgI3n8ftm+/8Pf7fOPnrDmwhotLX8w9Te658DdUCtdq7pcD2caYTQW2rQROq7kDLYHtIvJ1bpPMjyLSqKg3FZG+IrJcRJYfPHjw3CNXyiG1a0P37pCVBS9cYKOjMYbnfnoOgEGtBxEeEu6GCJVyLblHA4XHxEsEShdRtipwJ/AfoArwJTAnt7nmFMaYN40x8caY+Lg4HRRJ+ZYhQ+wPrFOmXNhdq99v+55le5YRVyqOPi36uC9AFfBcSe7JQOHBLWKApCLKpgKLjTFfG2MygBeB8kC9C4pSKS9Tv769azUj48JGjMyrtQ9sOVDvRlVu5Upy3wSEiEjtAtuaAGuLKLsK8KI545XynKFD7eNXX9kmmnP1y1+/8OP2HykTXob+Cf3dG5wKeMUmd2NMCjATGC0iUSLSBugCTCui+HSgpYhcKyLBwKPAIWC9+0JWyjs0bQpffw2rVkHIeXRwyau1P3zFw5SJKOPe4FTAc7UrZH8gEjgAzAD6GWPWikh1EUkWkeoAxpiNwN3A68BR7IdA59wmGqX8zt/+BmHncb/R73t/56s/v6JUaCkeafmI+wNTAc+l+oYx5ghwaxHbd2J/cC24bSa2pq9UwDh0CFavho4dXSuf16/9wRYPUqFUBQ9GpgKV3i2h1AXasQMaNLA1+B07oHRR/cgKWH9wPZ+t+4yw4DCeaP1EyQSpAo6OLaPUBbrkEmjeHI4etRN6FGfcz+MwGO5reh9VSlfxfIAqIGlyV8oNnrFzWTNhgp2S70y2H9vOB6s+IFiCearNUyUTnApImtyVcoNOnSAhAQ4cgDffPHO5F35+gWyTTfdG3akZW7PkAlQBR5O7Um4gkl97f/FFSE8/vczepL288/s7CMKQtkNKNkAVcDS5K+UmN98MjRrB7t12ULHCJiyZQHp2OrfVu416cXrTtvIsTe5KuUlQkB1zpmNHqFcodx8+cZjXl78OwNB2Qx2ITgUa7QqplBvdeSfcddfp219d+iopmSnccNkNNKvcrOQDUwFHa+5KuVFRc6weTz/Oa7+9BsDT7Z4u4YhUoNLkrpQHrFoF3brBnDkwadkkjqUdo/0l7Wlbva3ToakAoc0ySnnAwoXw6aewZWs2f/3jJUDb2lXJ0pq7Uh7wwANw0UXw+/+CObS6KfFV4ulUq5PTYakAosldKQ+IjIRHHs0d5P2noQxtNxQpqkFeKQ/R5K6Uh8S0+RAijsKODpQ72NnpcFSA0eSulAdk5WTx6spn4cr/APD8WP2vpkqW/sUp5QGfrP2EzUc2U+P6z4mKMsyfDzt3Oh2VCiTaW0YpN8sxOScn4xh6/T8pX1do1gyqV3c4MBVQNLkr5Wafb/ycNQfWcHHpi+nZuCfhzZ2OSAUibZZRyo2MMScnvh7UehDhIeEn9+XkwIoVTkWmAo0md6XcaP7W+Szbs4y4UnH0adHn5PbMTIiPhyuvhK1bHQxQBQxN7kq5UV5b+8CWAykVWurk9tBQaNwYsrNh/HinolOBRJO7Um7yy1+/8OP2HykTXob+Cf1P2z9kiB1YbOpUO+a7Up6kyV0pNxmzaAwAD1/xMGUiypy2v04duP12yMiwszUp5Uma3JVygxV7VvD15q+JCo3ikZaPnLHc07kj/r7xBhw8WELBqYCkyV0pN8jrIdMvvh8VSlU4Y7mmTeHGGyE1FV55pWRiU4FJk7tSF2jNgTXM2jCLiJAIHm/9eLHlhw6FihWhSpUSCE4FLL2JSakLlFdr79O8D5WiKxVbvnVr2LEDwsOLLarUedOau1IXYOOhjfx3zX8JDQplUOtBLr9OE7vyNE3uSl2AcT+Pw2Do1bQX1cpUO6fXJibC88/DO+94KDgV0LRZRqnztP3YdqatnEawBDO47eBzfv2iRbb3TJUq0KOH1uaVe7lUcxeRciIyS0RSRGSHiHR34TULRMSIiH6AKL80fvF4sk023Rt1p1ZsrXN+/U03QaNGsGcPvPeeBwJUAc3VZpmJQAZQEegBTBaRBmcqLCI90G8Fyo/tPr6bd/54B0EY0nbIeb1HUFB+v/fx4yEry40BqoBXbHIXkSigKzDMGJNsjFkMzAV6nqF8GWAE8KQ7A1XKm7z4y4tkZGdwe/3bqRdX77zfp1s3qF3bDib23/+6MUAV8FypuV8OZBtjNhXYthI4U819LDAZ2He2NxWRviKyXESWH9Rb9ZQP2Z+8nzdWvAHA0HZDL+i9goNhcG5z/dixdlhgpdzBleQeDSQW2pYIlC5cUETigTbAa8W9qTHmTWNMvDEmPi4uzpVYlfIKL/z8AqlZqXSu05kmlZpc8PvdfTdUqwbr1sG8eW4IUClcaxdPBmIKbYsBkgpuEJEgYBLwiDEmS0TcE6FSXmRv0l4mLZ8EwMgOI93ynmFhdiiCsDC4/nq3vKVSLiX3TUCIiNQ2xvyZu60JsLZQuRggHvhvbmIPzt2+S0S6GWN+ckfASjlp3OJxpGWl8fe6f6dZ5WZue9/bbnPbWykFuJDcjTEpIjITGC0iDwBNgS5A60JFE4GCo2VUA34DWgDaqK583u7ju0+2tY+8aqTHjrN1K9Ssacd+V+p8udoVsj8QCRwAZgD9jDFrRaS6iCSLSHVj7ctbyE/o+40xGR6IXakS9fzi50nPTqdb/W40rtjYI8cYPNj2npk1yyNvrwKIS33RjTFHgFuL2L4T+4NrUa/ZDmjdQ/mFvxL/4q3/vYUgjOgwwmPHqVbN9pgZPhy6dLG9aZQ6Hzq2jFIuGPvTWDKyM/hHw3/Q4KIz3r93wR54AKpXh7Vr4eOPPXYYFQA0uStVjO3HtjPl9ykESZBHa+1gx5cZPtyujxihd62q86fJXalijFk0hsycTLo36k7dCnU9frx77oFLL4U//4Tp0z1+OOWnNLkrdRYbDm3g3T/eJViCGd5+eIkcMzQURo6066NG2Qm1lTpXmtyVOouhC4aSY3J4oPkD1C5fu8SOe9dd0KABtGoFSUnFl1eqMB25UakzWLprKTPXzyQyJJLhHUqm1p4nOBiWLYPIyBI9rPIjWnNXqgjGGJ6a/xQAj1z5CFVKl/xs1prY1YXQ5K5UEb7Z/A0LdywkNiKWp9o+5WgsCxbAVVfBgQOOhqF8jCZ3pQrJMTkM+d5OwDGk7RDKRpR1NJ6XXoKFC2H0aEfDUD5Gk7tShcxYPYOV+1dSNaYqD13xkNPhMG6cnbXpjTdg06biyysFmtyVOkVqZipPL7Bz343sMJLIUOcbvhs2hPvuszc05U3Lp1RxNLkrVcDLv77MzsSdNK7YmF5NezkdzkmjRtkfWD/7DJYscToa5Qs0uSuVa2/SXsb+NBaAl69/meAg7xm16+KL4bHH7PoTT4AxzsajvJ8md6VyPbPgGVIyU+hcpzNX17za6XBO8+STEBcHGzbYMd+VOhu9iUkp4H97/8e7f7xLaFAoL3Z60elwihQTY8d5r1cPypVzOhrl7TS5q4BnjOGxbx/DYHj4iodLdJiBc9WmjdMRKF+hzTIq4H22/jMW7lhI+cjyDOswzOlwXJKWZrtIavOMOhNN7iqgJaUn8eg3jwIw5uoxjt+w5KqhQ2HIEHj8cacjUd5Kk7sKaKMWjmJ30m4SqiTQp3kfp8Nx2eOPQ1QUzJ4N8+Y5HY3yRprcVcBavX81r/z6CoIw+abJXtX1sThVqsCw3Bakhx6yzTRKFaTJXQWkHJNDvy/7kW2y6Z/QnxZVWjgd0jkbOND2nPnzTxg71ulolLfR5K4C0vsr3+fnv37moqiLGHP1GKfDOS9hYfDmm3Z93DhYt87ZeJR30eSuAs7BlIMM+m4QABOum+AzP6IWpW1b6NsXMjNh6lSno1HeRPu5q4Dzr2/+xaETh+hYoyM9GvVwOpwLNm4cdOhgp+ZTKo8mdxVQZm+YzUdrPqJUaCne7vw2IuJ0SBcsNha6d3c6CuVttFlGBYyjqUfp92U/AJ6/5nlqxdZyOCL327TJDgusA4sprbmrgPHYvMfYl7yPNtXaeMUkHO6Wnm6n49u7F2rXtmPAq8ClNXcVEL7c9CVT/5hKeHA4UzpPIUj8708/PBzGj7frjzwCO3Y4G49ylv/9hStVyP7k/dw/937ADjFQp0IdhyPynLvvhltvhaQkuP9+yMlxOiLlFJeSu4iUE5FZIpIiIjtEpMifb0TkXhFZISLHRWSXiLwgItr0oxxjjOH+ufdzIOUAV9e8msdaPeZ0SB4lYudarVABFiyAiROdjkg5xdWa+0QgA6gI9AAmi0iDIsqVAh4FKgBXAtcAT1x4mEqdn0nLJvHVn18RGxHLe7e+55fNMYVddJFN8GBnbfrjD0fDUQ4p9i9dRKKArsAwY0yyMWYxMBfoWbisMWayMeYnY0yGMWY38AGgI1ArR6w9sJYnvrN1i7dueYuqMVUdjqjk3HabvblJxM7cpAKPK9WYy4FsY8ymAttWAkXV3AtrD6wtaoeI9BWR5SKy/ODBgy68lVKuS8lI4R+f/oO0rDTub3o/Xet3dTqkEvfKK7B8Odx5p9ORKCe4ktyjgcRC2xKB0md7kYjcB8QDRc5ZZox50xgTb4yJj4uLcyVWpVxijKHP531Ye3AtdSvU5dUbXnU6JEdERkLDhvnPU1Odi0WVPFeSezIQU2hbDJB0pheIyK3AOOAGY8yh845OqfMwadkkZqyZQVRoFDPvmEl0WLTTITlu+nSoWRNWrnQ6ElVSXEnum4AQESk4sWQTztzc8jfgLeAWY8zqCw9RKdf9uutXBn47EIApnadQL66ewxF5h/nzYf9+6NIFDml1KyAUm9yNMSnATGC0iESJSBugCzCtcFkRuRr7I2pXY8xv7g5WqbPZm7SXbp90IzMnk0eufIR/NPyH0yF5jddfh4QEe2PTHXfYUSSVf3O1X1h/IBI4AMwA+hlj1opIdRFJFpHqueWGAWWAr3K3J4vI1+4PW6lTncg8QZePurDr+C7aVGvDC51ecDokrxIRATNnQsWK8MMPtouk8m8u3WBkjDkC3FrE9p3YH1zznnd0W2RKuSjH5HDv7HtZtmcZNcvWZNY/ZhEWHOZ0WF6nalWb4K+6Cv7zH6hfHx580OmolKf4/x0dyu8N/2E4n677lJjwGL7o/gVxUdr76kxat4bJk+364MFw5Iiz8SjP0aEBlE+bvGwyz/30HMESzMe3f0z9uPpOh+T1eveGY8fg6quhXDmno1Geosld+awZq2cw4KsBAEy+aTLXX3a9wxH5jscfP/V5Roadk1X5D22WUT7p6z+/5p7Z92AwPH/N8/Rp0cfpkHzWtGm2/X3bNqcjUe6kyV35nAXbFtD1465k5WTxRKsneKrNU06H5LOysuwgY1u22GaanTudjki5iyZ35VO+2/IdN314E6lZqTzQ7AFe6PSCX8yD6pSQEPjqK7jySti+HTp2hF27nI5KuYMmd+Uzvt38LbfMuIW0rDT6NO/DG7e8oYndDWJi4JtvID4etm61CX77dqejUhdKk7vyCZ+u+5QuH3UhPTudfvH9eP3m1wNibPaSUrYszJsHzZvD5s22y+RqHTzEp+n/DuX1Xlv6Gnd8cgfp2en864p/MfHGiZrYPSA21s7e1KGDHUFSvxT5Nu0KqbxWjsnh6e+fZvzPdtbnsVePZXDbwdoU40Flytgmmj//PHW4YOV7NLkrr3Q8/Tj3zLqHORvnECzBTOk8hXub3ut0WAEhIgIaNcp//n//B4cPw7BhEKRfmHyGJnfldf48/CddPurC+kPrKRtRlo+6fqQ3KDlk1y57w1NGBqxZA1OnQlSU01EpV+jnsPIqczfOJeGtBNYfWk/9uPr89sBvmtgdVLUqzJple9R8+im0bAlri5zJQXkbTe7KK6RmpjLgywF0+agLiemJ3Fr3Vn7t/Su1y9cu/sXKo268EZYuhTp1bO09IQHeeguMcToydTaa3JXjVu1fRcJbCUxaPonQoFAmXDeBz+74jNLhZ52mV5WgunXtZNu9etmeNH37wvjxTkelzkaTu3JMelY6w38YTvyb8aw9uJY65euw9IGlPNbqMe3q6IWio+Hdd+18rFWqQPfuTkekzkb/BylHLN65mKZvNOXZRc+SmZPJP1v8kxV9V9CscjOnQ1PF6NHDjkVTPXf+tZwcGDpUx6XxNprcVYnambiTHjN70O7ddmw4tIG6FeqyqNciJt88magw7YbhKyIi8tdffx3GjrVt8sOHQ0qKc3GpfJrcVYlISk9i6PdDqfN/dfhw9YeEB4czvP1w/njwD9pd0s7p8NQF6NIF7roL0tLg2Wdtkn//fcjOdjqywKbJXXlUUnoS4xaPo9Z/ajF28VjSstK4s+GdbHhoA6M6jiI8JNzpENUFuvhi+PBDWLwYWrSA3bvh3nvtHa6ff+50dIFLb2JSHnEk9QgTf5vIy7++zNG0owC0rtaaFzu9SKtqrRyOTnlCmzbw22928o+RI2HDBpvolTM0uSu3WrV/Fa8tfY0PVn9AalYqAG2rt2VEhxFcU/MaHRfGzwUF2Vp79+4wYwbceWf+vuees5OD/POfULGiczEGCjFecCdCfHy8Wb58udNhqPOUnJHMrPWzePv3t1m0Y9HJ7dddeh1D2g6hwyUdNKkHuKQke7fr8eN2rtauXe2HwLXXQnCw09H5LhFZYYyJL2qf1tzVecnOyeb7bd8zbdU0Zq6fyYnMEwBEh0XTq0kvHrriIepUqONwlMpbREfD7Nnw6qswd66t1c+YYdvr774bHn7Yriv30Zq7cllyRjLztsxj7sa5fLHpCw6nHj65r3W11vRs3JPujboTEx7jYJTK2+3YYXvTvPee7S8PdmKQvCGG9+61zTY6AmXxzlZz1+SuzijH5LBy30p+2P4D87fOZ8G2BaRnp5/cX7tcbXo06sHdje/m0nKXOhip8kXGwM8/2zlcn3suf3KQK66Abdvgmmtss82110KNGo6G6rU0uSuXpGam8vu+3/lt92/8uP1HFu1YdLKnC4AgtKzaks51OtO5TmfqVainbenKrZKToXFjm9wLuvRSaNvW/hjbsqUzsXkjbXNXpzmWdox1B9exev9qlu9ZzrI9y1hzYA3Z5tQ7T2qUrUHHGh25qsZVXH/p9VSM1m4OynOio21TzaZNMH++XRYssNu2bIHOnfPLzp4NixbZD4PGjaF+/VPvnA10mtz9WFpWGjuO7WD7se1sPbqVDYc2sO7QOtYeWMve5L2nlQ+SIBpXbEx85XjaVm9Lx5odqVG2RskHrgKaiL3LtU4dGDDAdp/8/XdYsgTaFbiZee5cO5BZnuBgqF3bJvlWreCJJ/L3GRN4c8JqcvdRSelJ7Eved3LZm7yXfcn7+Ov4X2w7uo1tx7axJ2nPGV8fGRJJvbh6NIhrQPPKzUmokkCzys0oFVqqBM9CqeKFhNgx5BMSTt1+3322uWbVKrts2mRvnNqwwTbv5CX3pCT7A+0ll9geOVWr2iVvvVUrqFCh5M/L01xK7iJSDpgCXAccAoYYYz48Q9mBwFNAJPAZ0M8Yk15U2UBljCEjO4OUzBRSMlJIyUwhOSOZpPQkjqYd5Wjq0VMej6QeOfn8cOph9ifvJyWz+NGZgiWYamWqUbNsTWqWrUmdCnWoH1ef+nH1qVG2hg6rq3xau3an1uRTU2H9eti4EcqWzd++bZvdl5f4C/v2W7juOrs+Zoz9NhAXZxN+3hIbC9WqwT335L/u99/tlIMxMXaJjPSubweu1twnAhlARaAp8KWIrDTGnDLhlohcDwwGrgb2ALOAUbnb3C45I5mUjBSyTTbZOdnkmJyT69km93nu+rlsK/g+WTlZZGRnkJGdQXp2ev56VnrR2wuVOZF54rQknhfzhYgIiaBydGUqRVeicunKVIqqRKXoSlwcc7FN5rE1qRpTlZAg/XKmAkNkJDRvbpeCGjeGxEQ7JPHu3XZe2F278tdr1covu3MnbN1ql8IaN85P7sbYXj1ZWfn7g4NtDJGR8O9/25u0AL78EiZMyN+Xt0RE2GM/+qhb/xlOKvZ/vohEAV2BhsaYZGCxiMwFenJ60r4XmJKX9EXkWeCDIsq5xaB5g3h9xeueeGuPCw0KJSosiqjQKKLCoogOiyY6LJrYiFhiI2PtY8H1yFjKRZYjNiKWStGViAmP0Z4qSrkoJsb2o8/rS38mL75om3MOHcpfDh6EY8dObbrJzIQmTeyHxvHjdklLs81BycmnJv0dO+CHH4o+3pVXOpjcgcuBbGPMpgLbVgIdiijbAJhTqFxFESlvjDlcsKCI9AX6AlTPG/X/HJWJKENcqTiCg4IJlmCCJOjk+rlsC5KgU/YX3BYSFEJ4SDhhQWGEBYfZ9eDc9eDwM27LW0qFljqZvPMSeVRoFKHBoed1zkopz8lrYrn88rOXCwuz0w4WlJVlm39SU21zTZ4uXeyPw3n7Ci4XXeT+c8hTbD93EWkHfGKMqVRgWx+ghzHmqkJltwADjDHf5D4PxTbn1DTGbD/TMbSfu1JKnbuz9XN35Re1ZKDw/eQxQJILZfPWiyqrlFLKQ1xJ7puAEBGpXWBbE2BtEWXX5u4rWG5/4SYZpZRSnlVscjfGpAAzgdEiEiUibYAuwLQiir8P9BaR+iISCzwDTHVjvEoppVzgakfn/th+6weAGdi+62tFpLqIJItIdYDctvYXgB+AHbnLCPeHrZRS6mxc6gRtjDkC3FrE9p1AdKFtLwEvuSM4pZRS50dvUVRKKT+kyV0ppfyQJnellPJDXjFZh4gcxP74ej4qYAcz8wd6Lt7JX87FX84D9FzyXGKMiStqh1ck9wshIsvPdIeWr9Fz8U7+ci7+ch6g5+IKbZZRSik/pMldKaX8kD8k9zedDsCN9Fy8k7+ci7+cB+i5FMvn29yVUkqdzh9q7koppQrR5K6UUn5Ik7tSSvkhn0juIlJORGaJSIqI7BCR7mcpO1BE9olIooi8IyLhJRnr2bh6HiLSS0Syc0fczFuuKtloz05EHhKR5SKSLiJTiynrzdfEpfPwkWsSLiJTcv+2kkTkdxG54SzlvfK6nMt5+Mh1mS4ie0XkuIhsEpEHzlLWbdfEJ5I7MBE7XV9FoAcwWUQaFC4kItdjJ+O+BqgB1AJGlVyYxXLpPHItMcZEF1h+LKkgXbQHGAO8c7ZCPnBNXDqPXN5+TUKAv7DzG5cBhgEfi0iNwgW9/Lq4fB65vP26PA/UMMbEAJ2BMSLSonAhd18Tr0/uIhIFdAWGGWOSjTGLgblAzyKK3wtMMcasNcYcBZ4FepVYsGdxjufh9YwxM40xs4HiZtny2msC53QeXs8Yk2KMGWmM2W6MyTHGfAFsA05LJHjxdTnH8/B6uf/G6XlPc5dLiyjq1mvi9ckduBzINsZsKrBtJVBUjbdB7r6C5SqKSHkPxueqczkPgGYicij3a9wwEXFp7H0v5M3X5Fz51DURkYrYv7uipsT0metSzHmAD1wXEZkkIieADcBe4Ksiirn1mvhCco8GEgttSwRKu1A2b72osiXtXM5jEdAQuAhb278LGOTR6DzHm6/JufCpayIiocAHwHvGmA1FFPGJ6+LCefjEdTHG9Mf+27bDTluaXkQxt14TX0juyUBMoW0xQJILZfPWiypb0lw+D2PMVmPMttyvpKuB0cDtJRCjJ3jzNXGZL10TEQnCznGcATx0hmJef11cOQ9fui7GmOzc5tiqQL8iirj1mvhCct8EhIhI7QLbmlD0V7S1ufsKlttvjPGG9tRzOY/CDCAeicrzvPmaXAivvCYiIsAU7I/2XY0xmWco6tXX5RzOozCvvC6FhFB0m7tbr4nXJ3djTAr2a8xoEYkSkTZAF+wnemHvA71FpL6IxALPAFNLLNizOJfzEJEbctsZEZG62N4Cc0oy3uKISIiIRADBQLCIRJyhrdNrrwm4fh6+cE1yTQbqAbcYY1LPUs6rrwsunoe3XxcRuUhE7hSRaBEJzu0RcxewoIji7r0mxhivX4BywGwgBdgJdM/dXh37VaZ6gbKPAfuB48C7QLjT8Z/reQAv5p5DCrAV+1Uz1On4C53LSPJ/+c9bRvrgNXHpPHzkmlySG39abux5Sw9fui7nch7efl2AOGAhcCz333k10Cd3n0eviQ4cppRSfsjrm2WUUkqdO03uSinlhzS5K6WUH9LkrpRSfkiTu1JK+SFN7kop5Yc0uSullB/S5K6UUn7o/wHMUy2QU38n9wAAAABJRU5ErkJggg==)]

Softmax regression

Softmax score for class k
sk ( x ) = x T Θ ( k ) s_k(x) = x^T\Theta^{(k)}sk​(x)=xT Th( k )
Softmax function
p ^ k = σ ( s ( x ) ) k = exp ( sk ( x ) ) ∑ j = 1 kexp ( sj ( x ) ) In this equation, K is the number of classes and s ( x ) is a vector that includes the per-class scores σ ( s ( x ) ) of instance x where k is the estimated probability that instance x belongs to class k, given the per-class scores of the instance \widehat{p}_k = \sigma(s (x))_k = \frac{exp(s_k(x))}{\sum_{j=1}^{k}exp(s_j(x))}\\ In this equation\\K is the number of classes\ \s(x) is a vector that includes the scores for each class of instance x \\\sigma(s(x))_k is the estimated probability that instance x belongs to class k, given the scores for each class of that instancep ​k​=σ ( s ( x ) )k​=∑j=1k​exp(sj​(x))exp(sk​(x))​In this equationK is the number of classess ( x ) is a vector containing the scores for each class of instance xσ ( s ( x ) )k​is the estimated probability that instance x belongs to class k , given the scores for each class of that instance
Softmax regression classification prediction
y ^ = argmax σ ( s ( x ) ) k = argmaxsk ( x ) = argmax ( ( Θ k ) T x ) \widehat{y} = argmax\sigma(s(x))_k = argmaxs_k(x) = argmax((\Theta^{k})^Tx)y ​=argmaxσ(s(x))k​=argmaxsk​(x)=a r g ma x (( Thk)T x)
cross entropy cost function
J ( Θ ) = − 1 m ∑ i = 1 m ∑ k = 1 kyk ( i ) log ( p ^ l ( i ) ) In this equation yk ( i ) is the class k The target probability of the ith instance of . Generally equal to 1 or 0, depending on whether the instance belongs to the J(\Theta) = -\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{ k}y_k^{(i)}log(\widehat{p}_l^{(i)}) \\In this equation y_k^{(i)} is the target probability of the i-th instance belonging to class k . Generally equal to 1 or 0, depending on whether the instance belongs to theJ(Θ)=−m1​i=1∑m​k=1∑k​yk(i)​log(p ​l(i)​)In this equation yk(i)​is the target probability of the ith instance belonging to class k . Generally equal to 1 or 0 , depending on whether the instance belongs to the

introduce

Cross Entropy (Cross Entropy) is an important concept in Shannon information theory, which is mainly used to measure the difference information between two probability distributions. The performance of language models is usually measured by cross entropy and complexity ( perplexity ). The meaning of cross-entropy is the difficulty of using the model to recognize text, or from a compression point of view, how many bits are used to encode each word on average. The meaning of complexity is to use the model to represent the average number of branches of this text, and its reciprocal can be regarded as the average probability of each word. Smoothing refers to assigning a probability value to unobserved N-gram combinations to ensure that the word sequence can always obtain a probability value through the language model. Commonly used smoothing techniques are Turing estimation , deletion interpolation smoothing, Katz smoothing and Kneser-Ney smoothing.

official

The cross-entropy is introduced into the field of computational linguistics disambiguation, and the real semantics of the sentence is used as the prior information of the cross-entropy training set , and the semantics of machine translation is used as the posterior information of the test set. Calculate the cross entropy of the two, and use the cross entropy to guide the identification and elimination of ambiguity. Examples show that the method is simple and effective. It is easy for computer adaptive realization. Cross-entropy is an effective tool for disambiguation in computational linguistics.

In information theory, cross entropy represents two probability distributions p, q, where p represents the real distribution, and q represents the non-real distribution. In the same set of events, the non-real distribution q is used to represent the occurrence of an event. The average number of bits required. From this definition, it is difficult for us to understand the definition of cross entropy. Let's take an example to describe it:

Suppose there are two probability distributions p, q in a sample set, where p is the real distribution and q is the unreal distribution. If, according to the real distribution p, the expectation of the code length required to identify a sample is:

H§=

[External link picture transfer failed, the source site may have an anti-leeching mechanism, it is recommended to save the picture and upload it directly (img-DYioVdqB-1658889211625)(https://bkimg.cdn.bcebos.com/formula/d8600ed8a168b3ccde14700c4edba145.svg)]

However, if the wrong distribution q is taken to represent the average code length from the true distribution p, it should be:

H(p,q)=

[External link picture transfer failed, the source site may have an anti-leeching mechanism, it is recommended to save the picture and upload it directly (img-lk1m7IsR-1658889211626)(https://bkimg.cdn.bcebos.com/formula/4f4bd894f88b051dbcb362809a886c10.svg)]

At this time, H(p,q) is called cross entropy. The cross entropy is calculated as follows:

For discrete variables, it is calculated in the following way: H(p,q)=

[External link picture transfer failed, the source site may have an anti-leeching mechanism, it is recommended to save the picture and upload it directly (img-CyFkN1Yr-1658889211626)(https://bkimg.cdn.bcebos.com/formula/39dffa5dea6b891ae6a322110c04ea1a.svg)]

For continuous variables, it is calculated in the following way:

[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-l9Gu7e8Y-1658889211626)(https://bkimg.cdn.bcebos.com/formula/86749cf53f17a226a4e45c56664cae10.svg)]

Θ ( k
) J ( Θ ) = 1 m ∑ i = 1 m ( p ^ k ( i ) − yk ( i ) ) x ( i ) \nabla_{\Theta(k)} J(\Theta) = \frac{1}{m}\sum_{i=1}^{m}(\width{p}_k^{(i)}-y_k^{(i)})x^{ (i)}∇Θ ( k )J(Θ)=m1​i=1∑m​(p ​k(i)​−yk(i)​)x(i)

'''Softmax回归'''
iris = datasets.load_iris()
X= iris["data"][:, (2, 3)]  # petal length, petal width
y = iris["target"]

softmax_reg = LogisticRegression(multi_class="multinomial",solver="lbfgs", C=10, random_state=42)
softmax_reg.fit(X, y)
x0, x1 = np.meshgrid(
        np.linspace(0, 8, 500).reshape(-1, 1),
        np.linspace(0, 3.5, 200).reshape(-1, 1),
    )
X_new = np.c_[x0.ravel(), x1.ravel()]


y_proba = softmax_reg.predict_proba(X_new)
y_predict = softmax_reg.predict(X_new)

zz1 = y_proba[:, 1].reshape(x0.shape)
zz = y_predict.reshape(x0.shape)
#画图
plt.figure(figsize=(10, 4))
plt.plot(X[y==2, 0], X[y==2, 1], "g^", label="Iris virginica")
plt.plot(X[y==1, 0], X[y==1, 1], "bs", label="Iris versicolor")
plt.plot(X[y==0, 0], X[y==0, 1], "yo", label="Iris setosa")

from matplotlib.colors import ListedColormap
custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])

plt.contourf(x0, x1, zz, cmap=custom_cmap)
contour = plt.contour(x0, x1, zz1, cmap=plt.cm.brg)
plt.clabel(contour, inline=1, fontsize=12)
plt.xlabel("Petal length", fontsize=14)
plt.ylabel("Petal width", fontsize=14)
plt.legend(loc="center left", fontsize=14)
plt.axis([0, 7, 0, 3.5])

plt.show()

code summary

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression,SGDRegressor,LogisticRegression
from sklearn.preprocessing import PolynomialFeatures,StandardScaler
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.linear_model import Ridge
from sklearn.base import clone
from sklearn.pipeline import Pipeline
from copy import  deepcopy
from sklearn import datasets


x = 2*np.random.rand(100,1)
y = 4+3*x+np.random.randn(100,1)

x_b = np.c_[np.ones((100,1)),x]
theta_best = np.linalg.inv(x_b.T.dot(x_b)).dot(x_b.T).dot(y)

x_new = np.array([[0],[2]])
x_new_b = np.c_[np.ones((2,1)),x_new]
y_predict = x_new_b.dot(theta_best)

#使用sklearn
lin_reg = LinearRegression()
lin_reg.fit(x,y)

'''批量梯度下降'''
eta = 0.1
n_iteration = 1000#迭代次数
m = 100

theta = np.random.randn(2,1)
for n in range(n_iteration):
    gradienters = 2/m*x_b.T.dot(x_b.dot(theta)-y)#偏导计算
    theta = theta-eta*gradienters #更改theta

'''随机梯度下降'''
n_epochs = 50#向前或向后迭代次数

t0,t1 = 5,50 #学习步长超参数

def learning_schedule(t):#计算步长
    return t0/(t+t1)#步长逐渐减小

theta = np.random.randn(2,1)
for n in range(n_epochs):
    for i in range(m):
        random_index = np.random.randint(m)
        xi = x_b[random_index:random_index+1]
        yi = y[random_index:random_index+1]
        gradienters = 2*xi.T.dot(xi.dot(theta)-yi)
        eta = learning_schedule(n*m+i)
        theta = theta-eta*gradienters


sgd_reg = SGDRegressor(max_iter=1000,tol = 1e-3,penalty=None,eta0=0.1)
sgd_reg.fit(x,y.ravel())
print(sgd_reg.intercept_,sgd_reg.coef_)

'''小批量梯度下降'''
theta_path_mgd = []

n_iterations = 50
minibatch_size = 20

np.random.seed(42)
theta = np.random.randn(2,1)  # random initialization

t0, t1 = 200, 1000
def learning_schedule(t):
    return t0 / (t + t1)

t = 0
for epoch in range(n_iterations):
    shuffled_indices = np.random.permutation(m)
    x_b_shuffled = x_b[shuffled_indices]
    y_shuffled = y[shuffled_indices]
    for i in range(0, m, minibatch_size):
        t += 1
        xi = x_b_shuffled[i:i+minibatch_size]
        yi = y_shuffled[i:i+minibatch_size]
        gradients = 2/minibatch_size * xi.T.dot(xi.dot(theta) - yi)
        eta = learning_schedule(t)
        theta = theta - eta * gradients
        theta_path_mgd.append(theta)

'''多项式回归'''
m = 100
x = 6*np.random.rand(m,1)-3
y = 0.5*x**2+x+2+np.random.randn(m,1)

poly_features = PolynomialFeatures(degree=2,include_bias=False)#聚类特征
x_poly = poly_features.fit_transform(x)

lin_reg = LinearRegression()
lin_reg.fit(x_poly,y)
画图
X_new=np.linspace(-3, 3, 100).reshape(100, 1)
X_new_poly = poly_features.transform(X_new)
y_new = lin_reg.predict(X_new_poly)
plt.plot(x, y, "b.")
plt.plot(X_new, y_new, "r-", linewidth=2, label="Predictions")
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.legend(loc="upper left", fontsize=14)
plt.axis([-3, 3, 0, 10])

plt.show()

'''学习曲线'''
def plot_learning_curves(model,x,y):
    x_train,x_val,y_train,y_val = train_test_split(x,y,test_size=0.2)
    train_erros,val_erros = [],[]
    for m in range(1,len(x_train)):
        model.fit(x_train[:m],y_train[:m])
        y_train_pred = model.predict(x_train[:m])
        y_val_pred = model.predict(x_val)
        train_erros.append(mean_squared_error(y_train[:m],y_train_pred))
        val_erros.append(mean_squared_error(y_val,y_val_pred))
    plt.plot(np.sqrt(train_erros),'r-+',linewidth = 2,label = 'train')
    plt.plot(np.sqrt(val_erros),'b--',linewidth = 3,label = 'val')
    plt.show()


lin_reg = LinearRegression()
plot_learning_curves(lin_reg,x,y)

'''岭回归'''
np.random.seed(42)
m = 20
X = 3 * np.random.rand(m, 1)
y = 1 + 0.5 * X + np.random.randn(m, 1) / 1.5
X_new = np.linspace(0, 3, 100).reshape(100, 1)

ridge_reg = Ridge(alpha=1, solver="cholesky", random_state=42)
ridge_reg.fit(X, y)
ridge_reg.predict([[1.5]])
ridge_reg = Ridge(alpha=1, solver="sag", random_state=42)
ridge_reg.fit(X, y)
ridge_reg.predict([[1.5]])
'''lasso'''
from sklearn.linear_model import Lasso
lasso_reg = Lasso(alpha=0.1)
lasso_reg.fit(X, y)
lasso_reg.predict([[1.5]])

'''提前停止'''
np.random.seed(42)
m = 100
x = 6 * np.random.rand(m, 1) - 3
y = 2 + x + 0.5 * x**2 + np.random.randn(m, 1)

x_train, x_val, y_train, y_val = train_test_split(x[:50], y[:50].ravel(), test_size=0.5, random_state=10)
poly_scaler = Pipeline([('poly_features',PolynomialFeatures(degree=90,include_bias=False)),('std_scaler',StandardScaler())])
x_train_poly_scaled = poly_scaler.fit_transform(x_train)
x_val_poly_scaled = poly_scaler.transform(x_val)
sgd_reg = SGDRegressor(max_iter=1,tol=-np.infty,warm_start=True,penalty=None,learning_rate='constant',eta0=0.0005)
minimum_val_error = float('inf')
best_epoch = None
best_model = None
for epoch in range(1000):
    sgd_reg.fit(x_train_poly_scaled,y_train)
    y_val_predict = sgd_reg.predict(x_val_poly_scaled)
    val_error = mean_squared_error(y_val, y_val_predict)
    if val_error < minimum_val_error:
        minimum_val_error = val_error
        best_epoch = epoch
        best_model = deepcopy(sgd_reg)

#画图
sgd_reg = SGDRegressor(max_iter=1, tol=-np.infty, warm_start=True,
                       penalty=None, learning_rate="constant", eta0=0.0005, random_state=42)

n_epochs = 500
train_errors, val_errors = [], []
for epoch in range(n_epochs):
    sgd_reg.fit(x_train_poly_scaled, y_train)
    y_train_predict = sgd_reg.predict(x_train_poly_scaled)
    y_val_predict = sgd_reg.predict(x_val_poly_scaled)
    train_errors.append(mean_squared_error(y_train, y_train_predict))
    val_errors.append(mean_squared_error(y_val, y_val_predict))

best_epoch = np.argmin(val_errors)
best_val_rmse = np.sqrt(val_errors[best_epoch])

plt.annotate('Best model',
             xy=(best_epoch, best_val_rmse),
             xytext=(best_epoch, best_val_rmse + 1),
             ha="center",
             arrowprops=dict(facecolor='black', shrink=0.05),
             fontsize=16,
            )

best_val_rmse -= 0.03  # just to make the graph look better
plt.plot([0, n_epochs], [best_val_rmse, best_val_rmse], "k:", linewidth=2)
plt.plot(np.sqrt(val_errors), "b-", linewidth=3, label="Validation set")
plt.plot(np.sqrt(train_errors), "r--", linewidth=2, label="Training set")
plt.legend(loc="upper right", fontsize=14)
plt.xlabel("Epoch", fontsize=14)
plt.ylabel("RMSE", fontsize=14)

plt.show()

'''决策边界'''
iris = datasets.load_iris()
x = iris['data'][:,3:]
y = (iris['target']==2).astype(np.int_)
log_reg = LogisticRegression()
log_reg.fit(x,y)
#画图展示模型估算出的概率
x_new = np.linspace(0,3,1000).reshape(-1,1)
y_proba = log_reg.predict_proba(x_new)
plt.plot(x_new,y_proba[:,1],'g-',label = 'Iris virginica')
plt.plot(x_new,y_proba[:,0],'b--',label = 'Not Iris virginica')
plt.show()
'''Softmax回归'''
iris = datasets.load_iris()
X= iris["data"][:, (2, 3)]  # petal length, petal width
y = iris["target"]

softmax_reg = LogisticRegression(multi_class="multinomial",solver="lbfgs", C=10, random_state=42)
softmax_reg.fit(X, y)
x0, x1 = np.meshgrid(
        np.linspace(0, 8, 500).reshape(-1, 1),
        np.linspace(0, 3.5, 200).reshape(-1, 1),
    )
X_new = np.c_[x0.ravel(), x1.ravel()]


y_proba = softmax_reg.predict_proba(X_new)
y_predict = softmax_reg.predict(X_new)

zz1 = y_proba[:, 1].reshape(x0.shape)
zz = y_predict.reshape(x0.shape)

plt.figure(figsize=(10, 4))
plt.plot(X[y==2, 0], X[y==2, 1], "g^", label="Iris virginica")
plt.plot(X[y==1, 0], X[y==1, 1], "bs", label="Iris versicolor")
plt.plot(X[y==0, 0], X[y==0, 1], "yo", label="Iris setosa")

from matplotlib.colors import ListedColormap
custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])

plt.contourf(x0, x1, zz, cmap=custom_cmap)
contour = plt.contour(x0, x1, zz1, cmap=plt.cm.brg)
plt.clabel(contour, inline=1, fontsize=12)
plt.xlabel("Petal length", fontsize=14)
plt.ylabel("Petal width", fontsize=14)
plt.legend(loc="center left", fontsize=14)
plt.axis([0, 7, 0, 3.5])

plt.show()

]

y_proba = softmax_reg.predict_proba(X_new)
y_predict = softmax_reg.predict(X_new)

zz1 = y_proba[:, 1].reshape(x0.shape)
zz = y_predict.reshape(x0.shape)

plt.figure(figsize=(10, 4))
plt.plot(X[y2, 0], X[y2, 1], “g^”, label=“Iris virginica”)
plt.plot(X[y1, 0], X[y1, 1], “bs”, label=“Iris versicolor”)
plt.plot(X[y0, 0], X[y0, 1], “I”, label=“Iris setosa”)

from matplotlib.colors import ListedColormap
custom_cmap = ListedColormap([‘#fafab0’,‘#9898ff’,‘#a0faa0’])

plt.contourf(x0, x1, zz, cmap=custom_cmap)
contour = plt.contour(x0, x1, zz1, cmap=plt.cm.brg)
plt.clabel(contour, inline=1, fontsize=12)
plt.xlabel(“Petal length”, fontsize=14)
plt.ylabel(“Petal width”, fontsize=14)
plt.legend(loc=“center left”, fontsize=14)
plt.axis([0, 7, 0, 3.5])

plt.show()

Guess you like

Origin blog.csdn.net/m0_63953970/article/details/126009543
Recommended
LFOSSA Yuanlaisusu Open Course | Mastering the Cloud Native Future: Comprehensive Guide to CNCF Certification and Exam Preparation Tips
Ranking
C++ Basic Syntax
bootstrapTable hides a column based on a condition
Why is reentrant lock recommended instead of Synchronized when dynamic high concurrency?
hexo create a blog
[Fully open source and non-encrypted version] Imitation of the eighth district distribution/online signature/multiple sets of download templates/APP distribution hosting/APP packaging and packaging
Polymerization combination
https://www.flysnow.org/2017/05/06/go-in-action-go-log.html
From the perspective of Flutter and the front-end, talk about how to ensure UI fluency under the single-threaded model
nginx-301, 302 redirect
Geolocation by IP Address in ASP.NET
Daily
More
2024-04-26(22)
2024-04-25(32)
2024-04-24(30)
2024-04-23(30)
2024-04-22(5)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)
2024-04-17(31)

Code World

© 2018 codetd.com