Machine Learning and High-Dimensional Information Retrieval - Note 5 - (Deep) Feedforward Neural Networks and related examples based on CVXOPT

Note 5 - (Deep) Feedforward Neural Networks and related examples

5.1 Definition and motivation of FNN

Roughly speaking, feedforward neural networks (FNN) are a special class of functions that are very powerful at minimizing any kind of expected loss, but at the cost of training a large number of parameters. More precisely, consider an input variable $\mathcal{X}\in {\mathbb{R}}^p$ and a function class$F$ , we want to find a function$f$ , so that a certain loss function$L$ has the smallest expected value. For example, consider the simple loss function$L(f(\mathcal{X}))=\Vert f(\mathcal{X})-\mathcal{X}{\Vert }_2^2$, for the purpose of reconstructing X \mathcal{X}Autoencoder for$X\; samples.$Here,$f$ consists of the connection of the encoder function (mapped to a low-dimensional space) and the decoder function (mapped back to the original data space).

Another example of supervised learning is regression, where we have a joint distribution of input and output variables $(\mathcal{X},Y)$ , the purpose is to be inF \mathcal{F}Find the bestff$in\; F$$f$，使$L(f(\mathcal{X}),\mathcal{Y})=\Vert f(\mathcal{X})-\mathcal{Y}{\Vert }_2^2$expectations are minimal. We discuss multilevel classification later in this section.

If F \mathcal{F}All functions in$F$$Th\in {\mathbb{R}}^N$ , if some training samples are given, such as$n$ , then these learning problems will produce a minimization process

$$\widehat{Th}=\arg \underset{\Theta \in {\mathbb{R}}^N}{\min }\frac{1}{n}\sum _{i}L\left(f_{}\left({\mathbf{x}}_i\right)\right).\text{\hspace{1em}(5.)}$$

A very important class of functions in many applications are the so-called feedforward neural networks $\left(FNN\right)$ . Feedforward neural network is a linear function${}^1$ in series.

$${Phi}_{\mathbf{W}}:{\mathbb{R}}^p\to R^m,\mathbf{h}\mapsto \mathbf{W}\mathbf{h}\text{\hspace{1em}(5.)}$$

${}^1$ This also includes affine functions, since we can simply append an extra component 1 to our input vector.

What follows is the so-called activation function, which performs component operations on a vector. Examples of activation functions are: Rectified Linear Unit (ReLU)
σ ( t ) : = max ⁡ { 0 , t } , (5.) \sigma(t):=\max \{0, t\} ,\tag{5.}s ( t ):=max{ 0,t},(5.)

and some other activation functions, see Wikipedia . With a slight misuse of notation, we also represent the vector activation function
$$p:{\mathbb{R}}^m\to R^m,\mathbf{x}\mapsto \left[\begin{array}{c}p\left(x_1\right)\\ ⋮\\ p\left(x_m\right)\end{array}\right]\text{\hspace{1em}(5.)}$$

A feedforward neural network is a function
$$f:R^p\to R^o,\mathbf{x}\mapsto p_l\circ {Phi}_{{\mathbf{W}}_l}\circ \cdots \circ p_1\circ {Phi}_{W_1}(\mathbf{x}),\text{\hspace{1em}(5.)}$$

where, as usual, $\circ$ represents the connection of functions, different function classes$F$ consists of different activation functions and number of layers$l$ and the size of the matrix$W_i\in {\mathbb{R}}^{n_i\times m_i}$definition. If the number of layers exceeds three, we usually call such FNN a deep feedforward neural network. Note that the output dimension $o$ is determined by the loss function, because the output of the FNN can be used as the input of the loss.

5.2 Training FNNs

Training FNNs in practice is an art in itself, and there are many tricks and regularization techniques that determine the failure or success of learning a powerful FNN. In this section, we focus on the basic method of finding the best FNN for the general problem (5.1), that is, finding the best weight Θ ^ = ( $\widehat{Th}=\left({\mathbf{W}}_1,\dots W_L\right)$ . In fact, it is a gradient descent method that iteratively updates weights, a method known in the literature asbackpropagation. Below we describe how it works.

Here, the most important tools we need in undergraduate mathematics courses are the chain rule and the Jacobian matrix . To recap, if $g：{\mathbb{R}}^k\Rightarrow {\mathbb{R}}^l$ 和$h:R^l\to R^m$ is two functions,$g$ at​​x \mathbf{x}Differentiable at$x$$h$在$\mathbf{y}=$Differentiable at$g$$($$x$$)$${\mathbf{J}}_g\left(x\right)$和${\mathbf{J}}_h\left(y\right)$ . So aboutx \mathbf{x}Function h ∘ g of$x$$h\circ g:{\mathbb{R}}^k\to R^m$ in the graphical function$J_{h\circ g}(\mathbf{x})=J_h(g(\mathbf{x})\cdot J_g(x)$ is differentiable.

Examples

• (linearity of derivative) If in the context of the above introduction, $h$ is a simple linear transformation consisting of matrix multiplication by$W$ gets, then

$${\mathbf{J}}_{\mathbf{W}\cdot g}(\mathbf{x})=\mathbf{W}\cdot J_g\left(x\right)$$

• Due to $The\; i$ -time output only depends on$x_i$,(5.4)Increase σ \sigmaThe Jacobian matrix of$\sigma$
  $$J_{}\left(x\right)=\left[\begin{array}{lll}{p}^{\prime }\left(x_1\right)& & \\ & \ddots & \\ & & p^{\prime }\left(x_m\right)\end{array}\right]$$

• To define the Jacobian matrix of a function, multiply
  $$\mathrm{mult}(\mathbf{x}):{\mathbb{R}}^{m\times n}\to R^m,\mathbf{W}\mapsto Wx,$$
  we first need to convert$mn$ variables (here given in matrix structures) are embedded in$R^{mn}$ .${}^{\_}$This can be achieved in various ways, but if we choose to embed row by row, i.e.
  $$Pi:\mathbf{W}=\left[\begin{array}{c}{\mathbf{w}}_1^{\top }\\ ⋮\\ w_m^{\top }\end{array}\right]\mapsto \left[\begin{array}{c}{w}_1\\ ⋮\\ w_m\end{array}\right]\in {\mathbf{R}}^{mn}$$

Note: $\pi$ is$m\times An\; n$ -dimensional matrix is ​​mapped into a single-dimensional vector whose length is$m\cdot n$.

Then the Jacobian matrix has a good arrangement of form

$${\mathbf{J}}_{\mathrm{m}\mathrm{u}\mathrm{l}\mathrm{t}(\mathbf{x})}=\left[\begin{array}{ccc}{\mathbf{x}}^{\top }& & \\ & \ddots & \\ & & x^{\top }\end{array}\right]\in {\mathbb{R}}^{m\times mn}.$$

Note that since $mult\left(x\right)$ is linear, and the Jacobian matrix does not depend on W.

In order to calculate the gradient of the cost function (5.1) with respect to the weight $Th=\left({\mathbf{W}}_l,\dots {\mathbf{W}}_1\right)$ , we note that it is an input datax \mathbf{x}The average value of the gradient of the lossfunction$of$
$$F\left({\mathbf{W}}_l,\dots ,{\mathbf{W}}_1\right):=L\circ p_l\circ {Phi}_{W_l}\circ \cdots \circ p_1\circ {Phi}_{W_1}(\mathbf{x}).$$

Therefore, just calculate the FF depending on one input signal$The\; gradient\; of\; F$ , then for all training data${\mathbf{x}}_i$average. For convenience, we denote as

$${\mathbf{h}}_j:=p_j\circ {Phi}_{W_j}\circ \cdots \circ p_1\circ {Phi}_{W_1}(\mathbf{x})$$

It is the jj of FNNThe output after$j\; layer.$For$0<j<l$，${\mathbf{h}}_j$The jjth called FNN$j$ hidden layer,$h_l$is called the output layer, $h_0:=x$ is the input of FNN.

We define the Jacobian matrix of the function by ${}^2$$\frac{\partial }{\partial {\mathbf{W}}_j}F\in {\mathbb{R}}^{1\times m_j{n}_j}$

$${\mathbf{W}}_j\mapsto F\left({\mathbf{W}}_l,\dots ,W_j,\dots ,W_1\right).\text{\hspace{1em}(5.13)}$$

${}^2$ due to$F$ is real-valued, which is also the transpose of the gradient.

Using the chain rule and the example in the previous section, regarding different weight matrices $W_i$The derivative of is given by the following formula
$$\begin{array}{rl}\frac{\partial }{\partial {\mathbf{W}}_l}F& ={\mathbf{J}}_L\left({\mathbf{h}}_l\right)\cdot J_{p_l}\left({\mathbf{W}}_l{\mathbf{h}}_{l-1}\right)\cdot J_{\mathbf{m}\mathbf{u}\mathbf{l}\mathbf{t}\left({\mathbf{h}}_{l-1}\right)}\\ \frac{\partial }{\partial W_{l-1}}F& =J_L\left(h_l\right)\cdot J_{p_l}\left(W_l{h}_{l-1}\right)\cdot {\mathbf{W}}_1\cdot J_{p_{l-1}}\left(W_{l-1}{h}_{l-2}\right)\cdot J_{\text{mult }\left({\mathbf{h}}_{l-2}\right)}\\ & \dots \\ \frac{\partial }{\partial W_j}F& =J_L\left(h_l\right)\cdot J_{p_l}\left(W_l{h}_{l-1}\right)\cdot W_1\cdot J_{p_{l-1}}\left(W_{l-1}{h}_{l-2}\right)\cdot J_{l-1}\cdots \cdots J_{\text{mult }\left(h_{j-1}\right)},\end{array}$$

In practice, the initial weights are usually chosen randomly from a normal distribution and then used with the above gradient and step size $a>0$ updated individually. Algorithms and methods for selecting step sizes are beyond the scope of this lecture. Note that we must "reshape" the gradient (i.e.$\frac{\partial }{\partial W_i}$transpose of$F$${Pi}_j$Perform an inversion. We finally get the update rule
$${\mathbf{W}}_j\leftarrow W_j-a{p}_j^{-1}{\left(\frac{\partial }{\partial {\mathbf{W}}_j}F\right)}^{\top }.\text{\hspace{1em}(5.14)}$$

5.3 Multi-class classification using FNNs

The problem considered in multi-class classification is to convert the input $\mathcal{X}\in {\mathbb{R}}^p$ is assigned to multiple (such as$C$ ) one of the classes. We pass random variables$(\mathcal{X},Y)$ models the problem, where$X\in R^p$和Y ∈ { e 1 , ... , e C } \mathcal{Y} \in\left\{\mathbf{e}_{1}, \ldots, \mathbf{e}_{C}\right\}Y∈{ e1​,…,eC​} is${\mathbb{R}}^{}$CCin${}^C$$One\; of\; the\; C$ standard basis vectors. A realization$\mathbf{y}=e_c$means belongs to $Events\; of\; type\; c$ are real. This modeling of output variables is also calledone-hot-encoding.

The idea of ​​using FNN for multi-class classification is: $X$ is used as the input of the network, and the output is$R^{}$A vector in${}^C$ that should reflect the given X \mathcal{X}The probability of the class distribution of$X.$More precisely, if$x$ is$\mathcal{X}$的实现，如果$f$表示FNN，那么输出向量${\mathbf{h}}_l:=f(\mathbf{x})$的第$c$分量应该是$\mathcal{Y}$属于$c$类的概率，给定$\mathrm{x}$，即
$${\mathbf{h}}_l\approx \left[\begin{array}{c}\mathrm{Pr}\left(\mathcal{Y}={\mathbf{e}}_1\mid \mathcal{X}=\mathbf{x}\right)\\ ⋮\\ \mathrm{Pr}\left(\mathcal{Y}={\mathbf{e}}_C\mid \mathcal{X}=\mathbf{x}\right)\end{array}\right]$$
Therefore, the last activation function $p_l$The motivation is for $Class\; C$ outputs a probability distribution, which essentially means that the entries of the output vector are between 0 and 1, and they add up to 1. A prominent choice here is the softmax function, given by
$$p:{\mathbb{R}}^C\to R^C,\left[\begin{array}{c}{a}_1\\ ⋮\\ a_C\end{array}\right]\mapsto \frac{1}{{\sum }_c\exp a_c}\left[\begin{array}{c}\exp a_1\\ ⋮\\ \exp a_C\end{array}\right]$$

Exercise: Calculate the Jacobi-Matrix of the softmax function

The loss function we need to train the network must measure how well the predicted distribution corresponds to the distribution observed through our training set $\left({\mathbf{x}}_i,{\mathbf{y}}_i\right),i=1,\dots ,n$ . Note that these observed distributions are deterministic, that is, if${\mathbf{x}}_i$tagged $c$或0, let$y_c:=\mathrm{Pr}\left(\mathcal{Y}={\mathbf{e}}_c\mid \mathcal{X}=x_i\right)=1$ . In general, compare two probability distributions$\mathbb{P}=\left(p_1,\dots p_C\right)$和$\mathbb{Q}=\left(q_1,\dots q_C\right)$ CCon the same basis$A\; common\; approach\; to\; C$ event collections is to use cross-entropy, seeWikipedia.

$$H(\mathbb{P},\mathbb{Q})=-\sum _{c=1}^C{p}_c\log q_c.$$

In our case, there is a distribution that is deterministic, which can be simplified to the loss function
$$L\left(f(\mathcal{X}),{\mathbf{y}}_c\right)=-\log f(\mathcal{X}{)}_c$$

Among them $f$ is a FNN with softmax as the output,$f(\mathcal{X}{)}_c$indicates its $c$ entry. For training, as usual we use the empirical expectation of the loss on the training data, which leads to the optimization problem
$$\left({\hat{\mathbf{W}}}_l,\dots ,{\hat{W}}_1\right)=\arg \underset{\left({\mathbf{W}}_l,\dots ,{\mathbf{W}}_1\right)}{\min }\left\{-\frac{1}{n}\sum _i\log f_{\left({\mathbf{W}}_{l,\dots },{\mathbf{W}}_1\right)}{\left({\mathbf{x}}_i\right)}_{c_i}\right\}$$

5.4 CVXOPT Example

Machine learning tasks are often thought of as optimization problems, e.g., minimizing an error function or maximizing a probability. Ideally, the optimization problem is convex, meaning that any local minimum is the global minimum of the recipe. It is assumed that you already have some basic knowledge about convex optimization. The purpose of this task is to familiarize us with CVXOPT, one of the most widely used convex optimization toolboxes. Note: If CVXOPT doesn't accept a NumPy array, try converting it to it double.

1. Go to cvxopt.org and install according to the installation instructions of the distribution version. For conda, you need to run
   conda install -c conda-forge crxopt
2. Browse the examples section on cvxopt.org for an overview of the capabilities of CVXOPT's different solvers.
3. Implement a function minsqthat takes as input a (m, n)NumPy array A of shape and a (m,)NumPy array y of shape as parameters and returns a (n,)NumPy array $x$ , solve the following problems.
   $$\underset{\mathbf{x}}{\min }\Vert \mathbf{A}\mathbf{x}-y\Vert$$
   Test the function with appropriate inputs andnp.linalg.pinvcompare the results with those obtained using y ∥. Experiment by adding white Gaussian noise to y.

5.4.1 Introduction to CVXOPT

5.4.1.1 Creating a matrix

CVXOPT has separate dense and sparse matrix objects. This example illustrates different ways of creating dense and sparse matrices.

Use this matrix()function to create a dense matrix; it can be created from a list (or iterator):

>>> from cvxopt import matrix
>>> A = matrix([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], (2,3))
>>> print(A)
[ 1.00e+00  3.00e+00  5.00e+00]
[ 2.00e+00  4.00e+00  6.00e+00]
>>> A.size
(2, 3)

or from a list of lists, where each inner list represents a column of the matrix:

>>> B = matrix([ [1.0, 2.0], [3.0, 4.0] ])
>>> print(B)
[ 1.00e+00  3.00e+00]
[ 2.00e+00  4.00e+00]

More broadly, inner lists can represent block columns.

print(matrix([ [A] ,[B] ]))
[ 1.00e+00  3.00e+00  5.00e+00  1.00e+00  3.00e+00]
[ 2.00e+00  4.00e+00  6.00e+00  2.00e+00  4.00e+00]

For more information, refer to the documentation .

5.4.1.2 Matrix index

There are two ways to index dense and sparse matrices: single-parameter indexing and two-parameter indexing. In two-parameter indexing, the matrix is ​​indexed using two index sets I and J.

>>> from cvxopt import matrix
>>> A = matrix(range(16),(4,4))
>>> print(A)
[  0   4   8  12]
[  1   5   9  13]
[  2   6  10  14]
[  3   7  11  15]
>>> print(A[[0,1,2,3],[0,2]])
[  0   8]
[  1   9]
[  2  10]
[  3  11]

An index set can be an integer, a list, a matrix of integers, or a slice.

>>> print(A[1,:])
[  1   5   9  13]
>>> print(A[::-1,::-1])
[ 15  11   7   3]
[ 14  10   6   2]
[ 13   9   5   1]
[ 12   8   4   0]

In single-parameter indexing, a matrix is ​​indexed in vector form by considering the matrix in column-major order (that is, by stacking the columns from left to right).

>>> A[::5] = -1
>>> print(A)
[ -1   4   8  12]
[  1  -1   9  13]
[  2   6  -1  14]
[  3   7  11  -1]

This is useful for accessing parts of a matrix that are not submatrices, for example, the diagonal part of a matrix.

5.4.1.3 Solving linear programs

solvers.lp()Linear programs can be specified through this function. For example, we can solve this problem

$$\begin{array}{ll}\text{minimize}& 2x{\_}_1+x_2\\ \text{subject to}& -x_1+x_2\le 1\\ & x_1+x_2\ge 2\\ & x_2\ge 0\\ & x_1-2x{\_}_2\le 4\end{array}$$

as follows:

>>> from cvxopt import matrix, solvers
>>> A = matrix([ [-1.0, -1.0, 0.0, 1.0], [1.0, -1.0, -1.0, -2.0] ]) # the factors of the bounds
>>> b = matrix([ 1.0, -2.0, 0.0, 4.0 ]) # constants
>>> c = matrix([ 2.0, 1.0 ]) # minimized function
>>> sol=solvers.lp(c,A,b)
     pcost       dcost       gap    pres   dres   k/t
 0:  2.6471e+00 -7.0588e-01  2e+01  8e-01  2e+00  1e+00
 1:  3.0726e+00  2.8437e+00  1e+00  1e-01  2e-01  3e-01
 2:  2.4891e+00  2.4808e+00  1e-01  1e-02  2e-02  5e-02
 3:  2.4999e+00  2.4998e+00  1e-03  1e-04  2e-04  5e-04
 4:  2.5000e+00  2.5000e+00  1e-05  1e-06  2e-06  5e-06
 5:  2.5000e+00  2.5000e+00  1e-07  1e-08  2e-08  5e-08
>>> print(sol['x'])
[ 5.00e-01]
[ 1.50e+00]

5.4.1.4 Solving quadratic programming

Quadratic programming can solvers.qp()be solved by this function. For example, we can solve QP

$$\begin{array}{ll}\text{minimize}& 2x{\_}_1^2+x_2^2+x_1{x}_2+x_1+x_2\\ \text{subject to}& x_1\ge 0\\ & x_2\ge 0\\ & x_1+x_2=1\end{array}$$
as follows:

>>> from cvxopt import matrix, solvers
>>> Q = 2*matrix([ [2, .5], [.5, 1] ])
>>> p = matrix([1.0, 1.0])
>>> G = matrix([[-1.0,0.0],[0.0,-1.0]])
>>> h = matrix([0.0,0.0])
>>> A = matrix([1.0, 1.0], (1,2))
>>> b = matrix(1.0)
>>> sol=solvers.qp(Q, p, G, h, A, b)
     pcost       dcost       gap    pres   dres
 0:  0.0000e+00  0.0000e+00  3e+00  1e+00  0e+00
 1:  9.9743e-01  1.4372e+00  5e-01  4e-01  3e-16
 2:  1.8062e+00  1.8319e+00  5e-02  4e-02  5e-16
 3:  1.8704e+00  1.8693e+00  6e-03  2e-03  1e-15
 4:  1.8749e+00  1.8748e+00  2e-04  6e-05  6e-16
 5:  1.8750e+00  1.8750e+00  2e-06  6e-07  7e-16
 6:  1.8750e+00  1.8750e+00  2e-08  6e-09  1e-15
>>> print(sol['x'])
[ 2.50e-01]
[ 7.50e-01]

5.4.2 Example solutions

Implement a function minsqthat takes as input a (m, n)NumPy array A of shape and a (m,)NumPy array y of shape as parameters and returns a (n,)NumPy array $x$ , solve the following problems.
$$\underset{x}{\min }\Vert \mathbf{A}\mathbf{x}-y\Vert$$
Test the function with appropriate inputs andnp.linalg.pinvcompare the results with those obtained using y ∥. Experiment by adding white Gaussian noise to y.

from cvxopt import matrix, solvers
import numpy as np

def minsq(A, y):
    P=matrix(np.dot(A.T,A).astype('double'))
    q=matrix(-np.dot(A.T,y).astype('double'))
    x=solvers.qp(P,q)
    return np.array(x['x'])

A=np.array([[10, 40],[20, 0],[-30, 40]])
y=np.array([50,20,10])+np.random.randn(3,)

print('A:', A)
print('y:', y)
print('x:', minsq(A,y).squeeze())
print('np.dot(pinv(A),y):', np.dot(np.linalg.pinv(A),y))

The results are as follows

A: [[ 10  40]
 [ 20   0]
 [-30  40]]
y: [49.88665691 19.21406554  9.38923507]
x: [0.99519146 0.98974651]
np.dot(pinv(A),y): [0.99519146 0.98974651]