[Machine Learning] Loss function and optimization process

---

In the framework of statistical learning, we usually try to find a function or concept that can explain or fit the given data well. This is usually achieved by minimizing some kind of risk or loss. To find a good model, we try in the predefined hypothesis space $Find\; a\; function\; in\; H$ that has the smallest Empirical Risk on the training data. However, our real goal is to find a function whose expected risk (for all possible data distributions) is minimized. This often requires balancing the complexity of the model with its performance on the data.

Suppose there is an optimal function or concept $c$ , which best fits the data, i.e.$c=\underset{h}{\mathrm{argmin}}R\left(h\right)$ , in which$R\left(h\right)$ is the function$h$ risk.

In practice, we cannot search in all possible function spaces, so we restrict the search to a predefined hypothesis class $H$ in. Given training sample$S=\{(X_1,Y_1),\dots ,(X_n,Y_n)\}$ , we can calculate the functionhhThe empirical riskRS ( h ) R_S(h)$of\; h$$R_S\left(h\right)$ is:

$$R_S(h)=\sum _{i=1}^n\ell (X_i,Y_i,h)$$

The expected risk is the average risk over all possible data distributions and is defined as:

$$R(h)=E[R_S(h)]=\mathbb{E}[\ell (X_i,Y_i,h)]$$

The best hypothesis (target concept) in the entire function space is:

$$c=\underset{h}{\mathrm{argmin}}R\left(h\right)$$

And in the predefined hypothesis class HHThe optimal hypothesis in$H\; is:$

$$h^{\ast }=\underset{h\in H}{\mathrm{argmin}}R\left(h\right)$$

Use empirical risk to estimate $h$ From the training data, we get:
$$h_S=\underset{h\in H}{\mathrm{argmin}}{R}_S(h)=\underset{h\in H}{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n\ell (X_i,Y_i,h)$$

Therefore, choose an appropriate hypothesis space and loss function $\ell$ is crucial.

Assume class $H$ is a set of functions, where each function attempts to map from input features to output labels,H = { h 1 , h 2 , … } H = \{ h_1, h_2, \dots \}H={ h1​,h2​,…} . Usually,$H$ is defined by a specific algorithm or model structure, such as linear regression, decision tree, etc.

First, the 0-1 loss function is the most direct classification error measure. For a given classifier $h$ , which simply counts the number of misclassified data points. Mathematically, this is defined as:$\underset{h}{\mathrm{argmin}}\mathbb{E}[1_{Y\ne sign(h(X))}]$ . But the problems we usually encounter are:

1. Distribution of real data $P(X,Y)$ is unknown, so we cannot directly calculate the above expectation.
2. The 0-1 loss is computationally difficult because it is discontinuous and non-convex, which complicates optimization.

The law of large numbers describes the relationship between the sample mean of a random variable and the population mean. It ensures that as the sample size approaches infinity, the sample mean approaches the overall mean. More formally, consider a random variable $X$ , whose expected value is$E\left[X\right]$ . For$X$ 's$n$ independent and identically distributed samples$X_1,X_2,\dots ,X_n$, their sample mean is defined as $\overline{X_n}=\frac{1}{n}{\sum }_{i=1}^n{X}_i$。当 $n\to \infty$ 时, $\overline{X_n}\to E[\_\_$ _

Through the law of large numbers, we can use these samples to estimate certain distribution-dependent quantities, such as expected loss. Suppose our goal is to estimate hh by the hypothesisThe expected loss E caused by$h$$\mathbb{E}[1_{Y\ne \text{sign}(h(X))}]$ . We can use samples D \mathcal{D}from the real distribution$D$ to estimate this expectation:

$$\frac{1}{n}\sum _{i=1}^n{1}_{Y_i\ne \text{sign}(h(X_i))}$$

With the number of samples $As\; n$ increases, the above estimate will approach the true expected loss.

To make the problem solvable in practice, we use so-called surrogate loss functions, which are more tractable in terms of optimization but still aim to approximate a 0-1 loss function.

• Hinge loss: This is the loss function used in support vector machines.
  ℓ ( X , Y , h ) = max ⁡ { 0 , 1 − Y h (ℓ(X,Y,h)=max{ 0,1−Yh ( X )}

• Logistic loss: This is used in logistic regression. It is more robust to outliers and provides good estimates of probabilities.

• Least square loss: Mainly used in regression problems.

• Exponential loss: is the loss function used in the AdaBoost algorithm.

Most popular alternative loss functions are designed to simulate the effect of a 0-1 loss function in the large sample limit. These are called classification-calibrated alternative loss functions. This means that if the training data is infinite, a classifier trained using these loss functions will perform consistently with the true best classifier on a 0-1 loss.

Given a surrogate loss function $\ell$ and the corresponding function$\varphi$ makesϕ$\varphi \left(Yh\right(X))=\ell (X,Y,h)$ . Here,$Y$ is the label, the value is$(-1,1)$ , and$h\left(X\right)$ is the classifier for input$X$ 's predicted score. To check$Whether\; \ell$ is classified calibrated, we usually check the following conditions:

1. $\varphi$ is convex.
2. $\varphi$ is differentiable at 0, and${\varphi }^{\prime }(0)<0$。

Satisfying the above conditions means that in most cases, for a given data point, the classifier $When\; h$ minimizes the agent loss, it also minimizes the 0-1 loss.

For example, consider Hinge lossℓ hinge ( X , Y , h ) = max ⁡ { 0 , 1 − Y h ( X ) } \ell_{\text{hinge}}(X,Y,h) = \max \{ 0 , 1-Yh(X) \}ℓhinge​(X,Y,h)=max{ 0,1−Yh ( X )}

The corresponding $The\; \varphi$ function isϕ ( z ) = max ⁡ { 0 , 1 − z } \phi(z) = \max \{ 0, 1-z \}ϕ ( z )=max{ 0,1−z}

This function is at $z=$It is not differentiable at$1$$z=$It is differentiable at$0\; ,\; and\; its\; derivative\; is\; less\; than\; 0,\; so\; the\; Hinge\; loss\; is\; classification\; calibrated.$

Now consider the following two classifier definitions:

• $h_s$is an optimal classifier based on limited training data and a surrogate loss function.
• $h_c$is the optimal classifier based on the entire data distribution and a 0-1 loss function.

Using a surrogate loss function and training data, we can find $h_s$：

$$h_s=\underset{h}{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n\ell (X_i,Y_i,h)$$

At the same time, if we know the distribution of the entire data, we can find $h_c$：

$$h_c=\underset{h}{\mathrm{argmin}}\mathbb{E}[1_{Y\ne \text{sign}(h(X))}]$$

When our training data is infinitely large, $h_s$Will be compared with hc h_c obtained using the 0-1 loss function$h_c$closer and closer. This can be expressed by the following formula:

$$E[1_{Y\ne \text{sign}(h_S(X))}]\stackrel{n\to \infty }{⟶}\mathbb{E}[1_{Y\ne \text{sign}(h_c(X))}]$$

This means that when we optimize a proxy loss based on a limited sample dataset, we are actually optimizing the empirical loss on that dataset. The law of large numbers guarantees that as the number of samples increases, the expectation of this empirical loss will be close to the true expected loss. Also, if our proxy loss is classification calibrated, then optimizing this proxy loss will implicitly optimize the 0-1 loss. As the size of the training data approaches infinity, the expected 0-1 loss of the classifier obtained by minimizing the surrogate loss function will approach the optimal 0-1 loss.

When the substitution loss function is convex and smooth, we can use a series of optimization algorithms, such as gradient descent, Newton's method, etc., to solve the following problem: h = arg min ⁡ h
$$h=\underset{h\in H}{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n\ell (X_i,Y_i,h)$$

Assuming that function (f(h)) is differentiable at point (h_k), we can use Taylor series to approximate the value of the function near (h_k): f ( x + Δ x ) ≈ f ( x ) + ∇
$f(x+\Delta x)\approx f(x)+\nabla f(x{)}^T\Delta x+\frac{1}{2}\Delta x{\_}^T{\nabla }^{2}f(x)\Delta x+\dots$

If we only consider the first and second terms of the above approximation (linear approximation), we get:
$f(h+\Delta h)\approx f(h)+\nabla f(h{)}^T\Delta h$

Now, consider our gradient descent update step: $h_{k+1}=h_k+the{d}_k$, where $d_k$是 descending direction，$\Delta h=the{d}_k$. We want to find a $\Delta h$ sakef$f(h+\Delta h)$ is as small as possible. From the linear approximation above, we can see that to make$f(h+$Increment of$\Delta$$h$$)$$\nabla f(h{)}^T\Delta h$is as small as possible,$\Delta h$ should be related to the gradient$\nabla \; Align\; in\; the\; opposite\; direction\; off\left(h\right)$ . Therefore, the negative direction of the gradient is usually chosen$-\nabla f\left(h\right)$ as the downward direction.

However, if we want to use the curvature information of the function at a certain point, or speed up the optimization process, we can introduce $D^k$ such matrix, matrix$D^k$ is often called the preconditioning matrix or scaling matrix.

Considering the general update rule $h_{k+1}=h_k-the{D}^k\nabla f(h_k)$ . Among them,$D^k$ is a positive definite symmetric matrix. Positive certainty ensures that the updated direction is the downward direction, that is,$\nabla f(h_k{)}^T{D}^k\nabla f(h_k)>0$。

• Steepest Descent

  $D^k=I$

  At this time, the matrix $D^k$ is the identity matrix, which means we only move in the negative direction of the gradient.

• Newton's Method

  $D^k=[{\nabla }^{2}f(h_k){]}^{-1}$

  Here, matrix $D^k$ is the inverse of the Hessian matrix, which gives the curvature information of the function. Using the inverse of the Hessian matrix can help the algorithm converge faster, but may be more computationally expensive, especially in high dimensions.

Suppose we have chosen the descent direction $d_k$,Exact Line Search Specifications: $the=\underset{the}{\mathrm{argmin}}f(h_k-\eta \nabla f(h_k))$

Simply put, it searches for a step size $\eta$ , such that from the current position$h_k$Start, move eta \eta along the gradient directionAfter$eta$$f$ reaches its minimum value. Although this method can find the optimal step size, in practical applications it can be expensive because each iteration requires solving a new optimization problem.

L-Lipschitz continuous describes a function whose growth rate is bounded. If a function satisfies Lipschitz continuity, then the rate of change of the function is finite between any two points and is constrained by a certain constant.

Given a function $f$ , if there exists a positive constant$L$ , such that for any two points x 1 x_1in the domain$x_1$and $x_2$, all satisfy the following conditions:
$$|f(x_1)-f(x_2)|\le L||x_1-x_2||$$
, then we call the function$f$ is L-Lipschitz continuous.

If the gradient is known to be L-Lipschitz continuous, then the learning rate can be chosen using the following formula:

$$h_{k+1}=h_k-\frac{1}{L}\nabla f(h_k)$$

• Gradient descent:

  • Assumptions:
    • Lipschitz gradient: This means that the change in gradient (or derivative) is subject to an upper limit.
    • Convex function: A function is at least locally convex over its domain.
  • Convergence rate:
    $O\left(\frac{1}{k}\right)$
    This means that the error of the algorithm decreases inversely with the number of iterations k.

• Gradient descent:

  • Assumptions:
    • Lipschitz gradient
    • Strong convexity: Stricter convexity than regular convex functions, which means that the function has a lower bound.
  • Convergence rate:
    $O{\left(1-\frac{}{L}\right)}^k$
    among them,$\mu$ is the strong convexity parameter,$L$ is the Lipschitz constant. This convergence rate is faster than simple convex functions.

• Newton's method:

  • Assumptions:
    • Lipschitz gradient
    • Strong convexity
  • Convergence rate:
    $\sum _{i=1}^k{r}_k$
    Among them, $r_k\to 0$ . This means that Newton's method converges very quickly, especially when approaching the optimal solution.

\textbf{Newton's method and its variants}:

Newton's method is an iterative optimization algorithm used to find the zeros of a function in the real or complex domain. In machine learning and optimization, Newton's method is often used to find the minimum value of the loss function. The core idea of ​​this method is to use the second-order Taylor series expansion of the function to iteratively approximate the zero point or minimum value of the function.

The advantage of Newton's method is that if the function is quadratic, the minimum can be found in one iteration.

• In practice, computing the Hessian matrix and its inverse can be difficult, especially when the dimensionality is high. Therefore, several practical variants of Newton's method have been proposed.

• Modify the Hessian to ensure that it is positive definite:
  To ensure that the Hessian matrix is ​​invertible and positive definite, we can modify it slightly, for example, by adding a regularization term to it.

• Calculate the Hessian every m iterations:
  Since the calculation of the Hessian matrix can be computationally intensive, one strategy is to calculate it every m iterations and use the most recently calculated Hessian in other iterations.

• Using only the diagonal of the Hessian:
  By considering only the diagonal elements of the Hessian matrix, the computational complexity can be greatly reduced. This method is called the diagonal Hessian Newton method.

• Quasi-Newton method:
  The quasi-Newton method is a method that aims to approximate the Hessian matrix without directly calculating it. Among them, BFGS (Broyden–Fletcher–Goldfarb–Shanno algorithm) and L-BFGS (Limited-memory BFGS) are the most famous quasi-Newton methods. L-BFGS is a memory-efficient version of BFGS that only stores the most recent updates of the Hessian matrix.

In machine learning, we are often concerned with two main sources of error: Approximation Error and Estimation Error. The approximation error is due to the predefined hypothesis space HHThe error caused by the restriction of$H$$h^{\ast }$ and target conceptccThe difference between$c\; .$The estimation error is the error due to the limited sample size, which is the hypothesis$h_S$and the optimal hypothesis $h^{\ast }$ The difference between.

If the target concept $c$ in the predefined hypothesis space$H$ , then the approximation error will be zero. However, while choosing a large and complex hypothesis space can reduce the approximation error, it will increase the estimation error because the model may overfit the data.

In practical applications, we would like to understand how well knowledge learned from limited training samples can be applied to new, unseen data. The PAC learning framework gives an upper bound on the number of training samples required to learn approximately optimal hypotheses from a predefined hypothesis class. This framework provides a theoretical foundation for the generalization performance of learning algorithms. It reveals the relationship between the size of the training data set, the complexity of the hypothesis classes, and the generalization error of the learned model.

Assume class $H$ is PAC learnable, which means that there is an algorithm that can select a hypothesis from a hypothesis class whose error on new data is less than ϵ \epsilon from the error of the best hypothesis in the hypothesis class$ϵ$ , and the probability of this happening is at least$1-d$ .

If there is a learning algorithm $A$ and the polynomial function$poly(\cdot ,\cdot )$ , such that for any$ϵ>0$ and$d>0$ , for all defined in$X\times$Distribution DDon$Y$$D$ , as long as the sample size$ninterpoly$ ( 1 δ , 1 ϵ ) poly(\frac{1}{\delta}, \frac{1} {$poly(\frac{1}{d},\frac{1}{ϵ})$ , then algorithm$A$ learned hypothesis$h_S$The following conditions:

p { R ( h S ) − min ⁡ h ∈ H R ( h ) ≤ ϵ } ≥ 1 − δ p\{ R(h_S) - \min\limits_{h \in H} R(h) \leq \epsilon\} \geq 1 - \delta p{ R(hS​)−h∈Hmin​R(h)≤ϵ }≥1−d

Among them, $ϵ$ is the upper bound of the error we can accept, and$1-\delta$ is the confidence we want,$R(h_S)$ is the minimum value of empirical risk,$R\left(h\right)$ is the minimum value of expected risk. .

If a hypothetical class $H$ is too complex, we may need exponential levels of training samples, that is,$n>exp(\frac{1}{d},\frac{1}{ϵ})$ to ensure that the above inequality holds. In this case, assume class$H$ is not learnable by PAC.

We use the Empirical Risk Minimization (ERM) algorithm to verify whether a hypothesized class is PAC learnable.

Empirical risk minimization (ERM) is a basic strategy in machine learning, whose goal is to find a hypothesis $h\in H$ minimizes the risk (or loss) on the training data. Formally, given a training set$S$ , the ERM strategy chooses a hypothesis$h$ to minimize the empirical risk$R_S(h)$。

$$h_S=\arg \underset{h\in H}{\min }{R}_S(h)$$

Among them $R_S\left(h\right)$ is the empirical risk, which is the average loss on the training data.

We use the Empirical Risk Minimization (ERM) algorithm to verify whether a hypothesized class is PAC learnable.

If the training data is large enough, then the hypothesis h S h_S selected by ERM$h_S$Performance on the training data (empirical risk) will be close to its performance on the entire distribution (real risk). However, even if the empirical risk is low, the real risk can be high. This phenomenon is called overfitting.

The more complex the hypothesis space is, the selected hypothesis $h_S$With the best hypothesis $h^{}$The difference between${}^{*\; may\; be\; even\; greater.}$The PAC framework provides a way to measure this difference and determine how much training data is needed to ensure that the difference between empirical risk and true risk is less than a certain threshold.

Therefore, to verify whether a hypothetical class is PAC learnable, we need to ensure that:

• For a sufficiently large training data set, the difference between the true and empirical risks of the hypotheses produced by ERM on that hypothesis class is less than a given threshold ϵ \ $ϵ$ .
• In order to achieve this, we need to estimate or bound the difference between the true risk and the empirical risk and ensure that it is less than $ϵ$ . This is usually done through complexity regularization and/or leveraging tools such as VC dimensionality, Rademacher complexity, etc.

The VC dimension (Vapnik-Chervonenkis dimension) is a tool used to measure the complexity of a hypothesis space (or model class). It provides us with a theoretical tool to judge how many sample data a given hypothesis class (or model class) needs to be learned and ensure good generalization performance.

Given hypothesis class $H$ , whose VC dimension is the size of the largest point set that can be broken up by the hypothesis class. In other words, if there exists size$d$ data set, such that for thisddFor any possible classification label of$d$$h\in H$ can perfectly separate the points, but for any size$d+$No set of$1$ can do this, then HHThe VC dimension of$H$$d$。

To understand more visually, consider a linear classifier on a two-dimensional plane. A linear classifier perfectly separates all possible classifications of any three points that are not on the same straight line. However, for four arbitrary points, we cannot guarantee that there is a linear classifier that can classify all possible $2^4=16$ tag combinations perfectly classify these four points. Therefore, the VC dimension of a linear classifier on a two-dimensional plane is 3.

A hypothesis class with high VC dimension is more complex than a hypothesis class with low VC dimension. This means that hypothesis classes with high VC dimensions may have a higher risk of overfitting.

VC inequality It provides the relationship between real risk (i.e. risk over the entire distribution) and empirical risk (i.e. risk over the training data). Given the VC dimensions of a hypothesis class and the size of a training data set, the VC inequality tells us the probability that the difference between true and empirical risks does not exceed a certain value.

$$\mathrm{Pr}(\left|R(h)-R_S(h)\right|>)\_\le 4m_H(2n)\_\exp \left(-\frac{n{ϵ}^2}{8}\right)$$

Where, $m_H\left(n\right)$ is the hypothesis space$The\; size\; of\; H$ that can be broken up isnnThe number of point sets of$n$$R\left(h\right)$ is the real risk,$R_S\left(h\right)$ is the empirical risk.

From the above inequality, we can estimate how many training samples are needed to achieve a specific generalization error. This is useful for determining whether a given hypothesis class is PAC learnable.

In PAC learning, in order to ensure that the learned hypothesis has good generalization performance, we need enough samples to control the difference between the real risk and the empirical risk. The VC dimension provides us with a way to estimate the required sample size, allowing us to verify whether the hypothesized class is PAC learnable.