Paper Reading (87): Accelerated Proximal Gradient Methods for Nonconvex Programming

1 Overview

1.1 Topics

2015: Accelerated proximal gradient methods for nonconvex programming

Attachments and codes are as follows:

1. https://zhouchenlin.github.io/Publications/2015-NIPS-APG_supp.pdf
2. https://zhouchenlin.github.io/NIPS2015_code.zip

The experimental data set is as follows:

1. https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/real-sim.bz2
2. https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html

1.2 Summary

Non-convex and non-smooth problems in image and signal processing have attracted much attention, among which Accelerated proximal gradient ( APG ) is an excellent method to deal with this problem. However, in non-convex programming, it is not known whether it can converge to a critical point.

This paper extends APG to general non-convex and non-smooth problems by introducing a monitor that satisfies the sufficient descent property, and proposes monotone APG and nonmonotone APG . Among them, nonmonotone APG does not require monotonous reduction of the objective function, and requires less calculation for each iteration.

Within the framework of existing knowledge, this is the first APG-type algorithm proposed for general non-convex and non-smooth optimization problems, which can ensure that each accumulation point is a critical point , and when the problem is convex optimization, convergence The rate is $O(\frac{1}{k^2})$ , where$k$ is the number of iterations. The final digital calculation verifies the advantage of the algorithm in speed.

1.3 References

@article{
    
    Li:2015:19,
author		=	{
    
    Huan Li and Zhou Chen Lin},
title		=	{
    
    Accelerated proximal gradient methods for nonconvex programming},
journal		=	{
    
    {
    
    NeurIPS}},
pages		=	{
    
    1--9},
year		=	{
    
    2015},
volume		=	{
    
    28},
}

2 introduction

In recent years, sparse and low-rank learning has received great attention and has been widely used in signal and image processing, statistics and machine learning. $l_1$Norm and kernel norm respectively as $l_0$Continuous and convex surrogates for norm and rank. However in many cases they are not optimal as they can only achieve sparsity and low rank under very limited conditions. For this, even non-convex regularizers are proposed, such as $l_p$Norm, Capped- $l_1$Penalty, Log-Sum Penalty, Maximum Bump Penalty, Geman Penalty, Smooth Truncated Absolute Deviation, and Schatte- $p$ -norm. Both of them aim to solvenon-convex and non-smooth problems:
$$\underset{\mathbf{x}\in {\mathbb{R}}^n}{\min }F(\mathbf{x})=f(\mathbf{x})+g(\mathbf{x}),\text{\hspace{1em}( 1 )}$$ where$f$ is differentiable, can be non-convex, and$g$ is non-convex and non-smooth.

Accelerated gradient methods are at the heart of convex optimization research, and at the same time several methods of this class have been proposed to deal with Equation 1. Among these methods, $k$ iterations are enough to find O ( 1 k 2 ) O(\frac{1}{k^2})from the best target value$O(\frac{1}{k^2})$ solutions within error. Recently, Ghadimi and Lan proposed aunified form of accelerated gradient(UAG) for convex, non-convex, and stochastic optimization. They proved their algorithm in non-convex$f$ and convex$Converges\; in\; non-convex\; programs\; under\; g$ , and in$O(\frac{1}{k^2})$ speeds up the convergence. Further, they analyzed the convergence rate on gradient maps.

Attouch et al. propose a unified framework to prove the convergence of the general descent method for solving Equation 1 using the Kurdyka-Łojasiewicz ( KL ) inequality. Frankel et al studied the anti-singularization function $\phi coefficient$ C θ t θ \frac{C}{\theta}t^\theta$\frac{C}{i}{t}^{}$The convergence rate of the general descent method in the form of${}^{\theta \; .}$A typical example in their framework is the proximal gradient method. However, there is currently no literature showing the existence of an accelerated gradient method that satisfies the constraints of their framework.

Other solutions to Equation 1 include inertial forward-backward (Inertial forward-backwar, IFB ), iPiano, general iterative shrinkage and thresholding ( GIST ), proximal average gradient descent (Gradient descent with proximal average, GDPA ), and Iteratively reweighted ( IR ), Table 1 provides a proper summary of these methods.

The purpose of this paper is to extend the APG algorithm of Beck and Teboulle to make it suitable for non-convex and non-smooth problems. APG first combines the current point and the previous point to infer the point ${\mathbf{y}}_k$, and then solve a proximal mapping problem. When extending APG to non-convex programming, the main difficulty is how to infer $y_k$. When convexity does not exist, we for $F({\mathbf{y}}_k)$ are almost unrestricted. In fact, when$y_k$is a bad inference when $F\left(y_k\right)$ compared to$F({\mathbf{x}}_k)$ is arbitrarily large. When in a bad${\mathbf{y}}_k$Calculate xk + 1 \mathbf{x}_{k+1} by proximal mapping${\mathbf{x}}_{k+1}$时，$F({\mathbf{x}}_{k+1})$ may also be compared to$F\left(x_k\right)$ is arbitrarily large. Beck and Teboulle's monotone APG ensures that$F(x_{k+1})\le F\left(x_k\right)$ . However, this does not ensure convergence to the critical point. For this, weintroduce a monitor satisfying the sufficient descent property to prevent and correct bad$y_k$。

2 preparation

2.1 Basic assumptions

For the function $g\ne \varnothing$ ,$g:{\mathbb{R}}^n\to (-\infty ,+\infty ]$ ,ifg = { x ∈ R : g ( x ) < + ∞ } g=\{ \mathbf{x} \in \mathbb{R}: g(\mathbf{x})<+\infty\ } }g={ x∈R:g(x)<+ ∞ } Let$\lim {\inf }_{\mathbf{x}\to {\mathbf{x}}_0}g\left(x\right)\ge g({\mathbf{x}}_0)$ , then$g$ at ​​point${\mathbf{x}}_0$It is the second half continuous. In Equation 1, assume $f$ hasa Lipschitz continuous gradient, and$g$ is semi-continuous. Suppose$F\left(x\right)$ is mandatory, i.e.$F$ has a lower bound and when$\Vert \mathbf{x}\Vert \to \infty$ , there is$F(\mathbf{x})\to \infty$ , where$\Vert \cdot \Vert$ is$l_2$norm.

2.2 KL inequality

Definition 1 : If there exists $the\in (0,+\infty ]$、$\overline{\mathbf{u}}$Neighborhood $U$ , and the function$Phi\in {Phi}_{}$, then the function ${\mathbb{R}}^n$在u ‾ ∈ dom ∂ f : { x ∈ R n : ∂ f ( u ) ≠ ∅ } \overline{\mathbf{u}}\in dom\partial f:\{\mathbf{x} \in \mathbf {R}^n:\partial f(\mathbf{u})\neq\empty\}u∈dom∂f:{ x∈Rn:∂f(u)=∅ }有KL property。 For example, if allu ∈ U ⋂ { u ∈ R n : f ( u ‾ ) < f ( u ) < f ( u ‾ ) + η } \mathbf{u}\in U\bigcap\ { \mathbf{u} \in \mathbf{R}^n: f(\overline{\mathbf{u}}) < f(\mathbf{u}) < f(\overline{\mathbf{u}}) + \eta \}u∈U⋂{ u∈Rn:f(u)<f(u)<f(u)+η}，以下不等式成立：
$${Phi}^{\prime }(f(\mathbf{u})-f(\overline{\mathbf{u}}))dist(0,\partial f(\mathbf{u}))>1,\text{\hspace{1em}( 2 )}$$ Among them,${Phi}_{}$Represents a class of functions $Phi:[0,h)\to {\mathbb{R}}^{+}$：

1. $\phi$ is concave, and$C^1$ is located at$(0,h)$ ;
2. $\phi$ at$0$ is continuous, and$\phi \left(0\right)=0$；
3. ${Phi}^{\prime }(x)>0,\forall \mathbf{x}\in (0,h)$ .

All semi-algebraic functions and subanalytical functions satisfy the KL property , among which semi-algebraic functions are a class of functions, which have the properties of semi-algebraic operations, that is, they can be expressed in the form of complex numbers. Their domain of definition is a closed subset of the complex plane, and its image is an open set. Subanalytical functions refer to functions with analytic properties, their domain of definition is a boundary on the complex plane, and their images can be completely expressed as a finite sum of many simple functions. In particular, the semi-algebraic function desingularization function $\phi \left(t\right)$ can be chosen as$\frac{C}{i}{t}^{The\; form\; of\; \theta }$ , where$i\in (0,1]$ . Typical semi-algebraic functions include polynomial functions$\Vert x{\Vert }_p$, where $p\ge 0$、$\text{rank}\left(X\right)$ , indicator functions for PSD cones, Stiefel popularity, and constant rank matrices.

2.3 APG under convex programming

First review APG under convex programming. Bech and Teboulle extend Nesterov's accelerated gradient method to non-smooth emotions. This is named the accelerated proximal gradient method , which consists of the following steps:
$${\mathbf{y}}_k={\mathbf{x}}_k+\frac{t_{k-1}-1}{t_k}({\mathbf{x}}_k-x_{k-1}),\text{\hspace{1em}( 3 )}$$$$x_{k+1}={\text{prox}}_{\alpha kg}({\mathbf{y}}_k-a_k\nabla f({\mathbf{y}}_k)),\text{\hspace{1em}(4)}$$$$t_{k+1}=\frac{\sqrt{4(t_k{)}^2+1}+1}{2},\text{\hspace{1em}( 5 )}$$ wherethe proximal mappingis defined as${\text{prox}}_{}(\mathbf{x})={\mathrm{argmin}}_{\mathbf{u}}g(\mathbf{u})+\frac{1}{2a}\Vert \mathbf{x}-\mathbf{u}{\Vert }^2$ . APG is not a monotonic method, which means that$F({\mathbf{x}}_{k+1})$ is not necessarily greater than$F\left(x_k\right)$ small. Therefore, Beck and Teboulle further proposed monotonic APG, which contains the following steps:
$$\begin{array}{rl}& {\mathbf{y}}_k={\mathbf{x}}_k+\frac{t_{k-1}}{t_k}\left({\mathbf{z}}_k-{\mathbf{x}}_k\right)+\frac{t_{k-1}-1}{t_k}\left({\mathbf{x}}_k-x_{k-1}\right),\\ & {\mathbf{z}}_{k+1}={\mathrm{prox}}_{a_{k}g}\left({\mathbf{y}}_k-a_k\nabla f\left(y_k\right)\right),\\ & t_{k+1}=\frac{\sqrt{4{\left(t_k\right)}^2+1}+1}{2},\\ & x_{k+1}=\{\begin{array}{ll}{z}_{k+1},& \text{ if }F\left(z_{k+1}\right)\le F\left(x_k\right),\\ x_k,& \text{ otherwise. }\end{array}\end{array}\text{\hspace{1em}(6–9)}$$

3 APG for non-convex programming

In this section, two non-convex and non-smooth programming algorithms based on APG are proposed. We establish convergence in the non-convex case and have O ( 1 k 2 ) O(\frac{1}{k^2}) in the convergent case$O(\frac{1}{k^2})$ convergence. When the KL property is satisfied, we provide excellent results on the convergence rate.

3.1 Monotonic APG

Two reasons make convergence analysis difficult for usual APGs as follows:

1. ${\mathbf{y}}_k$may be a poor inference;
2. In the existing methods, only the descending property $F({\mathbf{x}}_{k+1})\le F(x_k)$。

To solve these problems, we need a monitor to correct yk \mathbf{y}_k when possible failures${\mathbf{y}}_k$, and the monitor should encourage sufficient descent properties , which is the key to converge to the critical point. The proximal gradient method is known to ensure adequate descent, so we use the proximal gradient step as a monitor:
$$\begin{array}{rl}& {\mathbf{y}}_k={\mathbf{x}}_k+\frac{t_{k-1}}{t_k}\left({\mathbf{z}}_k-x_k\right)+\frac{t_{k-1}-1}{t_k}\left({\mathbf{x}}_k-x_{k-1}\right),\\ & {\mathbf{z}}_{k+1}={\mathrm{prox}}_{a_{y}g}\left({\mathbf{y}}_k-a_y\nabla f\left(y_k\right)\right),\\ & {\mathbf{v}}_{k+1}={\mathrm{prox}}_{a_{x}g}\left(x_k-a_x\nabla f\left(x_k\right)\right),\\ & t_{k+1}=\frac{\sqrt{4{\left(t_k\right)}^2+1}+1}{2},\\ & x_{k+1}=\{\begin{array}{ll}{z}_{k+1},& \text{ if }F\left(z_{k+1}\right)\le F\left({\mathbf{v}}_{k+1}\right),\\ v_{k+1},& \text{ otherwise. }\end{array}\end{array}\text{\hspace{1em}( 10–14 )}$$ Indefinition$a_y<\frac{1}{L}$和$a_x<\frac{1}{L}$are fixed constants, or they can be initialized according to a Barzilai-Borwein backtracking linear search, $L$ is$\nabla$ Lipschitz constant in$f\; .$

Our algorithm is an extension of Beck and Teboulle's monotonic APG. The difference is the extra $v$ , which acts as a monitor, will be atx \mathbf{x}when$x\; is\; updated\; as\; a\; corrective\; step.$Another difference is that our algorithmensures sufficient descent:
$$F({\mathbf{x}}_{k+1})\le F\left(x_k\right)-\delta \Vert {\mathbf{v}}_{k+1}-{\mathbf{x}}_k{\Vert }^2,\text{\hspace{1em}( 15 )}$$ where$d>0$ is a small constant. It is not difficult to understand that descent property alone does not guarantee convergence to the critical point in non-convex programming. We therefore give the convergence result in the following theorem.

Theorem 1 : Let$f$ is a legal function with Lipschitz continuous gradient,$g$ is a lower semicontinuous legal function. For non-convex$f$ and non-smoothed$g$ , assuming$F\left(x\right)$ is coercive. Then, { x } \{\mathbf{x}\}generated by Eq. 10–14$\left\{x\right\}$和{ vk } \{ \mathbf{v}_k \}{ vk​} is bounded. Let${\mathbf{x}}^{\ast }$ means{ xk } \{ \mathbf{x}_k \}{ xk​} , then there are$0\in \partial F({\mathbf{x}}^{\ast })$，即$x^{\ast }$ is a critical point.

Theorem 2 : For convex$f$ and$g$ , assuming$\nabla f$ is Lipschitz continuous, let${\mathbf{x}}^{\ast }$ represents any global optimal solution, and then{ xk } \{ \mathbf{x}_k \}{ xk​} Equation:
$$F({\mathbf{x}}_{N+1})-F(x^{\ast })\le \frac{2}{a_y(N+1{)}^2}\Vert {\mathbf{x}}_0-x^{\ast }{\Vert }^2.\text{\hspace{1em}( 16 )}$$ When the objective function is locally convex in the neighborhood of the local minimum, Theorem 2 shows that when facing the local minimum, APG can ensure$O(\frac{1}{k^2})$ , thus speeding up the convergence.

For a better presentation, we summarize the process of monotonic APG in Algorithm 1:

3.2 Convergence rate under KL assumption

KL properties are a powerful tool. The commonly used APG does not satisfy the sufficient descent property, but this property is crucial for using the KL property, so there is no conclusion under the KL assumption. On the other hand, since the intermediate variable ${\mathbf{y}}_k$、${\mathbf{v}}_k$, and ${\mathbf{z}}_k$, our algorithm is more complex than the general descent method, and does not satisfy the conditions. However, due to the monitor step in Equations 12-16, some modified conditions are satisfied and we are still able to obtain satisfactory results under the KL assumption. Under the existing framework, we have the following theorem:

Theorem 3 : Let$f$ is a legal function with Lipschitz continuous gradient,$g$ is a lower semicontinuous legal function. For non-convex$f$ and non-smoothed$g$ , assuming$F\left(x\right)$ is coercive. If it is further assumed$f$ and$g$ satisfies the KL property, and for$C>0,i\in (0,1]$ , the desingularization function has the form$\phi \left(t\right)=\frac{C}{i}{t}^{\theta }$ , then:

1. If $i=1$ , then there exists$k_1$, such that for all $k>k_1$，有$F({\mathbf{x}}_k)=F^{\ast }$ , and the algorithm can stop in finite steps;
2. 如果$i\in \left[\frac{1}{2},1\right)$ , then there exists$k_2$, such that for all $k>k_2$，有
   $$F\left({\mathbf{x}}_k\right)-F^{\ast }\le {\left(\frac{d_1{C}^2}{1+d_1{C}^2}\right)}^{k-k_2}{r}_{k_2}.\text{\hspace{1em}(17)}$$
3. 如果$i\in \left(0,\frac{1}{2}\right)$ , then there exists$k_3$, for all $k>k_3$，有
   $$F\left({\mathbf{x}}_k\right)-F^{\ast }\le {\left(\frac{C}{\left(k-k_3\right)d_2(1-2i)}\right)}^{\frac{1}{1-2i}},\text{\hspace{1em}( 18 )}$$ where$F^{\ast }$在{ xk } \left\{\mathbf{x}_k\right\}{ xk​} have the same function value,$r_k=F\left({\mathbf{v}}_k\right)-F^{\ast }$，以及
   $$d_1={\left(\frac{1}{a_x}+L\right)}^2/\left(\frac{1}{2a_x}-\frac{L}{2}\right)\text{ and }{d}_2=\min \left\{\frac{1}{2d_{1}C},\frac{C}{1-2i}\left(2^{\frac{2\theta -}{2\theta -2}}-1\right)r_0^{2\theta -}\right\}.$$

当$When\; F\left(x\right)$ is a semi-algebraic function, the desingularization function$\phi \left(t\right)$ can be chosen as$\frac{C}{i}{t}^{\theta }$ form, where$i\in (0,1]$ [23]. In this case, as Theorem 3, when$i=1$ , our algorithm converges within a finite number of times, when$i\in [\frac{1}{2},1)$ converges to a linear rate, when$i\in \left(0,\frac{1}{2}\right)$ converges to a sublinear rate.

3.3 Non-monotonic APG

Algorithm 1 is a monotonic algorithm. When the problem is ill-conditioned, the monotonic algorithm must crawl along the bottom of the narrow and curved valley so that the objective function value does not increase, resulting in a short step size or even a zigzag, resulting in slow convergence. Removing the requirement for monotonicity improves convergence because larger step sizes can be used when using line search.

On the other hand, Algorithm 1 needs to calculate zk + 1 \mathbf{z}_{k+1} in each iteration${\mathbf{z}}_{k+1}$和${\mathbf{v}}_{k+1}$, and use $v_{k+1}$As a monitor to correct $z_{k+1}$. This is a conservative strategy, in fact, when $y_k$is a good guess, we can accept $z_{k+1}$as ${\mathbf{x}}_{k+1}$. Then only when the condition is not satisfied $v_{k+1}$been accepted. Therefore, we can even reduce the average number of proximal mappings, and we will focus on non-monotonic APG next.

In monotonic APG, Equation 15 is guaranteed. In non-monotonic APG, we allow $x_{k+1}$Obtain a ratio $F({\mathbf{x}}_k)$ a larger target value. In particular, we allow$x_{k+1}$Generate a less than $F\left(x_k\right)$ relaxation$c_k$target value. $c_k$Should not be greater than $F({\mathbf{x}}_k)$ mostly. Therefore$F(x_k),F\left(x_{k-1}\right),\dots ,F\left(x_1\right)$ is a good choice:
$$c_k=\frac{{\sum }_{j=1}^k{the}^{k-j}F\left({\mathbf{x}}_j\right)}{{\sum }_{j=1}^k{the}^{k-j}},\text{\hspace{1em}( 19 )}$$ η$the\in [0,1)$ Control the degree of non-monotonicity. Actually,$c_k$It can be calculated recursively by following steps: In practice $c_k$ can be efficiently computed by the following recursion:
$$\begin{array}{rl}{q}_{k+1}& =\eta q_k+1\\ c_{k+1}& =\frac{\eta q_k{c}_k+F\left({\mathbf{x}}_{k+1}\right)}{q_{k+1}},\end{array}\text{\hspace{1em}( 20–21 )}$$ where$q_1=1$ and $c_1=F\left({\mathbf{x}}_1\right)$。

According to formula 14, we pass ${\mathbf{x}}_{k+1}$Divide it into two parts. Therefore, in non-monotonic APG, we use the following condition instead of Equation 15:
$$\begin{array}{rl}& F\left({\mathbf{z}}_{k+1}\right)\le c_k-d{\Vert {\mathbf{z}}_{k+1}-{\mathbf{y}}_k\Vert }^2,\\ & F\left({\mathbf{v}}_{k+1}\right)\le c_k-d{\Vert {\mathbf{v}}_{k+1}-x_k\Vert }^2.\end{array}\text{\hspace{1em}( 22–23 )}$$ We use Equation 22 as the criterion described earlier. When formula 22 is satisfied,${\mathbf{y}}_k$is a good guess and directly accepts ${\mathbf{z}}_{k+1}$; Otherwise, calculate vk + 1 \mathbf{v}_{k+1} satisfying formula 23 through formula 2${\mathbf{v}}_{k+1}$. When backtracking linear search is used, vk + 1 \mathbf{v}_{k+1} satisfying formula 23 can be found within a limited number of times$v_{k+1}$。

Combining formulas 20–22, and ${\mathbf{x}}_{k+1}=z_{k+1}$Then:
$$c_{k+1}\le c_k-\frac{d{\Vert {\mathbf{x}}_{k+1}-y_k\Vert }^2}{q_{k+1}}.$$ Correspondingly, the formula 22 and$x_{k+1}=z_{k+1}$公司的公式23 ${\mathbf{x}}_{k+1}={\mathbf{v}}_{k+1}$Let:
$$c_{k+1}\le c_k-\frac{d{\Vert {\mathbf{x}}_{k+1}-x_k\Vert }^2}{q_{k+1}}$$This means $c_k$The sufficient descent condition of replaces $F\left({\mathbf{x}}_k\right)$ recombination drop condition.

Algorithm 2 summarizes the non-monotonic APG. Similar to monotonic APG, non-monotonic APG has convergence in non-convex cases and O ( 1 k 2 ) O\left(\frac{1}{k^2}\right) in convex programming$O\left(\frac{1}{k^2}\right)$ convergence rate. We then give the convergence result in Theorem 4. Theorem 2 can support Algorithm 2 without modification.

LetΩ 1 = { k 1 , k 2 , ⋯ , kj , ⋯ } \Omega_1=\left\{k_1, k_2, \cdots, k_j, \cdots\right\}Oh1​={ k1​,k2​,⋯,kj​,⋯}以及 Ω 2 = { m 1 , m 2 , ⋯   , m j , ⋯   } \Omega_2=\left\{m_1, m_2, \cdots, m_j, \cdots\right\} Oh2​={ m1​,m2​,⋯,mj​,⋯} , for example in Algorithm 2, for all$k=k_j\in {Oh}_1$，${\mathbf{x}}_{k+1}={\mathbf{z}}_{k+1}$be executed. For all $k=m_j\in {Oh}_2$, Formula 22 is not supported, and Formula 14 is implemented. Then we have ${Oh}_1\bigcap {Oh}_2=\varnothing$、Ω 1 ⋃ Ω 2 = { 1 , 2 , 3 , ⋯ \Omega_1 \bigcup \Omega_2=\{1,2,3, \cdotsOh1​⋃Oh2​={ 1,2,3,⋯ }, and have the following theorem.

Theorem 4 : Let$f$ is a legal function with Lipschitz continuous gradient,$g$ is a lower semicontinuous legal function. For non-convex$f$ and non-smoothed$g$ , assuming$F\left(x\right)$ is coercive. Then for$k_j\in {Oh}_1$， { x k } \{\mathbf{x}_k\} { xk​}、{ vk } \{ \mathbf{v}_k \}{ vk​} ,from${\mathbf{y}}_{k_j}$is the bound generated by Algorithm 2, then

1. If ${Oh}_1$Or ${Oh}_2$finite, then for { xk } \{\mathbf{x}_k\}{ xk​} Arbitrary cumulative point$\{x^{\ast }\}$，有$0\in \partial F({\mathbf{x}}^{\ast })$；
2. If ${Oh}_1$ and ${Oh}_2$infinite, then for $\left\{{\mathbf{x}}_{k_j+1}\right\}$ any cumulative point${\mathbf{x}}^{\ast }$ ,from$\left\{{\mathbf{y}}_{k_j}\right\}$ any cumulative point${\mathbf{y}}^{\ast }$ , where$k_j\in {Oh}_1$, and $\left\{{\mathbf{v}}_{m_j+1}\right\}$ any cumulative point${\mathbf{v}}^{\ast }$，$\left\{{\mathbf{x}}_{m_j}\right\}$ any cumulative point${\mathbf{x}}^{\ast }$ , where$m_j\in {Oh}_2$，有$0\in \partial F\left({\mathbf{x}}^{\ast }\right)$、$0\in \partial F\left({\mathbf{y}}^{\ast }\right)$，以及$0\in \partial F\left({\mathbf{v}}^{\ast }\right)$。

4 Numerical results

This section uses sparse logistic regression (Logistic regression, LR ) to test the performance of the algorithm. Sparse LR is an extension of LR that is able to perform feature selection while reducing overfitting. Sparse LR is widely used in bioinformatics and text classification. The objective function to be optimized in this paper is as follows, which considers sparse LR and adds a non-convex regularization:
$$\underset{\mathbf{w}}{\min }\frac{1}{n}\sum _{i=1}^n\log \left(1+\exp \left(-y_i{\mathbf{x}}_i^T\mathbf{w}\right)\right)+r(\mathbf{w}),\text{\hspace{1em}( 26 )}$$ where$r\left(w\right)$ is chosen as$l_1$惩罚：
$$r(\mathbf{w})=l\sum _{i=1}^d\min \left(\left|w_i\right|,i\right),i>0.$$

Algorithms compared include: 1) Monotonic APG (mAPG); (2) Non-monotonic APG (nmAPG); (3) mGIST; (4) nmGIST; and (5) IFB. The experimental data set is a real simulation data set, which contains a total of 72309 samples, and the sample dimension is 20958. The parameters are set as follows:

1. $l=0.0001$ θ$i=0.1\lambda$ , the starting point is set to zero vector;
2. For nmAPG, set $the=0.8$；
3. For IFB, $b=0.01$ , the Lipschitz constant is calculated by backtracking;
4. For the sake of fairness, first run mGIST, the termination condition of the algorithm is that the difference between the two relative target values ​​is less than $10^{-5}$ or the number of iterations is greater than$1000$；
5. The termination condition of other algorithms is to obtain a target value smaller than mGIST or to reach the maximum number of iterations;
6. The ratio of the training set to the test set is 9:1;
7. The experimental results are the average of 10 independent experiments.