Optimization algorithm------nonlinear least squares optimization

Non-linear least squares optimization is also called unconstrained minimal sum-of-squares function problem, the form is as follows

If $f\left(x\right)$ isxxThe linear function of$x$ . At this time, the problem becomes a linear least squares problem. There is a special function to solve this problem.

1. G-N method

It is derived from the Newton's algorithm of unconstrained optimization, because the objective function in the nonlinear least squares optimization problem is relatively simple, and the specific form of the Jacobian matrix can be obtained. Substituting it into the iterative formula of Newton's method, G can be obtained. -N method. The expression of the objective function, there is

so

Newton unconstrained optimization algorithm, a gradient of the objective function is substituted, there is

due to the $R(x^k)$The amount of calculation is relatively large. Ignore the available G–N method.

It is a locally convergent algorithm, which has a great dependence on the initial point. It is possible to converge only when the initial point is close to the minimum point.

2. Modified G-N method

In order to overcome the shortcomings of the G-N method, there are two amendments.
1. After an approximate minimum point xkx^k has been obtained$x^{After\; k}$ , calculate

where$a^{The}$ size of${}^k$ is solved by one-dimensional unconstrained optimization method$minS(x^k)$Determine, because each step requires a one-dimensional search, the amount of calculation is very large.
2. Select a small positive number$d_k$, Such that $S(x^k+d_k{v}^k)<S\left(x^k\right)$, this kind of scheme has a relatively small amount of calculation, and a simpler method can be used to determine$d^{}$The value of${}^k$ .

3. L-M method

When matrix $(J_k{)}^T{J}_k$When it is an ill-conditioned matrix, the correct solution may not be obtained by the G-N algorithm, even when $(J_k{)}^T{J}_k$When it is irreversible, the G-N method cannot be calculated. The L-M algorithm uses the method of coefficient matrix damping to transform the matrix $(J_k{)}^T{J}_k$The nature of the algorithm enables the algorithm to proceed. There are two main steps in the L-M algorithm, one is to solve the equation to

find the increment of the independent variable, and the other is the damping coefficient $\mu$ 's adjustment algorithm.