LM (Levenberg-Marquardt) algorithm and Gauss-Newton algorithm are two optimization algorithms for nonlinear least squares problems. They also have some similarities :
-
Iterative optimization: Both the LM algorithm and the Gauss-Newton algorithm use iterative methods to optimize parameter values to gradually reduce the fitting residual.
-
Nonlinear fitting: Both algorithms are suitable for the task of fitting nonlinear functions and show good results in fitting the differences between parameterized models and observed data.
-
Gradient-based optimization: Both the Gauss-Newton algorithm and the LM algorithm utilize the gradient information of the objective function. The Gaussian-Newton algorithm uses the first derivative (Jacobian matrix) to approximate the objective function, while the LM algorithm introduces an adjustment factor in the gradient descent direction.
-
Local search: Both algorithms are local search methods, that is, parameter values are updated in each iteration to approach the local optimal solution.
Although there are differences in implementation details and convergence properties between the LM algorithm and the Gauss-Newton algorithm, they are both common methods for nonlinear least squares optimization problems and share some similar ideas and principles.
The LM (Levenberg-Marquardt) algorithm and the Gauss-Newton algorithm are both optimization algorithms for nonlinear least squares problems. They have some differences in calculation methods and convergence properties .
-
Calculation:
- The Gauss-Newton algorithm transforms the nonlinear least squares problem into a series of linear least squares subproblems by iteratively linearly approximating nonlinear functions. At each iteration, parameter values are updated by solving the linear system.
- The LM algorithm combines the ideas of Gauss-Newton algorithm and gradient descent algorithm. In each iteration, the LM algorithm balances the update step sizes of the Gaussian-Newton algorithm and the gradient descent algorithm to better adapt to problems in different situations.
-
Convergence properties:
- The Gaussian-Newton algorithm may converge to a local minimum in the parameter space because it relies on second-order derivative information, and the positive certainty of the second-order derivative matrix does not necessarily hold.
- The LM algorithm introduces an adjustment factor (also called a damping factor) to better explore the parameter space when approaching the minimum value. Such a mechanism can provide better global convergence and be less sensitive to the choice of initial parameters.
To summarize, the Gauss-Newton algorithm is a simpler and faster iterative method, but may converge to local minima. The LM algorithm is more complex and slightly time-consuming in calculation, but has better global convergence performance. In practical applications, appropriate optimization algorithms can be selected according to the characteristics of the problem.
The following is an example code that uses C++ to implement the Gauss-Newton method to fit a set of data:
#include <iostream>
#include <Eigen/Dense>
using namespace Eigen;
// 高斯函数模型
double gaussian(double x, double a, double b, double c) {
return a * exp(-(x - b) * (x - b) / (2 * c * c));
}
// 高斯牛顿拟合算法
void fitGaussian(const VectorXd& xData, const VectorXd& yData, double& a, double& b, double& c) {
int n = xData.size();
int m = 3; // 参数个数
MatrixXd J(n, m); // 雅可比矩阵
VectorXd residual(n); // 残差向量
VectorXd delta(m); // 参数增量向量
// 设定初始参数值
a = 1.0;
b = 0.0;
c = 1.0;
// 设置最大迭代次数和收敛阈值
int maxIter = 100;
double epsilon = 1e-6;
for (int iter = 0; iter < maxIter; ++iter) {
// 构造雅可比矩阵和残差向量
for (int i = 0; i < n; ++i) {
double xi = xData(i);
double residual_i = yData(i) - gaussian(xi, a, b, c);
J(i, 0) = -residual_i / a;
J(i, 1) = residual_i * (xi - b) / (c * c);
J(i, 2) = residual_i * (xi - b) * (xi - b) / (c * c * c);
residual(i) = residual_i;
}
// 计算参数增量
delta = (J.transpose() * J).inverse() * J.transpose() * residual;
// 更新参数估计值
a += delta(0);
b += delta(1);
c += delta(2);
// 判断是否收敛
if (delta.norm() < epsilon)
break;
}
}
int main() {
// 原始数据
VectorXd xData(10);
VectorXd yData(10);
xData << 1, 2, 3, 4, 5, 6, 7, 8, 9, 10;
yData << 0.98, 1.89, 3.02, 4.15, 4.97, 6.05, 6.92, 8.01, 8.94, 10.02;
// 拟合参数
double a, b, c;
fitGaussian(xData, yData, a, b, c);
// 输出拟合结果
std::cout << "Fitted parameters:\n";
std::cout << "a = " << a << "\n";
std::cout << "b = " << b << "\n";
std::cout << "c = " << c << "\n";
return 0;
}
Output: 1 3 2
In this example, we define a Gaussian function gaussian()to describe the Gaussian function model. fitGaussian()The function implements the Gaussian-Newton fitting algorithm, where parameter increments are calculated based on the Jacobian matrix and residuals, and parameter estimates are updated based on the increments. A set of original data is given in the main function, and then fitGaussian()the function is called for fitting. Finally, output the fitting results.
It should be noted that this is just a simple example. In actual applications, it may be necessary to select the appropriate model and parameter optimization calculation according to the specific situation.
The following is sample code for Gaussian function fitting using the Levenberg-Marquardt (LM) algorithm:
#include <iostream>
#include <Eigen/Dense>
#include <unsupported/Eigen/LevenbergMarquardt>
using namespace Eigen;
// 高斯函数模型
struct GaussianModel {
template<typename T>
bool operator()(const T* const x, T* residual) const {
T a = params[0];
T b = params[1];
T c = params[2];
residual[0] = y - a * exp(-(x[0] - b) * (x[0] - b) / (2 * c * c));
return true;
}
double y;
Vector3d params;
};
// 高斯牛顿拟合算法
void fitGaussian(const VectorXd& xData, const VectorXd& yData, double& a, double& b, double& c) {
int n = xData.size();
Vector3d params;
params << 1.0, 0.0, 1.0; // 初始参数值
// 定义 LM 算法参数
NumericalDiff<GaussianModel> numericalDiff;
LevenbergMarquardt<NumericalDiff<GaussianModel>> lm(numericalDiff);
lm.parameters.maxfev = 100; // 最大迭代次数
lm.parameters.xtol = 1e-6; // 收敛阈值
// 构造问题并求解
GaussianModel model;
model.params = params;
for (int i = 0; i < n; ++i) {
model.y = yData(i);
VectorXd x(1);
x << xData(i);
lm.minimize(x, model);
params = model.params;
}
// 更新拟合结果
a = params[0];
b = params[1];
c = params[2];
}
int main() {
// 原始数据
VectorXd xData(10);
VectorXd yData(10);
xData << 1, 2, 3, 4, 5, 6, 7, 8, 9, 10;
yData << 0.98, 1.89, 3.02, 4.15, 4.97, 6.05, 6.92, 8.01, 8.94, 10.02;
// 拟合参数
double a, b, c;
fitGaussian(xData, yData, a, b, c);
// 输出拟合结果
std::cout << "Fitted parameters:\n";
std::cout << "a = " << a << "\n";
std::cout << "b = " << b << "\n";
std::cout << "c = " << c << "\n";
return 0;
}
In this example, we used LevenbergMarquardtclasses from the Eigen library to implement the LM algorithm. GaussianModelFirst, a structure is defined in which the bracket operator is overloaded, which calculates the residual and returns it to the LM algorithm. Then, in fitGaussian()the function, an object is constructed LevenbergMarquardtand the maximum number of iterations and convergence threshold are set. Next, use this object to fit each data point and update the parameter estimates. Finally, the fitting results are output.
Please note that in order to use the LM algorithm, you need to add the relevant header files of the Eigen library to the code and use the command `-I /path
For specific principles, please refer to: https://zhuanlan.zhihu.com/p/421272861?utm_id=0