Matlab Gradient Descent Method

1. Introduction

Gradient descent or steepest descent is a common method for solving unconstrained optimization problems.

Suppose fx) is a function with first order continuous partial derivatives on R. The unconstrained optimization problem to be solved is . Its essence is an iterative method, select the initial value x0, and then iteratively update x0 to achieve the minimum value of the objective function until the end of convergence .

2. Case Analysis

f(x,y)=(x^2+y^2)/2, the minimum value is iterated. ,

step

1. Set the initial value x0 y0, step size L, and convergence accuracy e

2. Find the partial derivative dx df of the function f(x)

3. Calculate a=f(x0,y0),  

            x0=x0-L*dx(x,y);

            y0=y0-L*dy(x,y)

            b=f(x0,y0),

update x0 y0

4. Determine whether abs(ab)>e converges? Return 3 to start the cycle: end and return x0 y0

Matlab code:

% fun ：函数原型
% dx fun 的偏x导数
% dy fun 的偏y导数
% x0 y0初始值
% e 精度
% l 迭代步长 

% 
function [newX,newY,num_iterator,point] = Gradient_Descent(fun,dfunx,dfuny,x0,y0,e,l)
   value_a=feval(fun,x0,y0);
   % 开始计算第一次迭代
   % x_iterator=x-l*dx
   dx=feval(dfunx,x0,y0);
   newX=x0-l*dx;
   % 同理可以计算出新的y 
   dy=feval(dfuny,x0,y0);
   newY=y0-l*dy;
    % 开始计算第二次的值
    num_iterator=1;
    value_b=feval(fun,newX,newY);
    point(num_iterator,:) = [newX,newY,value_a];
    
    while(abs(value_a-value_b)>e)
       value_a=feval(fun,newY,newY);
      % x_iterator=x-l*dx
      dx=feval(dfunx,newX,newY);
      newX=newX-l*dx;
     % 同理可以计算出新的y 
      dy=feval(dfuny,newX,newY);
      newY=newY-l*dy;
      num_iterator=num_iterator+1;
      value_b=feval(fun,newX,newY);
      point(num_iterator,:) = [newX,newY,value_b];
    end
end

main function:

% 目标函数为 z=f(x,y)=(x^2+y^2)/2
close all;
clear all;
clc
fun = inline('(x^2+y^2)/2','x','y');  % 函数(x^2+y^2)/2'
dfunx = inline('x','x','y');          %对x的导数
dfuny = inline('y','x','y');          %对y的导数
x0 = 3;               % 初始位置
y0 = 3;
Epsilon1 = 0.00000000001;   % 精度
Lambda1 = 0.01;        % 步长/更新率%求解
[x,y,n,point] = Gradient_Descent(fun,dfunx,dfuny,x0,y0,Epsilon1,Lambda1)%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% figure 画图
x = -0.1:0.1:4;
y = x;
[x,y] = meshgrid(x,y);
z = (x.^2+y.^2)/2;
surf(x,y,z)    %绘制三维表面图形
xlabel('X');ylabel('Y');zlabel('z')% hold on
% plot3(point(:,1),point(:,2),point(:,3),'linewidth',1,'color','black')
hold on
scatter3(point(:,1),point(:,2),point(:,3),'r','*');

 

 Matlab implements gradient descent method - Programmer Sought

shortcoming

Disadvantages of gradient descent method:

(1) The convergence speed slows down when approaching the minimum value, as shown in the figure below;

(2) There may be some problems when searching in a straight line;