【RL-Notes】The Dynamic Programming Algorithm

DP algorithms

DP algorithms rest on the principle of optimality, which roughly as follows:
Let { u 0 ∗ , … , u N − 1 ∗ } \{u_0^*, \dots, u_{N-1}^*\} { u0∗​,…,uN−1∗​} be an optimal control sequence, which together with $x_0$ determines the corresponding state sequences { x 1 ∗ , … , x N ∗ } \{x_1^*, \dots, x_N^*\} { x1∗​,…,xN∗​} via the system equation $x_{k+1}=f_k(x_k,u_k)$ . Consider the subproblem whereby starting at $x_k^{\ast }$ at time $k$ and wish to minimize the cost-to-go from $k$ to time $N$:
$$\begin{array}{r}{g}_k(x_k^{\ast },u_k)+\sum _{m=k+1}^{N-1}{g}_m(x_m,u_m)+g_N(x_N)\end{array}$$
over { u k , … , u N − 1 } \{u_k, \dots, u_{N-1}\} { uk​,…,uN−1​} with $u_m\in U_m(x_m),m=k,\dots ,N-1$. Then the truncated optimal control seqence { u k ∗ , … , u N − 1 ∗ } \{u_k^*, \dots, u_{N-1}^*\} { uk∗​,…,uN−1∗​} is optimal for this subproblem.

The DP algorithms is based on this idea: it proceeds sequentially, by solving all the tail subproblems of a given time length, using the solution of the tail subproblems of shorter time length.

Optimal Sequence

The DP algorithm constructs functions sequentially:
$$J_N^{\ast }(x_N),J_{N-1}^{\ast }(x_{N-1}),\dots ,J_0^{\ast }(x_0)$$

DP Algorithm for Deterministic Finite Horizon Problems

Start with
$$J_N^{\ast }(x_N)=g_N(x_N)$$
for all $x_N$, and for $k=0,\dots ,N-1$, let
$$J_k^{\ast }(x_k)=\underset{u_k\in U_k(x_k)}{\min }[g_k(x_k,u_k)+J_{k+1}^{\ast }(f_k(x_k,u_k))]$$
The key fact about the DP algorithm is that for every initial state $x_0$, the number $J_0^{\ast }(x_0)$ obtained at the last step, is equal to the optimal cost $J^{\ast }(x_0)$. All states $x_k$ at time $k$, we have
$$J_k^{\ast }(x_k)=\underset{u_m\in U_m(x_m)}{\min }J(x_k;u_k,\dots ,u_{N-1})$$
where
$$J(x_k;u_k,\dots ,u_{N-1})=g_N(x_N)+\sum _{m=k}^{N-1}{g}_m(x_m,u_m)$$
$J_k^{\ast }(x_k)$ is the optimal cost for an $(N-k)$ stage tail subproblem that starts at state $x_k$ and time $k$, and ends at time $N$.

Proof:
By induction, the assertion holds for $k=N$ in view of the intial condition $J_N^{\ast }(x_N)=g_N(x_N)$. From above, we have
$$\begin{array}{rl}{J}_k^{\ast }(x_k)=& \underset{u_m\in U(x_m),m=k,\dots ,N-1}{\min }[g_N(x_N)+\sum _{m=k}^{N-1}{g}_m(x_m,u_m)]\\ & =\underset{u_k\in U_k(x_k)}{\min }[g_k(x_k,u_k)+\underset{u_m\in U_m(x_m),m=k+1,\dots ,N-1}{\min }[g_N(x_N)+\sum _{m=k+1}^{N-1}{g}_m(x_m,u_m)\left]\right]\\ & =\underset{u_k\in U_k(x_k)}{\min }[g_k(x_k,u_k)+J_{k+1}^{\ast }(f_k(x_k,u_k))]\end{array}$$

Once the functions $J_0^{\ast },\dots ,J_N^{\ast }$ have been obtained, an optimal control sequence { u 0 ∗ , … , u N − 1 ∗ } \{u_0^*, \dots, u_{N-1}^*\} { u0∗​,…,uN−1∗​} can be constructed by the following forward algorithm:
Construction of Optimal Control Sequence:
Set
$$u_0^{\ast }\in \arg \underset{u_0\in U_0(x_0)}{\min }[g_0(x_0,u_0)+J_1^{\ast }(f_0(x_0,u_0))]$$
and
$$x_1^{\ast }=f_0(x_0,u_0^{\ast })$$
Sequentially, for $k=1,2,\dots ,N-1$, set
$$u_k^{\ast }\in \arg \underset{u_k\in U(x_k^{\ast })}{\min }[g_k(x_k^{\ast },u_k)+J_{k+1}^{\ast }(f_k(x_k^{\ast },u_k))]$$
and
$$x_{k+1}^{\ast }=f_k(x_k^{\ast },u_k^{\ast })$$

Approximation in Value Space

The optimal control sequence construction is possible only after we have computed $J_k^{\ast }(x_k)$ by DP for all $x_k$ and $k$, but this is often prohibitively time-consuming. A similar forward algorithmic process can be used if the optimal cost-to-go functions $J_k^{\ast }$ are replaced by some approximations ${\tilde{J}}_k$.

Q-Factors and Q-Learning

$${\tilde{Q}}_k(x_k,u_k)=g_k(x_k,u_k)+{\tilde{J}}_{k+1}(f_k(x_k,u_k))$$
The associated computation of the approximately optimal control can be done through the Q-factor minimization:
$${\tilde{u}}_k\in \arg \underset{u_k\in U_k({\tilde{x}}_k)}{\min }{\tilde{Q}}_k({\tilde{x}}_k,u_k)$$
This suggests the possibility of using Q-factors in place of cost functions in approximation in value space schemes. Methods of this type use as starting point an alternative form of the DP algorithm, which instead of the optimal cost-to-go functions $J_k^{\ast }$, generates the optimal Q-factors, defined for all pairs $(x_k,u_k)$ and $k$ by:
$$Q_k^{\ast }(x_k,u_k)=g_k(x_k,u_k)+J_{k+1}^{\ast }(f_k(x_k,u_k))$$
Thus the optimal Q-factors are simply the expressions that are minimized in the right-hand of DP equation. The optimal cost function $J_k^{\ast }$ can be recovered from the optimal Q-factor $Q_k^{\ast }$ by means of
$$J_k^{\ast }(x_k)=\underset{u_k\in U_k(x_k)}{\min }{Q}_k^{\ast }(x_k,u_k)$$
Moreover, using the above relation, the DP algorithm can be written in an essentially equivalent form that involves Q-factors only
$$Q_k(x_k,u_k)=g_k(x_k,u_k)+\underset{u_{k+1}\in U_{k+1}}{\min }{Q}_{k+1}^{\ast }(f_k(x_k,u_k),u_{k+1})$$

Stochastic dynamic programming

This system includes a random ‘disturbance’ $w_k$, which is characterized by a probability distribution $P_k(\cdot |x_k,u_k)$ that may depend explicitly on $x_k$ and $u_k$, but not on values of prior disturbances $w_{k-1},\dots ,w_0$. The system has the form
$$x_{k+1}=f_k(x_k,u_k,w_k),k=0,1,\dots ,N-1$$

An important difference is that we optimize not over control sequences { u 0 , … , u N − 1 } \{u_0, \dots, u_{N-1}\} { u0​,…,uN−1​}, but rather over policies that consist of a sequence of functions
π = { μ 0 , … , μ N − 1 } \pi=\{\mu_0, \dots, \mu_{N-1}\} π={ μ0​,…,μN−1​}
where ${\mu }_k$ maps states $x_k$ into controls $u_k={\mu }_k(x_k)$, and satisfies the control constraints. Policies are more general objects than control sequences, and in the presence of stochastic uncertainty, they can result in improved cost, since they allow choices of controls $u_k$ that incorporate knowledge of the state $x_k$. Without this knowledge, the controller cannot adapt appropriately to unexpected values of the state, and as a result the cost can be adversely affected.

Source From

Reinforcement learning and optimal control