Quick Derivation of Hamilton-Jacobi-Bellman Equation

1. Settings

Objective:

\[J(T)=\int_{t_0}^T L(x(t),u(t))dt+\phi(x_T)\]

Dynamic programming:

\[V(x,t)=\min_{u(t)}\left\{\int_t^{t+h}L(x(\tau),u(\tau))d\tau+V(x(t+h),t+h)\right\}\]

System:

\[\dot{x}=f(x,u)\]

2. Approximation

Zero-order approximation:

\[ \int_t^{t+h}L(x(\tau),u(\tau))d\tau \approx hL(x(t),u(t))\]

First-order approximation (Taylor expansion):

\[\begin{aligned}V(x(t+h),t+h)&\approx V(x(t),t)+h\frac{dV}{dt}\\&= V(x(t),t)+h\frac{\partial V}{\partial x}\frac{\partial x}{\partial t}+h\frac{\partial V}{\partial t} \\ &= V(x(t),t)+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t} \end{aligned}\]

3. HJB Equation

Substitue the approximations for the exact representations in the dynamic programming.

\[\begin{aligned} V(x,t)&=\min_{u(t)}\left\{hL(x(t),u(t))+V(x,t)+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t}\right\} \\ &= V(x,t)+\min_{u(t)}\left\{hL(x(t),u(t))+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t}\right\}  \end{aligned}\]

\[ 0 =\min_{u(t)}\left\{hL(x(t),u(t))+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t}\right\}\]

\[\therefore\; 0 =\frac{\partial V}{\partial t}+\min_{u}\left\{L(x,u)+\frac{\partial V}{\partial x}f(x,u)\right\}\]