Quick Derivation of Hamilton-Jacobi-Bellman Equation

1. Settings

Objective:

$$J(T)={\int }_{t_0}^{T}L(x(t),u(t))dt+\varphi (x_T)$$

Dynamic programming:

$$V(x,t)=\underset{u(t)}{min}\{{\int }_t^{t+h}L(x(\tau ),u(\tau ))d\tau +V(x(t+h),t+h)\}$$

System:

$$\dot{x}=f(x,u)$$

2. Approximation

Zero-order approximation:

$${\int }_t^{t+h}L(x(\tau ),u(\tau ))d\tau \approx hL(x(t),u(t))$$

First-order approximation (Taylor expansion):

$$\begin{array}{rl}V(x(t+h),t+h)& \approx V(x(t),t)+h\frac{dV}{dt}\\ & =V(x(t),t)+h\frac{\partial V}{\partial x}\frac{\partial x}{\partial t}+h\frac{\partial V}{\partial t}\\ & =V(x(t),t)+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t}\end{array}$$

3. HJB Equation

Substitue the approximations for the exact representations in the dynamic programming.

$$\begin{array}{rl}V(x,t)& =\underset{u(t)}{min}\{hL(x(t),u(t))+V(x,t)+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t}\}\\ & =V(x,t)+\underset{u(t)}{min}\{hL(x(t),u(t))+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t}\}\end{array}$$

$$0=\underset{u(t)}{min}\{hL(x(t),u(t))+h\frac{\partial V}{\partial x}f(x,u)+h\frac{\partial V}{\partial t}\}$$

$$\therefore 0=\frac{\partial V}{\partial t}+\underset{u}{min}\{L(x,u)+\frac{\partial V}{\partial x}f(x,u)\}$$