Stochastic DP method

Given an initial state $x_0$ and a policy π={μ0,…,μN−1}\pi=\{\mu_0, \dots, \mu_{N-1}\}π={μ0​,…,μN−1​}, the future states $x_k$ and disturbances $w_k$ are random variables with distributions defined through the system equation
$$x_{k+1}=f_k(x_k,{\mu }_k(x_k),w_k),k=0,1,\dots ,N-1$$
Thus, for given functions $g_k,k=0,1,\dots ,N$, the expected cost of $\pi$ starting at $x_0$ is
$$J_{\pi }(x_0)=\mathbb{E}\{g_N(x_N)+\sum _{k=0}^{N-1}{g}_k(x_k,{\mu }_k(x_k),w_k)\}$$
An optimal policy ${\pi }^{\ast }$ is one that minimizes the cost, i.e.,
$$J_{{\pi }^{\ast }}(x_0)=\underset{\pi \in \Pi }{\min }{J}_{\pi }(x_0)$$
where $\Pi$ is the set of all policies.

The optimal cost depends on $x_0$ and is denoted by $J^{\ast }(x_0)$; i.e.,
$$J^{\ast }(x_0)=\underset{x\in \Pi }{\min }{J}_{\pi }(x_0)$$

DP algorithm for stochastic finite Horizon Problems

Start with
$$J_N^{\ast }=g_N(x_N)$$
for $k=0,\dots ,N-1$, let
$$J_k^{\ast }(x_k)=\underset{u_k\in U_k(x_k)}{\min }\mathbb{E}\{g_k(x_k,u_k,w_k)+J_{k+1}^{\ast }(f_k(x_k,u_k,w_k))\}$$
If $u_k^{\ast }={\mu }_k^{\ast }(x_k)$ minimizes the right side of this equation for each $x_k$ and $k$, the policy π∗={μ0∗,…,μN−1∗}\pi^*=\{\mu_0^*, \dots, \mu_{N-1}^*\}π∗={μ0∗​,…,μN−1∗​} is optimal.

Simultaneously with the off-line computation of the optimal cost-to-go functions $J_0^{\ast },\dots ,J_N^{\ast }$, we can compute and store an optimal policy π∗={μ0∗,…,μN−1∗}\pi^*=\{\mu_0^*, \dots, \mu_{N-1}^*\}π∗={μ0∗​,…,μN−1∗​}. We can then use this policy on-line to retrieve from memory and apply the control ${\mu }_k^{\ast }(x_k)$ once we reach state $x_k$.

Q-factors for stochastic problems

The Q-factors for a stochastic problem, similar to the case of deterministic problem, as the expressions that minimized in the right-hand side of stochastic DP equation:
$$Q_k^{\ast }(x_k,u_k)=\mathbb{E}=\{g_k(x_k,u_k,w_k)+J_{k+1}^{\ast }(f_k(x_k,u_k,w_k))\}$$
The optimal cost-to-go functions $J_k^{\ast }$ can be recovered from the optimal Q-factor $Q_k^{\ast }$ by means of
$$J_k^{\ast }(x_k)=\underset{u_k\in U(x_k)}{\min }{Q}_k^{\ast }(x_k,u_k)$$
and the DP algorithm can be written in terms of Q-factors as
$$Q_k^{\ast }(x_k,u_k)=\mathbb{E}\{g_k(x_k,u_k,w_k)+\underset{u_{k+1}}{\min }{Q}_{k+1}^{\ast }(f_k(x_k,u_k,w_k),w_{k+1})\}$$

Source From

RL & OC