Kalman filter

Conditional mean and variance of Gaussian

Let $X, Y$ be joint Gaussian, i.e. \begin{eqnarray}\left[\begin{array}{c} X \\ Y\end{array}\right] \,\sim \, {\cal{N}}\left(\left[\begin{array}{c}\mu_X \\ \mu_Y\end{array}\right],\,\left[\begin{array}{cc}\Sigma_{XX} & \Sigma_{XY} \\ \Sigma_{YX} & \Sigma_{YY}\end{array}\right]\right)\,,\end{eqnarray} what is $\mathbb{E}(X|Y)$ and $\text{Var}(X|Y)$? 

Trick: Find constant matrix $A$ such that $X+AY$ is independent with $Y$. Then we have \begin{eqnarray} \mathbb{E}(X|Y) &=& \mathbb{E}(X + AY|Y) - \mathbb{E}(AY|Y) = \mathbb{E}(X + AY) - AY\nonumber\\ &=&\mu_X + A\left(\mu_Y - Y\right)\,,\nonumber\\ \text{Var}(X|Y) &=& \text{Var}(X+AY|Y) = \text{Var}(X+AY) = \text{Cov}(X+AY, X^T + Y^T A^T)\nonumber\\ &=&\Sigma_{XX} + \Sigma_{XY} A^T + A \Sigma_{YX} + A \Sigma_{YY} A^T\,. \nonumber \end{eqnarray} Finally, from $0=\text{Cov}(X+AY, Y)=\Sigma_{XY}+A\Sigma_{YY}$, we have $A=-\Sigma_{XY}\Sigma_{YY}^{-1}$. As a result, \begin{eqnarray} \mathbb{E}(X|Y) &=& \mu_X + \Sigma_{XY}\Sigma_{YY}^{-1} \left(Y-\mu_Y\right)\,, \tag{1}\\ \text{Var}(X|Y) &=& \Sigma_{XX} - \Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}\,. \tag{2} \end{eqnarray}

Kalman filter

We consider a simple linear model with white noises:  \begin{eqnarray} x_t &=& A\, x_{t-1} + w_t\,,\tag{3}\\ y_t &=& B\, x_t + v_t\,, \tag{4} \end{eqnarray} in which $w_t \sim {\cal{N}}(0, W)$, $v_t \sim {\cal{N}}(0, V)$ and $x_0 \sim {\cal{N}}(\bar{x}_0, \Sigma)$.

Let $Y_t$ be the set of variables $(y_1, \cdots, y_t)$ and $x_t \left| Y_s \right.$ be Gaussian with mean and covariance denoted by $\hat{x}_{t|s}$ and $\Sigma_{t|s}$.

Prediction

From (3), we have $x_t \left| Y_{t-1} \right. = A\, x_{t-1} \left| Y_{t-1} \right. + w_t$ and thus \begin{eqnarray} \hat{x}_{t|t-1} &=& A\, \hat{x}_{t-1|t-1}\,,\nonumber \\ \Sigma_{t|t-1} &=& A\, \Sigma_{t|t-1}\, A^T + W\,.\nonumber \end{eqnarray}

Filtering

From (4), $y_t \left| Y_{t-1} \right. = B\, x_t \left| Y_{t-1} \right. + v_t$. As a result, $x_t \left| Y_{t-1} \right.$ and $y_t \left| Y_{t-1} \right.$ are joint Gaussian \begin{eqnarray} \left[\begin{array}{c} x_t \left| Y_{t-1} \right. \\ y_t \left| Y_{t-1} \right.\end{array}\right] \,\sim \, {\cal{N}}\left(\left[\begin{array}{c}\hat{x}_{t|t-1} \\ B\hat{x}_{t|t-1}\end{array}\right],\, \left[\begin{array}{cc}\Sigma_{t|t-1} & \Sigma_{t|t-1} B^T \\ B\Sigma_{t|t-1} & B\Sigma_{t|t-1}B^T + V\end{array}\right]\right)\,.\nonumber \end{eqnarray} Note that $x_t \left| Y_t \right. = \left(x_t \left| Y_{t-1} \right.\right) \left| \left(y_t \left| Y_{t-1}\right. \right) \right.$. Using (1) and (2), we have \begin{eqnarray} \hat{x}_{t|t} &=& \hat{x}_{t|t-1} + \Sigma_{t|t-1} B^T \left(B\Sigma_{t|t-1}B^T + V\right)^{-1}\left( y_t - B\hat{x}_{t|t-1}\right)\,, \nonumber\\ \Sigma_{t|t} &=& \Sigma_{t|t-1} - \Sigma_{t|t-1} B^T \left(B\Sigma_{t|t-1}B^T + V\right)^{-1}B\Sigma_{t|t-1}\,.\nonumber \end{eqnarray} This gives our updated estimate of $x_t$ based on the measurement $y_t$ becoming available.

Smoothing

From (1), $x_{t+1} \left| Y_{t} \right. = A\, x_{t} \left| Y_{t} \right. + w_{t+1}$. As a result, $x_t \left| Y_{t} \right.$ and $x_{t+1} \left| Y_t \right.$ are joint Gaussian \begin{eqnarray} \left[\begin{array}{c} x_t \left| Y_{t} \right. \\ x_{t+1} \left| Y_{t} \right.\end{array}\right] \,\sim \, {\cal{N}}\left(\left[\begin{array}{c}\hat{x}_{t|t} \\ \hat{x}_{t+1|t}\end{array}\right],\, \left[\begin{array}{cc}\Sigma_{t|t} & \Sigma_{t|t} A^T \\ A\Sigma_{t|t} & \Sigma_{t+1|t} \end{array}\right]\right)\,.\nonumber \end{eqnarray} As a result, $\left(x_t \left| Y_{t}\right.\right)\left| \left(x_{t+1} \left| Y_{t}\right.\right)\right.$ is Gaussian with mean and covariance \begin{eqnarray} \mathbb{E}\left[\left(x_t \left| Y_{t}\right.\right)\left| \left(x_{t+1} \left| Y_{t}\right.\right) = z \right.\right] &=& \hat{x}_{t|t} + \Sigma_{t|t} A^T \Sigma_{t+1|t}^{-1} \left( z - \hat{x}_{t+1|t}\right)\,,\nonumber\\ \text{Var}\left[\left(x_t \left| Y_{t}\right.\right)\left| \left(x_{t+1} \left| Y_{t}\right.\right) = z \right.\right] &=& \Sigma_{t|t} - \Sigma_{t|t} A^T \Sigma_{t+1|t}^{-1} A \Sigma_{t|t} \,.\nonumber \end{eqnarray} Using the law of total expectation/variance, as well as the Markov property $\left(x_t \left| Y_{T}\right.\right)\left| \left(x_{t+1} \left| Y_{T}\right.\right)\right.= \left(x_t \left| Y_{t}\right.\right)\left| \left(x_{t+1} \left| Y_{T}\right.\right)\right.$, we have \begin{eqnarray} \hat{x}_{t|T} &=& \mathbb{E}_{x_{t+1} \left| Y_{T}\right.}\left[\left(x_t \left| Y_{T}\right.\right)\left| \left(x_{t+1} \left| Y_{t}\right.\right) = \left(x_{t+1} \left| Y_{T}\right.\right) \right.\right]\nonumber\\ &=&\mathbb{E}_{x_{t+1} \left| Y_{T}\right.}\left[\left(x_t \left| Y_{t}\right.\right)\left| \left(x_{t+1} \left| Y_{t}\right.\right) = \left(x_{t+1} \left| Y_{T}\right.\right) \right.\right]\nonumber\\ &=&\hat{x}_{t|t} + \Sigma_{t|t} A^T \Sigma_{t+1|t}^{-1} \left( \hat{x}_{t+1|T} - \hat{x}_{t+1|t}\right)\,,\nonumber\\ \hat{\Sigma}_{t|T} &=& \mathbb{E}_{x_{t+1} \left| Y_{T}\right.}\left\{\text{Var}\left[\left(x_t \left| Y_{t}\right.\right)\left| \left(x_{t+1} \left| Y_{t}\right.\right) = \left(x_{t+1} \left| Y_{T}\right.\right) \right.\right]\right\}+\nonumber\\ &&\text{Var}_{x_{t+1} \left| Y_{T}\right.}\left\{\mathbb{E}\left[\left(x_t \left| Y_{t}\right.\right)\left| \left(x_{t+1} \left| Y_{t}\right.\right) = \left(x_{t+1} \left| Y_{T}\right.\right) \right.\right]\right\}\nonumber\\ &=& \left(\Sigma_{t|t} - \Sigma_{t|t} A^T \Sigma_{t+1|t}^{-1} A \Sigma_{t|t}\right)+ \Sigma_{t|t} A^T \Sigma_{t+1|t}^{-1} \Sigma_{t+1|T}\Sigma_{t+1|t}^{-1} A\Sigma_{t|t}\nonumber\\ &=&\Sigma_{t|t} + \Sigma_{t|t} A^T \Sigma_{t+1|t}^{-1} \left(\Sigma_{t+1|T}-\Sigma_{t+1|t}\right)\Sigma_{t+1|t}^{-1} A\Sigma_{t|t}\,.\nonumber \end{eqnarray}