Accounting for observation uncertainty and bias due to unresolved scales with the Schmidt-Kalman filter

Zackary Bell; Sarah L. Dance; Joanne A. Waller

doi:10.1080/16000870.2020.1831830

Accounting for observation uncertainty and bias due to unresolved scales with the Schmidt-Kalman filter

Tellus

Volume 72 (2020): Issue 1

By: Zackary Bell, Sarah L. Dance and Joanne A. Waller

Open Access

|Jan 2020

Figures & Tables

Table 1.

The filter matrices and vectors for the Optimal Kalman filter (OKF), reduced-state Kalman filter (RKF) and Schmidt Kalman filter (SKF). The equations for the three filters are obtained through substituting these terms into (3.1) - (3.4). As discussed in section 3.4, we note that while the SKF uses $H^{l}$ in the innovation only, the complete observation operator is used in the calculation of $K^{l}$ and $D .$

OKFRKFSKFState: x

(\begin{matrix} x^{l} \\ x^{s} \end{matrix}) \in R^{N_{t}}

x^{l} \in R^{N_{l}}

x^{l} \in R^{N_{l}}

State error covariance: P

(\begin{matrix} P^{l l} & P^{l s} \\ P^{s l} & P^{s s} \end{matrix}) \in R^{N_{t} \times N_{t}}

P^{l l} \in R^{N_{l} \times N_{l}}

(\begin{matrix} P^{l l} & P^{l s} \\ P^{s l} & C^{s} \end{matrix}) \in R^{N_{t} \times N_{t}}

Observation operator: H

(\begin{matrix} H^{l} & H^{s} \end{matrix}) \in R^{p \times N_{t}}

H^{l} \in R^{p \times N_{l}}

(\begin{matrix} H^{l} & H^{s} \end{matrix}) \in R^{p \times N_{t}}

Observation Error Covariance: R

R^{I} + R^{G}

R^{I} + R^{H}

R^{I} + R^{G}

Innovation:

d^{o, f}

y - (\begin{matrix} H^{l} & H^{s} \end{matrix}) (\begin{matrix} x^{l, f} \\ x^{s, f} \end{matrix})

y - H^{l} x^{l, f}

y - H^{l} x^{l, f}

Innovation covariance: D

(\begin{matrix} H^{l} & H^{s} \end{matrix}) (\begin{matrix} P^{l l, f} & P^{l s, f} \\ P^{s l, f} & P^{s s, f} \end{matrix}) (\begin{matrix} {(H^{l})}^{T} \\ {(H^{s})}^{T} \end{matrix}) + R

H^{l} P^{l l, f} {(H^{l})}^{T} + R

(\begin{matrix} H^{l} & H^{s} \end{matrix}) (\begin{matrix} P^{l l, f} & P^{l s, f} \\ P^{s l, f} & C^{s} \end{matrix}) (\begin{matrix} {(H^{l})}^{T} \\ {(H^{s})}^{T} \end{matrix}) + R

Kalman gain: K

(\begin{matrix} K^{l} \\ K^{s} \end{matrix}) = (\begin{matrix} P^{l l, f} & P^{l s, f} \\ P^{s l, f} & P^{s s, f} \end{matrix}) (\begin{matrix} {(H^{l})}^{T} \\ {(H^{s})}^{T} \end{matrix}) D^{- 1}

K^{l} = (P^{l l, f} {(H^{l})}^{T}) D^{- 1}

\begin{matrix} State update \\ K^{l} = (P^{l l, f} {(H^{l})}^{T} + P^{l s, f} {(H^{s})}^{T}) D^{- 1} \\ Covariance update \\ (\begin{matrix} K^{l} \\ K^{s} \end{matrix}) = (\begin{matrix} (P^{l l, f} {(H^{l})}^{T} + P^{l s, f} {(H^{s})}^{T}) D^{- 1} \\ 0_{N_{s} \times p} \end{matrix}) \end{matrix}

Forecast model: M

(\begin{matrix} M^{l} & 0_{N_{l} \times N_{s}} \\ M^{s l} & M^{s} \end{matrix}) \in R^{N_{t} \times N_{t}}

M^{l} \in R^{N_{l} \times N_{l}}

\begin{matrix} State update \\ M^{l} \in R^{N_{l} \times N_{l}} \\ Covariance update \\ (\begin{matrix} M^{l} & 0_{N_{l} \times N_{s}} \\ M^{s l} & M^{s} \end{matrix}) \in R^{N_{t} \times N_{t}} \end{matrix}

Model Error Covariance: Q

(\begin{matrix} Q^{l l} & Q^{l s} \\ Q^{s l} & Q^{s s} \end{matrix}) \in R^{N_{t} \times N_{t}}

Q^{l l} \in R^{N_{l} \times N_{l}}

(\begin{matrix} Q^{l l} & Q^{l s} \\ Q^{s l} & Q^{s s} \end{matrix}) \in R^{N_{t} \times N_{t}}

Table 2.

Matrices and vectors used in the true error calculations for Case 1 and 2 described in sections 4.1 and 4.2. The tildes indicate true error covariances. Case 1 corresponds to analysing all scales and includes the OKF. Case 2 corresponds to analysing the large scales only and includes the RKF and SKF. The true analysis error equation, analysis error covariance and forecast error covariance for each case are obtained by substituting the corresponding components into equations (4.1), (4.2) and (4.3) respectively.

Case 1 (OKF)Case 2 (SKF and RKF)Analysis Errors:

e^{a}

(\begin{matrix} e^{l, a} \\ e^{s, a} \end{matrix})

(\begin{matrix} e^{l, a} \\ e^{s, a} \end{matrix})

Model Errors: η

(\begin{matrix} η^{l} \\ η^{s} \end{matrix})

(\begin{matrix} η^{l} \\ η^{s} \end{matrix})

Observation Errors:

e^{o}

ϵ + γ^{l} + γ^{s}

ϵ + γ^{l} + H^{s, t} x^{s, t}

Kalman Gain: K

(\begin{matrix} K^{l} \\ K^{s} \end{matrix})

(\begin{matrix} K^{l} \\ 0_{N_{s} \times p} \end{matrix})

Observation Operator: H

(\begin{matrix} H^{l} & H^{s} \end{matrix})

(\begin{matrix} H^{l} & 0_{p \times N_{s}} \end{matrix})

Model: M

(\begin{matrix} M^{l} & 0_{N_{l} \times N_{s}} \\ M^{s l} & M^{s} \end{matrix})

(\begin{matrix} M^{l} & 0_{N_{l} \times N_{s}} \\ M^{s l} & M^{s} \end{matrix})

State Error Covariance:

\tilde{P}

(\begin{matrix} {\tilde{P}}^{l l} & {\tilde{P}}^{l s} \\ {\tilde{P}}^{s l} & {\tilde{P}}^{s s} \end{matrix})

(\begin{matrix} {\tilde{P}}^{l l} & {\tilde{P}}^{l s} \\ {\tilde{P}}^{s l} & {\tilde{P}}^{s s} \end{matrix})

Observation Error Covariance:

\tilde{R}

{\tilde{R}}^{I} + {\tilde{R}}^{G}

{\tilde{R}}^{I} + {\tilde{R}}^{H}

Model Error Covariance:

\tilde{Q}

(\begin{matrix} {\tilde{Q}}^{l l} & {\tilde{Q}}^{l s} \\ {\tilde{Q}}^{s l} & {\tilde{Q}}^{s s} \end{matrix})

(\begin{matrix} {\tilde{Q}}^{l l} & {\tilde{Q}}^{l s} \\ {\tilde{Q}}^{s l} & {\tilde{Q}}^{s s} \end{matrix})

Contour plot of the values of $C^{s}$ that give the minimum true large-scale analysis error variance at the final assimilated observation for the SKF for different ratios of $Q^{s}$ and $R^{I} .$

(a): The optimal $C^{s}$ values when $R^{I} = 0.1$ (dashed line) and $R^{I} = 0.5$ (dotted line) as functions of $Q^{s} .$ The grey region shows all points between S (lower edge) and 2S (upper edge) for different values of $Q^{s} .$ (b): The effect of changing $C^{s}$ on the final true large-scale analysis error covariance for the SKF (solid line) when $Q^{s} = 0.35$ and $R^{I} = 0.1 .$ Also shown are the OKF and RKF true large-scale analysis error variance (lower dashed line and upper dotted line respectively). The grey region shows all points between $C^{s} = S$ (left edge) and $C^{s} = 2 S$ (right edge). The optimal value of $C^{s}$ (i.e. the minimum of the solid line) lies in this region.

(a): Comparison of the RKF to the OKF in terms of relative error percentage given by equation (6.1). (b): Comparison of the SKF with optimal $C^{s}$ to the OKF at the final time-step in terms of relative error percentage.

(a): Difference between the perceived and true analysis error variances at the final time-step for the SKF with optimal $C^{s} .$ (b): Difference between the perceived and true analysis error variances at the final time-step for the RKF.

Table 3.

The filter matrices and vectors for the SKFbc and RKFbc. The equations for the the two filters are obtained through substituting these terms into (3.1)–(3.4).

RKFbcSKFbcState: x

(\begin{matrix} x^{l} \\ x^{β} \end{matrix}) \in R^{N_{t}}

(\begin{matrix} x^{l} \\ x^{β} \end{matrix}) \in R^{N_{t}}

State error covariance: P

(\begin{matrix} P^{l l} & P^{l β} \\ P^{β l} & P^{β β} \end{matrix}) \in R^{N_{t} \times N_{t}}

(\begin{matrix} P^{l l} & P^{l β} & P^{l δ} \\ P^{β l} & P^{β β} & P^{β δ} \\ P^{δ l} & P^{δ β} & C^{δ} \end{matrix}) \in R^{N_{a} \times N_{a}}

Observation operator: H

(\begin{matrix} H^{l} & H^{β} \end{matrix}) \in R^{p \times N_{t}}

(\begin{matrix} H^{l} & H^{β} & H^{δ} \end{matrix}) \in R^{p \times N_{a}}

Observation error covariance: R

R^{I} + R^{H} \in R^{p \times p}

R^{I} + R^{G} \in R^{p \times p}

Innovation:

d^{o, f}

y - (\begin{matrix} H^{l} & H^{β} \end{matrix}) (\begin{matrix} x^{l, f} \\ x^{β, f} \end{matrix})

y - (\begin{matrix} H^{l} & H^{β} \end{matrix}) (\begin{matrix} x^{l, f} \\ x^{β, f} \end{matrix})

Innovation covariance: D

(\begin{matrix} H^{l} & H^{β} \end{matrix}) (\begin{matrix} P^{l l, f} & P^{l β, f} \\ P^{β l, f} & P^{β β, f} \end{matrix}) (\begin{matrix} {(H^{l})}^{T} \\ {(H^{β})}^{T} \end{matrix}) + R

(\begin{matrix} H^{l} & H^{β} & H^{δ} \end{matrix}) (\begin{matrix} P^{l l, f} & P^{l β, f} & P^{l δ, f} \\ P^{β l, f} & P^{β β, f} & P^{β δ, f} \\ P^{δ l, f} & P^{δ β, f} & C^{δ} \end{matrix}) (\begin{matrix} {(H^{l})}^{T} \\ {(H^{β})}^{T} \\ {(H^{δ})}^{T} \end{matrix}) + R

Kalman gain: K

(\begin{matrix} K^{l} \\ K^{β} \end{matrix}) = (\begin{matrix} P^{l l, f} & P^{l β, f} \\ P^{β l, f} & P^{β β, f} \end{matrix}) (\begin{matrix} {(H^{l})}^{T} \\ {(H^{β})}^{T} \end{matrix}) D^{- 1}

\begin{matrix} State update \\ (\begin{matrix} K^{l} \\ K^{β} \end{matrix}) = (\begin{matrix} P^{l l, f} {(H^{l})}^{T} + P^{l β, f} {(H^{β})}^{T} + P^{l δ, f} {(H^{δ})}^{T} \\ P^{β l, f} {(H^{l})}^{T} + P^{β β, f} {(H^{β})}^{T} + P^{β δ, f} {(H^{δ})}^{T} \end{matrix}) D^{- 1} \\ Covariance update \\ (\begin{matrix} K^{l} \\ K^{β} \\ K^{δ} \end{matrix}) = (\begin{matrix} P^{l l, f} {(H^{l})}^{T} + P^{l β, f} {(H^{β})}^{T} + P^{l δ, f} {(H^{δ})}^{T} \\ P^{β l, f} {(H^{l})}^{T} + P^{β β, f} {(H^{β})}^{T} + P^{β δ, f} {(H^{δ})}^{T} \\ 0_{N_{s} \times p} \end{matrix}) D^{- 1} \end{matrix}

Forecast model: M

(\begin{matrix} M^{l} & 0_{N_{l} \times N_{s}} \\ M^{s l} & M^{s} \end{matrix}) \in R^{N_{t} \times N_{t}}

\begin{matrix} State update \\ (\begin{matrix} M^{l} & 0_{N_{l} \times N_{s}} \\ M^{s l} & M^{s} \end{matrix}) \in ℝ^{N_{l} \times N_{l}} \\ Covariance update \\ (\begin{matrix} M^{l} & 0_{N_{l} \times N_{s}} & 0_{N_{l} \times N_{s}} \\ M^{s l} & M^{s} & 0_{N_{s} \times N_{s}} \\ 0_{N_{s} \times N_{l}} & 0_{N_{s} \times N_{s}} & M^{s} \end{matrix}) \in ℝ^{N_{a} \times N_{a}} \end{matrix}

Model error covariance: Q

(\begin{matrix} Q^{l l} & 0_{N_{l} \times N_{s}} \\ 0_{N_{s} \times N_{l}} & 0_{N_{s} \times N_{s}} \end{matrix}) \in R^{N_{t} \times N_{t}}

(\begin{matrix} Q^{l l} & 0_{N_{l} \times N_{s}} & Q^{l s} \\ 0_{N_{s} \times N_{l}} & 0_{N_{s} \times N_{s}} & 0_{N_{s} \times N_{s}} \\ Q^{s l} & 0_{N_{s} \times N_{s}} & Q^{s s} \end{matrix}) \in ℝ^{N_{a} \times N_{a}}

(a): The large-scale analysis for the SKFbc (square markers) and SKF (diamond markers) obtained through assimilation of biased observations to recreate the true large-scale state (grey dashed line). For this realization the large-scale analysis mean-square-error for the SKFbc is 0.29 and for the SKF is 1.53. (b): The SKFbc bias analysis estimate (square markers) and the true small-scale state (grey dashed line) for the same realization as panel (a).

The values of $C^{δ}$ which give the minimum true large-scale analysis error variance at the end of the assimilation window for the SKFbc.

(a): Comparison of the RKFbc to the OKF in terms of relative error percentage given by equation (6.1). (b): Comparison of the SKFbc with optimal $C^{δ}$ to the OKF at the final time-step in terms of relative error percentage.

References

Authors

Metrics

Articles in this issue

DOI: https://doi.org/10.1080/16000870.2020.1831830 | Journal eISSN: 3035-9554

Journal RSS Feed

Language: English

Page range: 1831830 - 1831830

Published on: Jan 1, 2020

Published by: Stockholm University Press

In partnership with: Paradigm Publishing Services

Keywords:

data assimilation,

observation uncertainty,

error due to unresolved scales,

observation bias,

Schmidt-Kalman filter

© 2020 Zackary Bell, Sarah L. Dance, Joanne A. Waller, published by Stockholm University Press
This work is licensed under the Creative Commons Attribution 4.0 License.

Volume 72 (2020): Issue 1

Accounting for observation uncertainty and bias due to unresolved scales with the Schmidt-Kalman filter

Figures & Tables

Table 1.

Table 2.

Fig. 1.

Fig. 2.

Fig. 3.

Fig. 4.

Table 3.

Fig. 5.

Fig. 6.

Fig. 7.

Paradigm

My account