Improving the condition number of estimated covariance matrices
References
- Axelsson, O. 1996. Iterative Solution Methods. Cambridge: Cambridge University Press.
- Bannister, R. N. 2017. A review of operational methods of variational and ensemble variational data assimilation. Q. J. R. Meteorol. Soc. 143, 607–633. doi:10.1002/qj.2982
- Bickel, P. J. and Levina, E. 2008. Covariance regularization by thresholding. Ann. Stat. 36, 2577–2604. doi:10.1214/08-AOS600
- Bormann, N., Bonavita, M., Dragani, R., Eresmaa, R., Matricardi, M. and co-authors. 2015. Enhancing the Impact of IASI Observations through an Updated Observation Error Covariance Matrix. ECMWF Technical Memoranda. ECMWF, Reading.
- Bormann, N., Bonavita, M., Dragani, R., Eresmaa, R., Matricardi, M. and co-authors. 2016. Enhancing the impact of IASI observations through an updated observation error covariance matrix. Q. J. R. Meteorol. Soc. 142, 1767–1780. doi:10.1002/qj.2774
- Budd, C. J., Freitag, M. A. and Nichols, N. K. 2011. Regularization techniques for ill-posed inverse problems in data assimilation. Comput. Fluids 46, 168–173. doi:10.1016/j.compfluid.2010.10.002
- Campbell, W. F., Satterfield, E. A., Ruston, B. and Baker, N. L. 2017. Accounting for correlated observation error in a dual-formulation 4D variational data assimilation system. Mon. Wea. Rev. 145, 1019–1032. doi:10.1175/MWR-D-16-0240.1
- Cheney, W. 2005. Numerical Mathematics and Computing. 5th ed., International ed. Brooks/Cole Cengage Learning, Pacific Grove, CA.
- Cordoba, M., Dance, S. L., Kelly, G. A., Nichols, N. K. and Waller, J. A. 2016. Diagnosing atmospheric motion vector observation errors for an operational high resolution data assimilation system. Q. J. R. Meteorol. Soc. 143, 333–341.
- Daley, R. 1991. Atmospheric Data Analysis. Cambridge University Press, New York.
- Desroziers, G., Berre, L., Chapnik, B. and Poli, P. 2005. Diagnosis of observation, background and analysis-error statistics in observation space. Q. J. R. Meteorol. Soc. 131, 3385–3396. doi:10.1256/qj.05.108
- Fan, K. 1959. Convex sets and their applications. Lecture notes. Applied Mathematics Division, Argonne National Laboratory, Lemont, IL.
- Fowler, A. M., Dance, S. L. and Waller, J. A. 2018. On the interaction of observation and prior error correlations in data assimilation. Q. J. R. Meteorol. Soc. 144, 48–62. doi:10.1002/qj.3183
- Ghil, M. 1989. Meteorological data assimilation for oceanographers. Part I: description and theoretical framework. Dyn. Atmos. Oceans. 13, 171–218. doi:10.1016/0377-0265(89)90040-7
- Ghil, M. and Malanotte-Rizzoli, P. 1991. Data assimilation in meteorology and oceanography. Adv. Geophys. 33, 141–266. doi:10.1016/S0065-2687(08)60442-2
- Gill, P. E., Murray, W. and Wright, M. H. 1986. Practical Optimization. Academic Press, Amsterdam/London.
- Golub, G. H. and Van Loan, C. F. 1996. Matrix Computations. 3rd ed. The John Hopkins University Press, Baltimore, MD.
- Gray, R. M. 2006. Toeplitz and circulant matrices: a review. Found. Trends Commun. Inform. Theory 2, 155–239.
- Guillet, O., Weaver, A. T., Vasseur, X., Michel, Y., Gratton, S. and co-authors. 2019. Modelling spatially correlated observation errors in variational data assimilation using a diffusion operator on an unstructured mesh. Q. J. R. Meteorol. Soc. 145, 1947–1967. doi:10.1002/qj.3537
- Haben, S. A. 2011. Conditioning and preconditioning of the minimisation problem in variational data assimilation. PhD thesis, Department of Mathematics and Statistics, University of Reading, Reading.
- Haben, S. A., Lawless, A. S. and Nichols, N. K. 2011a. Conditioning and preconditioning of the variational data assimilation problem. Comput. Fluids 46, 252–256. doi:10.1016/j.compfluid.2010.11.025
- Haben, S. A., Lawless, A. S. and Nichols, N. K. 2011b. Conditioning of incremental variational data assimilation, with application to the Met Office system. Tellus A 64, 782–792.
- Hamill, T. M., Whitaker, J. S. and Snyder, C. 2001. Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev. 129, 2776–2790. doi:10.1175/1520-0493(2001)129<;2776:DDFOBE>2.0.CO;2
- Hansen, P. C. 1998. Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion. Society for Industrial and Applied Mathematics, Philadelphia, PA.
- Healy, S. B. and White, A. A. 2005. Use of discrete Fourier transforms in the 1D-Var retrieval problem. Q. J. R. Meteorol. Soc. 131, 63–72. doi:10.1256/qj.03.193
- Heilliette, S. and Garand, L. 2015.
Impact of accounting for inter-channel error covariances at the Canadian meteorological centre . In: Proceedings of the 2015 EUMETSAT Meteorological Satellite Conference, Toulouse, France, p. 8. Online at: https://www.eumetsat.int/website/home/News/ConferencesandEvents/PreviousEvents/DAT_2305526.html - Higham, N. J. 2002. Computing the nearest correlation matrix – a problem from finance. IMA J. Numer. Anal. 22, 329–343. doi:10.1093/imanum/22.3.329
- Higham, N. J., Strabić, N. and Šego, V. 2016. Restoring definiteness via shrinking, with an application to correlation matrices with a fixed block. SIAM Rev. 58, 245–263. doi:10.1137/140996112
- Hoerl, A. E. and Kennard, R. W. 1970. Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12, 55–67. doi:10.1080/00401706.1970.10488634
- Horn, R. A. and Johnson, C. R. 1991. Topics in Matrix Analysis. Cambridge University Press, Cambridge.
- Janjić, T., Bormann, N., Bocquet, M., Carton, J. A., Cohn, S. E. and co-authors. 2018. On the representation error in data assimilation. Q. J. R. Meteorol. Soc. 144, 1257–1278. doi:10.1002/qj.3130
- Johnson, C., Hoskins, B. J. and Nichols, N. K. 2005. A singular vector perspective of 4D-Var: filtering and interpolation. Q. J. R. Meteorol. Soc. 131, 1–19. doi:10.1256/qj.03.231
- Ledoit, O. and Wolf, M. 2004. A well-conditioned estimator for large-dimensional covariance matrix. J. Multivar. Anal. 88, 365–411. doi:10.1016/S0047-259X(03)00096-4
- Liu, Z.-Q. and Rabier, F. 2003. The potential of high-density observations for numerical weather prediction: a study with simulated observations. Q. J. R. Meteorol. Soc. 129, 3013–3035. doi:10.1256/qj.02.170
- McNally, A. P., Watts, P. D., Smith, J. A., Engelen, R., Kelly, G. A. and co-authors. 2006. The assimilation of AIRS radiance data at ECMWF. Q. J. R. Meteorol. Soc. 132, 935–957. doi:10.1256/qj.04.171
- Ménétrier, B., Montmerle, T., Michel, Y. and Berre, L. 2015. Linear filtering of sample covariances for ensemble-based data assimilation. Part I: optimality criteria and application to variance filtering and covariance localization. Mon. Wea. Rev. 143, 1622–1643. doi:10.1175/MWR-D-14-00157.1
- Michel, Y. 2018. Revisiting Fisher’s approach to the handling of horizontal spatial correlations of the observation errors in a variational framework. Q. J. R. Meteorol. Soc. 144, 2011–2025. doi:10.1002/qj.3249
- Migliorini, S. 2012. On the equivalence between radiance and retrieval assimilation. Mon. Wea. Rev. 140, 258–265. doi:10.1175/MWR-D-10-05047.1
- Moodey, A. J. F., Lawless, A. S., Potthast, R. W. E. and van Leeuwen, P. J. 2013. Nonlinear error dynamics for cycled data assimilation methods. Inverse Probl. 29, 025002. doi:10.1088/0266-5611/29/2/025002
- Nakamura, G. and Potthast, R. 2015. Inverse Modeling. Bristol: IOP Publishing, pp. 2053–2563.
- Pourahmadi, M. 2013. High-Dimensional Covariance Estimation. John Wiley & Sons, Hoboken, NJ.
- Prates, C., Migliorini, S., Stewart, L. and Eyre, J. 2016. Assimilation of transformed retrievals obtained from clear-sky IASI measurements. Q. J. R. Meteorol. Soc. 142, 1697–1712. doi:10.1002/qj.2764
- Qi, H. and Sun, D. 2010. Correlation stress testing for value-at-risk: an unconstrained convex optimization approach. Comput. Optim. Appl. 45, 427–462. doi:10.1007/s10589-008-9231-4
- Schiff, S. J. 2011. Neural Control Engineering: The Emerging Intersection between Control Theory and Neuroscience. Computational Neuroscience. MIT Press, Cambridge, MA.
- Simonin, D., Waller, J. A., Ballard, S. P., Dance, S. L. and Nichols, N. K. 2019. A pragmatic strategy for implementing spatially correlated observation errors in an operational system: demonstration for Doppler radar winds. Q. J. R. Meteorol. Soc. 145, 2772–2790. doi:10.1002/qj.3592
- Skøien, J. O. and Blöschl, G. 2006. Sampling scale effects in random fields and implications for environmental monitoring. Environ. Monit. Assess. 114, 521–552. doi:10.1007/s10661-006-4939-z
- Smith, P. J., Lawless, A. S. and Nichols, N. K. 2018. Treating sample covariances for use in strongly coupled atmosphere-ocean data assimilation. Geophys. Res. Lett. 45, 445–454. doi:10.1002/2017GL075534
- Stewart, L. M. 2010. Correlated observation errors in data assimilation. PhD thesis, University of Reading, Reading.
- Stewart, L. M., Cameron, J., Dance, S. L., English, S., Eyre, J. and Nichols, N. K. 2009. Observation Error Correlations in IASI Radiance Data. Mathematics Report Series, Report number 1/2009, University of Reading, Reading.
- Stewart, L. M., Dance, S. L. and Nichols, N. K. 2008. Correlated observation errors in data assimilation. Int. J. Numer. Meth. Fluids 56, 1521–1527. doi:10.1002/fld.1636
- Stewart, L. M., Dance, S. L. and Nichols, N. K. 2013. Data assimilation with correlated observation errors: experiments with a 1-D shallow water model. Tellus A 65, 19546 (14 pp.), doi:10.3402/tellusa.v65i0.19546
- Stewart, L. M., Dance, S. L., Nichols, N. K., Eyre, J. R. and Cameron, J. 2014. Estimating interchannel observation-error correlations of IASI radiance data in the Met Office system. Q. J. R. Meteorol. Soc. 140, 1236–1244. doi:10.1002/qj.2211
- Tabeart, J. M., Dance, S. L., Haben, S. A., Lawless, A. S., Nichols, N. K. and Waller, J. A. 2018. The conditioning of least squares problems in variational data assimilation. Numer. Linear Algebra Appl. 25, e2165. doi:10.1002/nla.2165
- Tabeart, J. M., Dance, S. L., Lawless, A. S., Migliorini, S., Nichols, N. K., and co-authors.2019. The impact of reconditioning of the correlated observation error covariance matrix on the Met Office system. Online at: http://arxiv.org/abs/1908.04071. Submitted.
- Tanaka, M. and Nakata, K. 2014. Positive definite matrix approximation with condition number constraint. Optim. Lett. 8, 939–947. doi:10.1007/s11590-013-0632-7
- Thiebaux, H. J. 1976. Anisotropic correlation functions for objective analysis. Mon. Wea. Rev. 104, 994–1002. doi:10.1175/1520-0493(1976)104<;0994:ACFFOA>2.0.CO;2
- Trefethen, L. N. and Bau, D. 1997. Numerical Linear Algebra. Society for Industrial and Applied Mathematics, Philadelphia, PA.
- Waller, J. A., Ballard, S. P., Dance, S. L., Kelly, G., Nichols, N. K. and Simonin, D. 2016. Diagnosing horizontal and inter-channel observation error correlations for SEVIRI observations using observation-minus-background and observation-minus-analysis statistics. Remote Sens. 8, 581. doi:10.3390/rs8070581
- Waller, J. A., Dance, S. L. and Nichols, N. K. 2016. Theoretical insight into diagnosing observation error correlations using observation-minus-background and observation-minus-analysis statistics. Q. J. R. Meteorol. Soc. 142, 418–431. doi:10.1002/qj.2661
- Waller, J. A., Simonin, D., Dance, S. L., Nichols, N. K. and Ballard, S. P. 2016. Diagnosing observation error correlations for Doppler radar radial winds in the Met Office UKV model using observation-minus-background and observation-minus-analysis statistics. Mon. Wea. Rev. 144, 3533–3551. doi:10.1175/MWR-D-15-0340.1
- Weston, P. 2011. Progress towards the Implementation of Correlated Observation Errors in 4D-Var. Forecasting Research Technical Report 560, Met Office, Exeter, UK.
- Weston, P. P., Bell, W. and Eyre, J. R. 2014. Accounting for correlated error in the assimilation of high-resolution sounder data. Q. J. R. Meteorol. Soc. 140, 2420–2429. doi:10.1002/qj.2306
- Yaglom, A. M. 1987. Correlation Theory of Stationary and Related Random Functions, Volume I: Basic Results. Springer-Verlag, New York.
DOI: https://doi.org/10.1080/16000870.2019.1696646 | Journal eISSN: 3035-9554
Language: English
Page range: 1696646 - 1696646
Published on: Jan 1, 2020
Published by: Stockholm University Press
In partnership with: Paradigm Publishing Services
Keywords:
© 2020 Jemima M. Tabeart, Sarah L. Dance, Amos S. Lawless, Nancy K. Nichols, Joanne A. Waller, published by Stockholm University Press
This work is licensed under the Creative Commons Attribution 4.0 License.