Robustness of Observing System Simulation Experiments

2.1 Nature Run

The Nature Run (NR) is a long, free model run that acts as the ‘truth’ in the OSSE in place of the real atmosphere, and is also used as the basis for generating simulated observations. Ideally, the Nature Run should have high spatiotemporal resolution and the most realistic parameterizations available. The Nature Run must be able to capture both the phenomena of interest for the experiments, and also all the fields necessary for simulation of the global observing network.

A two-year long forecast of the GEOS model at 7 km horizontal resolution with 72 vertical levels and 30-minute output is used as the Nature Run for both the Previous and Updated OSSE frameworks. This Nature Run is generally referred to as the “G5NR”. The version of GEOS used to generate the NR dates approximately to 2014 and is similar in nature to the model version used for MERRA-2 (Molod et al., 2015). The G5NR has been extensively validated as documented in Gelaro et al. (2014).

2.2 Forecast Model

The NWP model used for the experiments is the version of GEOS (Rienecker et al., 2008) that was quasi-operational in 2022–2023, run with 25 km horizontal resolution (C360) and 72 vertical levels, and will be referred to as the Updated framework. This model differs from the version of GEOS used for the Nature Run in terms of the resolution, the advection schemes for thermodynamic and momentum variables, higher order divergence damping schemes, the land model, the convective and radiation schemes, and the boundary layer parametrizations. The model used in the Previous version of the OSSE framework (El Akkraoui et al., 2023) was the GEOS version that was quasi-operational in 2018, and that had fewer differences from the NR model, with very similar radiation and boundary layer schemes. It is expected that the Updated model version will have better simulation of model error in the OSSE framework as a result of more substantial differences between the experiment model and the NR model versions.

The data assimilation system used in the Updated version is the hybrid 4-dimensional ensemble variational (4DEnVar) GSI. The major differences between this version of GSI and that used in the Previous OSSE framework are the all-sky treatment of GPM/GMI, MHS, and AMSR2 observations and an update to the incremental analysis update (IAU) scheme.

2.3 Synthetic Observations

Simulated observations are generated for existing real-world data types that were used quasi-operationally during a corresponding timeframe. For the Updated OSSE, this timeframe is summer 2020, while the Previous framework was based on the 2015 global observing network. A comparison of the data types used in the Previous and Updated systems is shown in Table 1. The goal of simulating observations is to represent what the observations would have been if the real atmosphere were replaced with the NR during that time period, including both the spatiotemporal distribution of the observations and the statistical quality of the data as used by the data assimilation system. Ideally, a randomly selected subset of the simulated observations would not be easily distinguished from real observations.

For many observing system types, the spatiotemporal location of observations can be based directly on the corresponding locations of real data. For example, surface observations, GNSS-RO, or satellite orbits may be based on the locations of real data. These observations may be simulated by interpolating the NR fields to the location of real observations, and possibly the additional use of an observation operator, e.g. a radiative transfer model, to convert the model fields to the variables measured by the instrument. A thorough description of the methodology used in the GMAO OSSE framework is given by Errico et al. (2017), with updates to the simulation of Atmospheric Motion Vectors (AMVs) discussed in Errico et al. (2020). GNSS-RO observations are simulated using the Radio Occultation Processing Package two dimensional operator (Culverwell et al., 2015). Rawinsondes and dropsondes are launched at the same time and location as real sonde releases, but are advected with the NR wind field. Aircraft observation locations, however, do not account for wind-routing of flight tracks.

Radiance observations are generated using the Community Radiative Transfer Model (CRTM; Han et al. (2006); Ding et al. (2011)) version 2.2.3 for both the Previous and Updated OSSE frameworks. While geolocation of radiance observations are taken from real observations, the NR cloud field is used to determine observation locations (Errico et al., 2013) for clear-sky radiances. The radiance types treated as all-sky observations (GMI, MHS, and AMSR2) used the NR fields of liquid, frozen, and vapor moisture and precipitation to produce cloud-affected radiances. An example of a simulated GMI channel compared with real GMI is shown in Figure 1, illustrating the close match of the distribution of observation innovations (O-B), with the simulated observations reflecting the locations of NR moisture fields. Several types of radiance observations suffered significant outages during 2020. For any radiance type that experienced an outage of more than one cycle time, the locations of 2021 observations for that data type on the same day of the year were substituted for simulation of that instrument.

Figure 1

Similarly, AMVs are based on the location of cloud and water vapor features in the NR modulated by the footprint of the corresponding satellites (Errico et al., 2020). One exception to this is the simulation of Himawari AMVs in the 2020 dataset, where the distribution of simulated observations using the NR fields was not representative of the grid-like characteristics of real observations, so the corresponding real observation locations were used for simulation.

2.3.1 Calibration

Simulated observation errors are added to most simulated data types following the methodology described in Errico et al. (2013) and Errico et al. (2017). Uncorrelated random errors are added to all data types, and random correlated errors are added to select data types as noted in Table 1. The magnitude of the simulated errors and the characteristics of the error correlations are adjusted to match the statistics of real observations (Privé et al., 2021).

The calibration process involves adjusting the statistical thresholds used to determine cloud effects for the simulated observations to match the counts of real observations, and also tuning the simulated observation errors until the variance of the observation innovations matches that of real data. This process is performed for a weeklong period at the end of June prior to the start of the experiment timeframe on 1 July. The short period is rerun numerous times while making adjustments until the calibration is satisfactory. This period also allows for the spinup of the radiance bias correction, with the iterative process over the same short period adding up to several months of adjustment for the bias correction coefficients.

Figure 2 compares the standard deviation of observation innovations for several real and simulated radiance instruments for the month of July. The OSSE statistics for simulated observations are relatively close to real statistics for most channels. The tuning of the error coefficients and cloud probability functions is expected to be seasonally dependent. Figure 3 compares the calibration of IASI Metop-A observation innovations for July, August, and September. Some deviation of the simulated observations from real is noted in the September timeframe, but the difference is not sufficient to warrant recalibration. A short test of the October timeframe (not shown) indicates that the calibration does not hold well beyond the end of September, possibly due to seasonal changes in the climatology of the G5NR.

Figure 2

Figure 3

2.4 OSSE Control

The OSSE Control run ingests only those data types available in the quasi-operational real system and acts as a basis of comparison for experiments in which new observing platforms are tested. The Control run is also used to validate the performance of the OSSE framework, employing a corresponding Real case with the same model and DAS versions run with real data for the time period used as a basis for the simulated observations (2015 for the Previous OSSE and 2020 for the Updated OSSE). The Updated framework Control run was expanded to the period of July-September in the second year of the G5NR. This is an extension of the Previous framework period of July–August. September was included in the Updated framework in order to capture the Atlantic basin tropical cyclone season in support of several proposed missions.

Initialization of the OSSE framework began at the start of June, with initial conditions taken from a pre-existing real world run. The GEOS/GSI was run in 3D variational mode for several days while ingesting a global network of highly idealized simulated rawinsonde-type observations arranged on a global grid in order to pull the model state closer to the NR state. The system was then switched to the standard set of simulated observations that mimic the real world global observing network, and the 32-member ensemble was spun up for a period of approximately three weeks.

After completing calibration of the simulated observations during the late June period, observations were produced for this entire time period. The OSSE Control was initialized from the end of the calibration period, and an ensemble run was performed for the entire period of interest. In addition to cycling the DAS, 10-day forecasts were produced daily initialized at 0000 UTC, along with FSOI also at the 0000 UTC cycle time.

The Previous OSSE framework Control was similarly executed for the period July–August, as described in detail in Privé et al. (2022) and El Akkraoui et al. (2023) and illustrated in Privé et al. (2020), Privé et al. (2021), and Privé et al. (2021).

3.1 Background Error

The background error in the OSSE can be directly calculated using the NR fields as verification, but cannot be calculated for real systems because the true state of the atmosphere is not known. Figure 4 shows the zonal mean root-temporal-mean-square (RTMS) background error for temperature, humidity, and zonal wind for both the Previous and Updated OSSE Control cases. For temperature, there are localized maxima of RTMS error in the lower troposphere in the polar boundary layers and also associated with low cloud decks over cold maritime waters, such as off the west coast of South America. Humidity background errors are greatest in the tropical lower troposphere, where the specific humidities are much larger than in the colder latitudes or upper troposphere. The greatest zonal wind background error is associated with the upper level equatorial easterly jet, with local maxima near the two extratropical westerly jets at the tropopause.

Figure 4

In the real world system, the newer DAS and model used in the Updated framework would be expected to have less background error than for the older DAS and model used in the Previous framework. However in the OSSE, the substantial changes to the GEOS model between the versions used for the Previous and Updated OSSE frameworks should introduce larger differences between the Updated OSSE and the NR compared to the Previous framework, which would tend to increase background error in the Updated OSSE. The overall change in background error in the OSSE framework depends on the relative contributions of these two opposing tendencies. As illustrated in the bottom row of Figure 4, the net tendency is for overall increases in background error in the Updated OSSE framework of 5–10%, with some localized variation.

The background error affects the calibration of the simulated observations in the OSSE. The magnitude of simulated observation error added to the simulated observations is adjusted during the calibration process to match the standard deviation of the observation innovations, O-B, to real observation innovations. If the standard deviation of the background error is smaller in the OSSE than in the real world, the simulated observation error in the OSSE may be overinflated compared to real observation errors in order to compensate. The relationship between the standard deviation of O-B and the variance of the background field is shown in the following equations:

It is assumed that the background error and observation errors are independent so that the covariance, cov(O,B) = 0. Thus it is expected that if the variance of background error in the OSSE framework, var(B), increases, then the magnitude of the variance of simulated observation error, var(O) needed to match real observation innovation standard deviations, stdev(O – B), will decrease. Figure 5 compares the magnitude of the standard deviation of simulated observation error added to the simulated rawinsonde observations in the Previous and Updated OSSE frameworks. Through most of the troposphere, the simulated rawinsonde errors in the OSSE have smaller magnitude in the Updated OSSE framework. Notably for temperature, there is a particularly large drop in the simulated rawinsonde errors in the Updated framework at 900 hPa, corresponding to a level where an especially large increase in background error is noted in Figure 4g.

Figure 5

3.2 Analysis Increments

The zonal mean standard deviations of analysis increments (A-B) for the July–September period are illustrated in Figure 6 for temperature, humidity, and zonal wind for the Updated OSSE framework and corresponding Real data case. The analysis increments for the Updated system can be compared with a similar figure in Privé et al. (2021) for the Previous OSSE framework. In general, it has been found that OSSEs tend to have smaller magnitudes of analysis increments compared to real systems. This can be affected in part by the observations (Privé et al., 2021) but is also a reflection of the model error growth between cycle times. It is clear from the bottom row in Figure 6 that the analysis increments in the OSSE are weaker than for the real world, although the spatial structure of the increments is well represented in the OSSE. The magnitude of standard deviation of analysis increments in the OSSE is generally within 20% of the magnitude of real world increments in the Updated framework, compared to analysis increments that were 30% weaker than the Real case in the Previous OSSE framework.

Figure 6

3.3 Forecast Skill

The regional root-areal-mean-square errors (RAMS) for the 5-day forecast period in the Updated OSSE framework averaged over the three-month period are compared with real world forecast errors in Figure 7 by normalizing the OSSE forecast error statistics by the Real forecast error statistics. For consistency, the RAMS forecast error statistics for the OSSE Control and Real cases are calculated using self-analysis for verification for each case. Note that the analysis state generated by applying an analysis increment to the background field is used throughout this manuscript as the ‘analysis’, differing from the actual model state that is produced by integration of the forward model using the Incremental Analysis Update procedure implemented at GMAO (El Akkraoui et al., 2023). Correlations between short term forecast errors and self-analysis errors can result in substantial underestimations of short-term forecast errors for both Real and OSSE. The fractional forecast error statistics are not expected to approach 1.0 with extended forecast period even as forecast errors saturate due to the use of self-analysis verification, with different analysis states used for the OSSE and Real cases respectively. If the bias and/or variance of the analysis errors are weaker in the OSSE compared to real, the RAMS error-saturated forecast skill will have smaller magnitude and the fractional error will be less than one. The Previous OSSE (heavy black lines) is compared with the Updated OSSE (thin red lines).

Figure 7

OSSE temperature forecast RAMS errors in the troposphere are within 25% of the real case forecast RAMS errors, with equal or higher fractional error in the Updated OSSE framework compared to the Previous framework. For some levels and regions, the OSSE temperature forecast RAMS errors in the Updated system are greater than in the real system, such as for tropical temperatures, which may be related to changes in the model convective scheme. Humidity forecast RAMS errors and zonal wind forecast RAMS errors are somewhat smaller in the OSSE than in the real system, with relatively consistent deficiencies in RAMS error over the five day forecast period, and RAMS errors that are closer to the real system in the Updated OSSE compared to the Previous OSSE.

Figure 8 compares the RAMS forecast error for temperature, humidity, and zonal wind calculated with NR verification in the Previous and Updated OSSE Controls. Temperature at 500 hPa shows significantly greater RAMS forecast error in the Updated framework in the Tropics and Northern Hemisphere extratropics, but little difference in the Southern Hemisphere extratropics. This may be due to seasonal differences, with model-differing convective processes being more dominant in the tropics and summer hemisphere, and baroclinic processes dominant in the winter hemisphere. A diurnal cycle in forecast error is noted in the Updated framework most strongly in the Tropics but also to a slight degree in the Northern Hemisphere extratropics, likely due to the use of only forecasts for the 0000 UTC cycle time. Specific humidity in the lower troposphere demonstrates significantly greater forecast error in the Updated framework in all regions. The humidity RAMS errors are greater not only at the initial time, but there is an increased separation of the error curves as the forecast progresses, indicating faster error growth in the Updated framework compared to the Previous framework.

Figure 8

The upper tropospheric zonal wind forecast RAMS error in Figure 8 shows smaller differences between the OSSE versions than the mass field. As with temperature, there is minimal difference in forecast RAMS error in the Southern Hemisphere extratropics, which is expected due to the strong coupling of wind and temperature in that region. In the Northern Hemisphere, there is slightly faster error growth for wind in the Updated version with a steeper error curve. In the Tropics, mixed behavior is observed, with initially slower wind error growth in the Updated version during the first day of the forecast period, then more rapid error growth after the initial forecast period.

Anomaly correlations of 500 hPa geopotential heights have been a standard metric of forecast performance for decades. In prior versions of the GMAO OSSE, the OSSE skill scores have tended to be higher than that of real forecasts, with fewer dropouts i.e. very low scores, even for frameworks that were not fraternal or identical twins (Privé et al., 2013b). Figure 9 shows distributions of the 120 hour forecast anomaly correlation of 500 hPa geopotential height for the OSSE and Real in both hemispheres. In the Northern Hemisphere, the range of skill scores in the Updated OSSE is similar between the OSSE and Real system, although the OSSE framework has fewer low skill scores and a higher mean. The overall match between Real and OSSE in the Updated framework in the Northern Hemisphere is better than in the Previous framework, although the skill scores are lower for both Real and OSSE in the Updated framework. It should be noted that for the OSSE, the same period of the NR is forecast for both the Previous and Updated OSSE, but for the corresponding Real cases, the Previous Real case is for 2015 while the updated case is for 2020, so the predictability of the real world may differ. In the Southern Hemisphere, the corresponding Real anomaly correlation scores increased significantly for the updated model version compared to the previous model. There is a substantial shift toward higher skill scores in the OSSE compared to Real in the Southern Hemisphere for both the Previous and Updated frameworks. It is notable however that the OSSE skill scores are not out of line with actual performance of other operational systems, such as ECMWF.

Figure 9

3.4 Forecast Sensitivity Observation Impact

The GEOS/GSI system includes a forecast sensitivity observation impact (FSOI) tool that employs an adjoint model to estimate observation impacts on a short-term forecast error norm. A moist adjoint has been developed (Halliwell et al., 2014), and a standard choice of error norm is the total wet energy of the 24 hour forecast. The characteristics of the Previous OSSE framework FSOI have been extensively described by Privé and Errico (2019), Privé et al. (2020), and Privé et al. (2021). Figure 10 shows the global net FSOI results compared to real observations for July. August and September 2020 were subject to multiple real observation outages and are thus omitted, although results are similar to July overall. In the previous OSSE framework, the net impacts of the simulated observations were roughly half the magnitude of corresponding real observations, but the relative impact of different types of observing systems was generally well characterized. This reduced impact was deduced to be due to insufficient model error growth in the OSSE framework compared to the real world, resulting in less “work” for the observations to perform.

Figure 10

The net FSOI impacts for the Updated simulated observations are closer to real observation impacts than for the Previous OSSE framework, although some data types still have substantially lower impacts in the OSSE. Rawinsonde and ATMS impacts are expected to be somewhat lower in the OSSE due to fewer simulated than real observations. Because of the limited number of model levels, simulated rawinsondes have fewer observations in each profile compared to real profiles, but as seen in Figure 10c, the per-observation impact for rawinsondes is similar in the OSSE to real rawinsondes. ATMS uses a complicated ‘super-obbing’ in the real world that proved very difficult to simulate, and as a result the count of actual ingested simulated ATMS observations is approximately 20% lower than for real ATMS. The anomalously small impacts for AMSU-A types in the Updated framework are consistent across all regions and AMSU-A platforms, and are not well understood, especially considering the excellent matching of AMSU-A counts and observation innovations.

A major change to the global observing system in the Updated framework is the implementation of all-sky radiances for AMSR2, GMI, and MHS. The FSOI net impact estimates for the all-sky data types have low magnitude total global impacts for both the Real and OSSE cases, in part due to relatively small observation counts for these data types. Looking at the per-observation impacts, AMSR2 has the largest per-observation impacts, with good agreement between OSSE and Real. Per-observation impacts for GMI and MHS are weaker in the OSSE than for Real, but are in line with the relative performance of clear-sky radiance types.

4.1 Geostationary Hyperspectral Infrared (IR)

In preparation for the proposed sounder onboard the Geostationary eXtended Observations (GeoXO) program’s central satellite, McGrath-Spangler et al. (2022) evaluated the potential impact of geostationary hyperspectral infrared (GEO IR) sounders on global numerical weather prediction. Using the Previous OSSE framework, hourly observations were generated for 2960 channels operating from 650 to 2500 cm^–1 with a spectral wavenumber resolution of 0.625 cm^–1 using the methodology described in Errico et al. (2017). Channel selection was based on an expanded range of the specified Meteosat Third Generation Infrared Sounder (MTG-IRS). Assimilated channels included temperature sounding and water vapor channels, providing atmospheric profiling. A global ring of geostationary satellites was generated based on the GOES-R coverage, relocated to the satellite subpoints of 105°W, 0°, 105°E, and 140°E longitude, consistent with proposed locations of GeoXO’s central satellite, MTG-S, Fengyun-4A, and Himawari-10, respectively. Synthetic errors were not added to the simulated geostationary hyperspectral IR observations. The experiments of McGrath-Spangler et al. (2022) were repeated using the Updated OSSE framework.

First, a comparison of the RTMS analysis error impacts of the GEO IR observations between the Previous OSSE and the Updated OSSE is provided in Figure 11. In this figure, the zonal mean normalized differences in the error show the impact of assimilating geostationary IR sounder radiances on temperature (T), specific humidity (Q), and zonal wind (U) relative to the Control. Temperature and specific humidity are directly impacted by the GEO IR radiances through assimilation of temperature and water vapor sensitive channels. The zonal wind, however, is impacted indirectly through covariances with the directly observed variables (Peubey and McNally, 2009) and benefits from the high temporal resolution possible with a geostationary orbit.

Figure 11

In the Previous OSSE, the temperature impact showed a beneficial impact throughout the tropical troposphere aloft of about 900 hPa. The largest improvement occurring in the tropics is expected because these are the latitudes with the broadest observational coverage by a geostationary orbit. However, there is a degradation south of approximately 30°S associated with an increased error off the west coast of South America (McGrath-Spangler et al., 2022). An even larger temperature degradation exists near the South Polar tropopause, outside the coverage of the assimilated GEO IR satellite observations. Tests showed that this South Polar impact (and all impacts outside the GEO IR observed range) is not directly related to the observations, but is instead a result of model interactions.

With the Updated framework, the strong polar degradation aloft is eliminated and most of the global troposphere is improved by the addition of the observations targeted in these experiments. In addition to the large improvements in the upper tropical troposphere and around 850 hPa, an additional mid-tropospheric region of improvement is present around 10°N in the Updated OSSE. This region corresponds to the areas of increased background temperature error in the Updated framework at the same latitude in Figure 4g. Aloft of 800 hPa over the South Pole, the response is beneficial due to a transference of the information content beyond the latitudes observed by the GEO sounders. At these high latitudes, a small degradation exists above the Antarctic ice and below 800 hPa in the Updated OSSE that was not present in the previous iteration.

The specific humidity is subtly affected by the choice of data assimilation system chosen. In the Previous OSSE, the main benefit exists in the mid-troposphere from between roughly 700 and 300 hPa and equatorward of 30°. A mild degradation below 800 hPa within this latitude range and minor degradations within the extratropical troposphere exist, which is detrimental to the analysis. An examination of the impact in the Updated OSSE reveals a less intense, but more widespread beneficial impact. The lower tropospheric tropics now show a beneficial impact throughout the tropics that is noteworthy due to its co-location with the high water vapor concentrations in this region and their relevance for low-level cloud formation. This region of enhanced beneficial humidity impacts in the Updated framework corresponds to the large increase in background humidity error as seen in Figure 4h. Weaker humidity impacts in the mid-tropospheric tropics in the Updated framework may similarly be related to smaller magnitudes of background error in this region compared to the Previous framework. Additionally, the extratropical degradations are nearly eliminated in the Updated OSSE.

For wind, the Previous OSSE shows benefits mostly within the tropical tropospheric column, but a mild degradation in the extratropics poleward of 30°. The largest improvements are at about 20°S near the tropopause but extend downward to about 800 hPa. In the Updated OSSE, the degradations in the extratropics are nearly eliminated while the tropical benefits are maintained. The largest intensity improvement is retained in the upper troposphere at about 20°S and that extends through the mid-troposphere. Overall, the differences between impacts in the two OSSE frameworks are minimal in the tropics.

A comparison of the relative impact of geostationary IR radiance assimilation on forecast RAMS error is shown in Figure 12. This figure shows the global mean error impacts normalized by the respective Control experiment’s forecast errors, computed for the months of July and August. Shown are the impacts on T, Q, and U. Statistical significance at the 90% level is indicated by stippling.

Figure 12

In the Previous OSSE, the temperature impact presented predominantly between 400 and 300 hPa up to the one day forecast with a weaker, beneficial response between 900 and 400 hPa and aloft of 150 hPa. The maximum extent of statistical significance was to 2 days at the higher altitudes. As evaluated in this framework, a small, statistically insignificant degradation was present at about 200 hPa starting from about the 12 h forecast and throughout the troposphere beyond the 4 day forecast. This is consistent with prior work that found that the largest impacts occur during the early forecast period (Privé and Errico, 2013a; Cucurull and Casey, 2021; Privé et al., 2022) and that the impact of the initial condition improvement weakens as model error and chaotic error growth begin to dominate. Tropical forecast improvements are the largest (not shown), with longest duration of significant benefit, with little to no improvement in the Southern Hemisphere extratropics, and weaker, short-lived improvements in the Northern Hemisphere extratropics.

In the Updated OSSE framework, the beneficial impact extends throughout the troposphere and throughout the forecast period with statistical significance extending to nearly the 5 day forecast at the upper levels and approximately 3 days elsewhere. Within the atmospheric column, particular layers of improvement exist with the most intense occurring between 700 and 500 hPa. In this framework, the beneficial impact penetrates deeper into the atmosphere and persists longer with only limited areas of degradation. The extended beneficial impacts of GEO IR on temperature in the Updated framework are due to contributions from both improved impacts in the Southern Hemisphere extratropics and the tropics, where significant beneficial impacts persist into the 2–3 day range (not shown). The improved temperature analysis impacts in the Southern Hemisphere extratropics likely contribute to the longer duration of improved forecast impacts seen in the Updated framework compared to the Previous framework.

Similarly, the impact on water vapor forecasts is affected by the choice of OSSE framework used. In the Previous OSSE framework, the specific humidity is strongly affected between 700 and 300 hPa with a weaker beneficial impact aloft of 300 hPa and near the surface, and a degradation around 800 hPa. The mid-tropospheric improvement and the improvement at 100 hPa are statistically significant until approximately 4 days while the impacts at other levels are for shorter duration. Although an improved mid-tropospheric specific humidity forecast is important for entrainment, the degradation at the height of boundary layer clouds is not ideal. The strong initial impacts that diminish rapidly during the early forecast period are indicative of a model error growth process that acts quickly to eliminate information from the observations, such as moist convection.

In the Updated OSSE (Figure 12d), the impact of assimilating geostationary IR radiances is significantly beneficial throughout the troposphere, up to 200 hPa. The largest improvement occurs at 900 hPa and retains significance throughout the 5 day forecast with the greatest intensity through the 1 day forecast. This beneficial impact in the lower layers at days 3–5 is largely due to improvements in the tropics (not shown). Although a lower intensity mid-tropospheric improvement is also present in the Updated OSSE, the improvement in the lower troposphere, rather than the degradation seen in the Previous OSSE is meaningful for the forecasting of boundary layer clouds, a critical component of the climate system.

In the Previous OSSE framework, there is a weak beneficial impact on the zonal wind forecast throughout the troposphere that is statistically significant until day 2 in the mid-troposphere. After day 3, the forecast impact evolves into a degradation. The degradation is caused by both a convective feedback in the tropics that evolves after day 2, as well as detrimental impacts in the Southern Hemisphere extratropics after the initial 24-hour forecast period (not shown). In the Updated version, the tropospheric zonal wind improvement persists throughout the 5 day forecast with statistical significance continuing almost until day 3 through much of the atmospheric column and until day 5 at certain levels such as at 300 and 750 hPa. The long duration improvement at 300 hPa is consistent with the height of the midlatitude jets, and improvements above 300 hPa have a substantial contribution from the Southern Hemisphere extratropics. While there is still some evidence of a detrimental convective feedback in the tropics in the Updated OSSE, the Southern Hemisphere extratropics retains beneficial impacts through the five-day forecast period, unlike in the Previous OSSE framework.

4.2 GNSS-RO

In this experiment, approximately 100,000 additional GNSS-RO profiles per day are added to the Control case, following the method described in detail in Privé et al. (2022). Here, the Radio Occultation Processing Package operator (ROPP; Culverwell et al. (2015)) was used to generate the simulated observations using 2-dimensional ray tracing techniques below 10 km and 1-dimensional methods at upper levels. The spatiotemporal locations of the simulated RO profiles were based on the locations of actual GNSS-RO observations in 2009, but with 40 days of real observations occurring in a single simulated day. Simulated observation errors were added to the simulated RO profiles with characteristics matching those calibrated for the 2015 GNSS-RO dataset in the Previous OSSE framework. In the Updated OSSE framework, the experiment was repeated using both the same RO dataset as in the Previous OSSE framework and then again with simulated RO errors matching those calibrated for the 2020 GNSS-RO dataset. The two datasets yielded nearly identical results, and the results using the latter dataset are shown here. In this comparison, cycling was performed for the month of July only; forecasts and FSOI were not produced.

Figure 13 compares the zonal mean RTMS analysis error impacts normalized by the respective Control RTMS analysis errors for the 100k RO experiment in the Previous and Updated OSSE frameworks for the month of July. Temperature and specific humidity are shown as these are directly influenced by the RO data. In the Previous OSSE, temperature improvements were seen in the stratosphere and above 500 hPa in the troposphere. However, some degradation of the analysis state was observed poleward of 40°N/S due to ingestion of the additional GNSS-RO data. This was thought to be due to the use of an observation error weighting (R) by the GSI that has a step function form, with lower observation errors poleward of 40 N/S and higher errors in the tropics (Privé et al., 2022). A localized region of low level degradation over Arctic marine areas was also noted and assessed as being a cold bias related to a boundary level inversion near 900 hPa.

Figure 13

In the Updated OSSE framework, improvements to the analysis due to ingestion of additional RO data are similar to that seen for the Previous framework, but the regions of degradation are amplified, particularly in the Northern Hemisphere. It is noted that there was not a change to the assigned observation error weighting in GSI between the Previous and Updated versions. The strong degradation over Arctic marine surfaces remains in the Updated framework, but now extends up to the mid troposphere. This degradation takes the form of a strong cold bias over marine areas (not shown) below 700 hPa, similar to that seen in the Previous OSSE framework (Privé et al., 2022) but with considerably larger magnitude. Between 700 hPa and 500 hPa, the cold bias is weaker but a large temperature error variance is present. While the temperature increments are clearly detrimental for lower tropospheric RO over the Arctic region, the bending angles show a closer fit for observation minus analysis (O-A) compared to O-B, and the exact cause of this discrepancy is the subject of ongoing investigation.

In the Southern Hemisphere, the RO impacts are fairly similar between the Previous and Updated OSSE. As in the Northern Hemisphere, there is a stronger region of degradation poleward of 40°S in the mid to lower troposphere, indicating that the Updated framework is more sensitive to the assigned observation error weighting than in the Previous framework. Over Antarctica, a region of enhanced beneficial impacts between 900–400 hPa corresponds to a similar area of greater background error in the Updated framework as noted in Figure 4g.

For specific humidity, experiments with the Previous OSSE framework showed beneficial impacts of the additional GNSS-RO observations throughout the troposphere. Degradation of specific humidity in the stratosphere was assessed to be due to poor representation of stratospheric moisture in the OSSE. Improvements to the humidity field were seen in regions where the temperature field was degraded. In the Updated OSSE framework, beneficial impacts of additional GNSS-RO profiles are again observed in the troposphere, but are somewhat weaker overall than in the Previous OSSE. Impacts are most beneficial in the tropics and subtropics, but weaker or even slightly deleterious in polar regions. The change from beneficial to neutral/degradation north of 50°N between the Previous and Updated OSSE versions is likely due to the stronger degradation of the temperature field in the Updated version, as humidity is affected by temperature increments. The lower tropospheric area of degradation of the humidity field near 90°N may also be related to the improvement of the Control background Q field in this region (Figure 4h), as a higher quality background is more likely to be degraded by imperfect observations.

4.3 MISTiC Winds

In this experiment, simulated atmospheric motion vectors for the Midwave Infrared Sounding of Temperature and humidity in a Constellation for Winds (MISTiC Winds) instrument were tested in both OSSE frameworks for the July–August period. The simulated AMVs were generated based on the location of cloud and water vapor features in the NR using probabilistic methods to determine whether an observation was present following the procedure outlined in McCarty et al. (2021). MISTiC radiances were not included in this experiment, and the simulated MISTiC AMVs did not include additional simulated errors for simplicity. A single orbit was selected from the MISTiC constellation of sun-synchronous orbits at 705 km, having local time of ascending node of 1330. Figure 14 compares the zonal mean RTMS zonal wind analysis error of the MISTiC AMV observations normalized by the RTMS Control analysis error in the Previous and Updated OSSE frameworks for the months of July and August, with 4 times daily analyses. The spatial pattern of observation impacts is quite similar, with largest impacts poleward of 60°N/S. The observation impacts have somewhat greater magnitude in the Updated OSSE framework compared to the Previous OSSE, particularly near the poles. This is in line with the overall greater background error for winds in the extratropics seen in Figure 4i for the Updated framework. The background wind error is reduced near the equator in the Updated Control, with a local region of reduction between 100–200 hPa, which may account for the shift of the greatest equatorial MISTiC impact from 100–200 hPa in the Previous framework to 200–300 hPa in the Updated framework.

Figure 14

The FSOI estimates of global observation impacts on the 24 hour forecast total wet energy error norm are shown for the Control and MISTiC cases in Figure 15. The impact magnitudes for individual types should not be compared between the Previous and Updated OSSE frameworks, it is the relative ranking of impacts that is more pertinent. In the Previous OSSE, MISTiC AMV impacts ranked fourth in net global impact, below combined other AMV types, rawinsondes, and the constellation of AMSU-A instruments. In the Updated OSSE, MISTiC AMVs rank a very close third to rawinsondes and other AMV types, although it is noted in Figure 10 that AMSU-A observations are relatively underweighted in the Updated OSSE. In both the Previous and Updated OSSE frameworks, the impacts of other data types, in particular AMVs, rawinsondes, and AMSU-A, are reduced when MISTiC observations are added, implying a likely degree of redundancy between these data types. While it could be argued that the MISTiC impacts in the Updated OSSE are slightly larger than in the Previous OSSE, which would align with the slightly greater analysis impacts in the Updated framework, the overall FSOI estimate of MISTiC performance is similar in both frameworks.

Figure 15

It is notable from Figure 10 that in the OSSE Control, existing AMV types have the largest net impact of any instrument type, while in the Real world, AMVs are ranked third. AMV observations have persistently had an overly large impact in the OSSE framework through multiple changes to the Nature Run, simulated observations, DAS, and forward model (Privé et al. (2013b), Privé and Errico (2019)). Significant changes to the simulated AMV errors and the method of generating AMVs have not lead to a reduction in the relative overestimation of AMV impacts in the OSSE framework over numerous versions of the OSSE.