Table 1
Criteria for Appraisal of Quasi-Experimental Studies.
| CRITERIA | DESCRIPTION |
|---|---|
| Exchangeability Domain | |
| More than one pre-period | To assess whether trends for the outcome in the pre-policy period are parallel, there is more than one pre-period time point [21, 22]. |
| Graphical and/or statistical evaluation of parallel trends | The trends are evaluated graphically and/or statistically to determine if they are comparable [21, 23]. |
| Weighted/matched regression | Use propensity score matching or weighting to balance intervention and comparison groups on observable baseline characteristics [24]. |
| Comparison of changes in observable characteristics | Assess changes over time in the composition of the intervention and comparison groups [21]. |
| No Time-Varying Confounding Domain | |
| Test whether pre-policy trend predicts policy change | Use statistical tests to determine whether the trend for the outcome in the pre-policy period predicts the policy change [21]. |
| Control for or discuss potential sources of time-varying confounding | Provide a discussion of sources of potential time-varying confounding (e.g., contemporaneous policy changes) and control for them where possible). |
| Triple-difference model | Employ a difference-in-difference-in-differences model to control for potential time-varying confounding [25]. |
| Modelling Domain | |
| Different functional forms are considered | If the outcome is non-linear, consider alternative functional forms [26]. |
| Standard errors are adjusted for clustering and serial correlation | Adjust standard errors and inferential statistics for correlation between individuals in a practice/group and within individuals over time [27]. |
| Large number of groups (organizations, regions, practices) | Include a large number of groups (e.g., ACOs) to improve the power of inferential statistics [27]. |
| Placebo testing | Test the robustness of estimates by determining whether the statistical models find an effect in places they should not (e.g., outcomes not affected by policy change, time-periods before policy change) [21, 28]. |
Table 2
Major CMS ACO Models in the US.
| MODEL TYPE | RISK AND SHARED SAVINGS | NUMBER OF ACOS | NUMBER OF BENEFICIARIES |
|---|---|---|---|
| Medicare Shared Savings Program, MSSP (2012-ongoing) [29] | One-sided: Share savings with the CMS up to a maximum of 50% (if quality performance standards are met). Two-sided: Larger share of savings in exchange for sharing losses with CMS. Maximum 60% (if quality performance standards are met). | One-sided (2020): 325 Two-sided (2020): 192 | Total (2020): 11.2 million Mean per ACO (2020): 21,663 |
| Pioneer ACO Program (2012–2016) [34, 35] | Originally less financial risk. Not responsible to pay CMS for any losses during contract period. | 32 launched (2012) 9 remaining in 2016 | Total (2014): 816,362 Mean per ACO (2014): 35,494 |
| Advanced Savings Model, ASM (2012–2015) [30, 31] | One-sided: Share savings only with the CMS 50%. Two-sided: Larger share of savings in exchange for sharing loses with CMS. Savings/loss rates: 2–3.9% based on ACO size (difference between an ACOs benchmark and actual spending). | 36 launched (2012) 33 remaining in 2015 | Total (2014): 288,278 Mean per ACO (2014): 8,237 |
| ACO Investment Model, AIM (2015–2020) [32, 33] | Purpose of AIM is to enable smaller/rural ACOs to transition from one-sided to two-sided risk, wherein they become liable for paying CMS a percentage of Medicare spending above their benchmark. | 45 launched (2015) 14 remaining in 2020 9 moved to two-sided risk by 2019 and 7 of these remained in 2020 | Total (2017): 487,000 Mean per ACO (2017): 10,822 |
| Next Generation ACO, NGACO (2016-ongoing) [36, 37] | Providers take on higher levels of financial risk for greater rewards. If spending exceeds benchmark 80–100% loss share rate. If spending is below the benchmark 80–100% savings share rate. Physicians eligible for 5% bonuses starting in 2019. | 18 launched (2016) 41 operating in 2019 | Total (2019): 1,399,398 Mean per ACO (2019): 34,132 |
Table 3
Summary of ACO Success Factors.
| STUDY* | GLOBAL BUDGETS, AQC, AND INCENTIVES | INDEPENDENT PHYSICIAN GROUP-LED ACOS | HOSPITAL-LED ACOs | BASELINE OUTCOMES AND STARTING POINTS | CONSISTENCY OF CARE AND PROVIDER BUY-IN | SHIFTING CARE TO OUTPATIENT SETTINGS | RISK** |
|---|---|---|---|---|---|---|---|
| Barry (2015) [72] | ✔ | ||||||
| Borza (2019) [57] | ✔ | ✔ | ✔ | ||||
| Chien (2014) [42] | ✔ | ||||||
| Christensen (2016a) [45] | ✔ | ✔ | |||||
| Christensen (2016b) [64] | ✔ | ✖ | ✔ | ||||
| Colla (2016) [60] | ✔ | ||||||
| Colla (2019) [70] | ✔ | ||||||
| Geyer (2016) [54] | ✔ | ✔ | |||||
| Huskamp (2016) [46] | ✔ | ||||||
| Joyce (2017) [73] | ✔ | ||||||
| Kelleher (2015) [55] | ✔ | ✔ | ✔ | ||||
| Lowell (2018) [71] | ✖ | ✖ | |||||
| McWilliams (2013) [51] | ✔ | ||||||
| McWilliams (2014) [61] | ✔ | ||||||
| McWilliams (2015) [58] | ✖ | ✔ | |||||
| McWilliams (2016) [52] | ✔ | ✖ | |||||
| McWilliams (2017) [74] | ✖ | ||||||
| McWilliams (2018) [53] | ✔ | ||||||
| Nyweide (2015) [62] | ✔ | ✔ | ✔ | ||||
| Resnick (2018) [44] | ✔ | ||||||
| Rutledge (2019) [47] | ✔ | ✔ | |||||
| Ryan (2017) [56] | ✔ | ✔ | |||||
| Song (2011) [43] | ✔ | ✔ | ✔ | ||||
| Song (2012) [59] | ✔ | ✔ | ✔ | ✔ | |||
| Song (2017) [48] | ✔ | ||||||
| Stuart (2017) [49] | ✔ | ||||||
| Trombley (2019) [63] | ✔ | ✔ |
[i] * This table only lists studies that have demonstrated a positive impact of ACOs on at least one pre-specified outcome.
** Since very few ACOs assumed 100% risk in the first year of operation, whether assuming 100% risk was associated with reduced spending remained unclear. However, the few ACOs that did assume 100% risk showed significantly lower Medicare spending.
Legend: ✔ = factors linked to success of ACOs, as identified by thematic analysis; ✖ = factors that challenged the success of ACOs, as identified by thematic analysis; blank = factors not discussed in the study.
Abbreviations: Alternative Quality Contract, AQC.
Table 4
Findings from Seven Higher Quality Quasi-Experimental Studies.
| STUDY | SUMMARY OF FINDINGS |
|---|---|
| Song (2011) [43] | The implementation of the Blue Cross Blue Shield of Massachusetts AQC was associated with “modest slowing of spending growth and improved quality.” While a higher-quality study based on our criteria, the authors only observed one year of outcomes post-implementation. |
| McWilliams (2013) [51] | Studied the impact of the AQC observing two years of implementation (2009 and 2010) and two years post-implementation. The authors found that the implementation of the AQC was associated with lower spending after the second year, particularly in outpatient care, procedures, imaging, and tests. They also found associations with improvements in some quality of process measures for diabetes and cardiovascular disease, but not with hospitalization, readmission, or cancer screening. |
| McWilliams (2016) [52] | Evaluated the performance of MSSP ACOs and compared primary care groups to hospital-integrated groups. The authors found that the introduction of the MSSP ACOs was associated with reduced Medicare spending by the ACOs that entered the MSSP in 2012, but not those that entered in 2013. Generally, savings were greater among primary care groups than hospital-integrated groups. The authors found mixed results on measures of quality. |
| McWilliams (2017) [74] | This study evaluated the impact of the MSSP on post-acute care spending and utilization. The authors found that participation in an MSSP was associated with reductions in post-acute care spending without any reduction in care quality. |
| Song (2017) [48] | Studied the impact of the AQC on spending and quality of process and outcome measures comparing enrollees with both lower- and higher socioeconomic statuses. The difference-in-difference-in-differences approach was used to compare enrollees to non-enrollees across these socioeconomic strata. Their findings suggested that the implementation of the AQC was generally associated with improvements in quality of process measures, and that the magnitude of the improvement was higher among those of lower socioeconomic status. However, the authors found no difference in outcome measures or spending across SES strata. |
| McWilliams (2018) [53] | This study evaluated the impact of the MSSP after three years of operation. In particular, the authors studied whether the savings achieved by early adopters were replicated by newer ACOs. The authors found that participation in the MSSP was associated with reductions in Medicare spending among physician-led groups, but not among hospital-integrated ACOs. |
| Resnick (2018) [44] | This study evaluated the impact of MSSP ACO enrollment on changes in appropriate cancer screening rates. Appropriateness was determined based on patient age and predicted survival. If screening increased for those who would most benefit and decreased for those who would not, then appropriateness was improved. The authors found that enrollment in an MSSP ACO was associated with “modest” improvements in appropriate breast and colorectal cancer screening. MSSP ACO enrollment was also associated with decreased prostate cancer screening regardless of age or predicted survival. |
[i] Abbreviations: Alternative Quality Contract, AQC; Medicare Shared Savings Program, MSSP; Socioeconomic status, SES.