Quality of cost evaluations of physician continuous professional development: Systematic review of reporting and methods

Study design

Added option for economic modeling studies (score 1.5)

1‑group post-only (1)

 6 (10%)

1‑group pre-post, or modeling (1.5)

20 (32%)

2‑group non-randomized (2)

16 (26%)

2‑group randomized (3)

20 (32%)

Sampling: No. of institutions studied

No change

1 (0.5)

54 (87%)

2 (1)

 1 (2%)

>2 (1.5)

 7 (11%)

Sampling: Response rate

For cost data: Data derived from large record sets unlikely to reflect bias (e.g., institutional electronic health record or regional claims database) count as high (score 1.5)

<50% or not specified (0.5)

24 (39%)

50–74% (1)

 7 (11%)

≥75% or large record

31 (50%)

Type of data (data source)

For cost data: Details of resource quantitation (both data source and quantity [number of units, not just total cost]) count as high (score 3). Cost alone counts as low (score 1)

Self-reported data, or cost without resource quantitation (1)

 8 (13%)

Objective measurement, or cost with data source and quantity (3)

54 (87%)

Validation of evaluation instrument: Content

For cost data: “The degree to which the cost estimation encompasses all aspects of the true cost, encompassing processes to both identify and measure cost” [15]. Evidence could include use of a formal framework (e.g., the Ingredients Method) or the involvement of experts in planning, empiric identification and selection of relevant resources (e.g., time-motion studies or process mapping), and substantiation that a robust data source was used to select, quantitate, or price resources (e.g., detailed description of a computer database)

Reported (1)

 8 (13%)

Validation of evaluation instrument: Internal structure

For cost data: “The degree to which the cost estimate is reproducible if the same method is followed” [15]. Evidence could include replicability of the valuation or analysis (e.g., robust examination of the uncertainty of input parameter estimates [sensitivity analysis], independent valuation of costs by two investigators [inter-rater reliability], or comparing cost estimates derived at two different time points [temporal stability])

Reported (1)

 9 (15%)

Validation of evaluation instrument: Relations with other variables

For cost data: “The degree to which the cost estimate relates to cost estimates formed using alternative approaches” [15]. Evidence could include examining predicted associations among results obtained using alternative approaches to economic modeling (e.g., sensitivity analysis comparing different base assumptions, valuation methods, statistical models, or economic theories)

Reported (1)

 1 (2%)

Data analysis: Appropriateness

For cost data: The following count as “appropriate” (score 1): cost effectiveness ratio, net benefit, or other similar analysis of cost data

Inappropriate for study design (0)

37 (60%)

Appropriate (1)

25 (40%)

Data analysis: Complexity

For cost data: The following count as “beyond descriptive” (score 2): cost effectiveness ratio, net benefit, visual display of cost-effectiveness

Descriptive analysis only (1)

37 (60%)

Beyond descriptive analysis (2)

25 (40%)

Outcomes

For cost outcomes: As per Foo, we distinguished education costs in a “test setting” or a “real setting,” namely: “Test settings are those in which the context does not match how the intervention would be utilized in actual practice (e.g., a hypothetical program that was not actually implemented). Real settings are where the intervention is evaluated in a context similar to its anticipated utilization in practice (e.g., an evaluation of a program that is taught to real students)” [15]. However, we assigned points differently than Foo: score 1.5 for cost of education in test setting, score 2 for cost of education in real setting, score 3 for health care costs. Outcomes estimated from previously published research (including health care costs and non-cost outcomes) also score 1.5

Knowledge, skills, or education costs in a “test” or hypothetical training setting, or estimated from literature (1.5)

 1 (2%)

Behaviors in practice or education costs in a “real” training setting (2)

25 (40%)

Patient effects, including health care costs (3)

36 (58%)