Outcomes Affect Evaluations of Decision Quality: Replication and Extensions of Baron and Hershey&rsquo;s (1988) Outcome Bias Experiment 1

Sriraj Aiyer; Hoi Ching Kam; Ka Yuk Ng; Nathaniel A. Young; Jiaxin Shi; Gilad Feldman

doi:10.5334/irsp.751

Figures & Tables

Table 1

LeBel et al. (2018) replication taxonomy for this replication study to classify the methodological similarity to the original study from Baron and Hershey (1988). Each facet of the experimental design within Baron and Hershey (1988) and our current study are compared to each other for similarity, with any key differences between the two described.

DESIGN FACET	SAME/DIFFERENT TO ORIGINAL PAPER	NOTES
Effect/hypothesis	Same
IV construct	Same
DV construct	Same
IV operationalization	Same
DV operationalization	Same
Population (e.g., age)	Different	Original study recruited undergraduates.
IV stimuli	Same
DV stimuli	Similar	Original DV stimuli used, with added extensions.
Procedural details	Different	Study conducted via Qualtrics, changed from within-participants to a between-participants study design.
Physical setting	Different	Study conducted online rather than in person.
Contextual variables	Different
Overall	Close to far replication

Table 2

Baron and Hershey (1988) replication and extension hypotheses.

#	HYPOTHESIS
Original
1	Decisions that resulted in successful outcomes are rated as better than decisions that resulted in failed outcomes.**
2	Participants who report thinking that judgments should not be based on outcomes demonstrate an outcome bias.**
Extensions
3a	Successful outcomes are rated higher on outcome importance than failed outcomes.*
3b	Perceived outcome importance partially accounts for (mediates) outcome bias.
4a	Failed outcomes are rated as higher perceived level of responsibility of the decision-maker than in successful outcomes.*
4b	Perceived decision-maker responsibility partially accounts for (mediates) outcome bias.
5a	Decisions resulting in failed outcomes are perceived as less normative than decisions resulting in positive outcomes.
5b	Perceived norms partially account for (mediates) outcome bias.

[i] Note: ** Pre-registered hypotheses in both OSF pre-registrations. * Pre-registered hypotheses in one pre-registration. Two pre-registrations were created by independent analysts. Refer to Table 3 for details on the divergence between pre-registrations.

Table 3

Comparison between the two crowdsourced pre-registrations.

HYPOTHESIS/ANALYSIS	H. C. K. PRE-REGISTRATION	K. Y. N. PRE-REGISTRATION
Original hypotheses	Included	Included
Extension hypotheses	Included	Not included
2 × 2 ANOVAs on DVs	Included	Not included
Power analyses	Included	Included
Exclusion criteria	Included	Included

[i] Note: H. C. K.’s pre-registration is available on OSF (https://osf.io/pwgq4), and K. Y. N.’s pre-registration file is available here: https://osf.io/ydxg7. Our aim was to crowdsource the pre-registrations by having two coauthors independently analyze the target article and plan analyses. We aimed to address both by following the strictest, most conservative combination of the two. The Qualtrics survey was included in the pre-registration and is available on OSF: https://osf.io/q4xbf (exported Word file) and https://osf.io/vfw38 (QSF Qualtrics import file).

Table 4

Replication study experimental design.

IV1: Outcome [Between]	IV1: Outcome manipulation Outcome: Success	IV1: Outcome manipulation Outcome: Failure
IV2: Decision-maker [Between]	Manipulation: ‘The operation succeeded.’	Manipulation: ‘The operation failed.’
IV2: Decision-maker manipulation condition A Decision-maker: Patient Manipulation: Participant was told ‘the patient decided to go ahead with the operation’.	Dependent variables Decision quality: ‘Please evaluate the physician’s/patient’s decision on a scale from 3 (Clearly correct and the opposition decision would be inexcusable) to –3 (Incorrect and inexcusable).’ Perceived outcome importance: ‘Do you think you should take the outcome into account in evaluating the decisions? Please rate on a scale from 1 (Definitely not) to 5 (Definitely yes).’ Perceived responsibility: Item: ‘Rate the level of responsibility of the patient for the decision made to go ahead with the operation on a scale from 1 (No responsibility) to 7 (Full responsibility).’ Perceived norms: Item: ‘Do you think that most people in this situation would decide to go ahead with the operation? Please rate on a scale from 1 (Definitely not) to 5 (Definitely yes).’
IV2: Decision-maker manipulation condition B Decision-maker: Physician Manipulation: Participant was told ‘his (the patient’s) physician decided to go ahead with the operation.’

Table 5

Means and standard deviations for each measured variable for all conditions.

	SUCCESS					FAILURE
	PHYSICIAN(n = 173)		PATIENT(n = 171)		PHYSICIAN(n = 172)		PATIENT(n = 176)
	M	SD	M	SD	M	SD	M	SD
Evaluation	1.81	0.84	1.76	0.79	0.45	1.55	0.90	1.36
Outcome importance	4.49	0.82	4.40	0.77	4.13	1.04	4.28	0.91
Responsibility	5.88	1.02	6.13	0.97	5.23	1.36	5.78	1.21
Act the same	4.03	0.68	4.06	0.62	3.71	0.87	3.86	0.75

Decision quality evaluations: decision-maker and outcomes.
**Note:** The effect of outcome type (success vs. failure) and decision-maker type (physician vs. patient) on evaluations of decisions. Successes were rated as more correct than failures. Patients were rated more correct on average regardless of the outcome compared to physicians. Outcome type and decision-maker type interacted to account for decision evaluations. Physicians’ decisions were evaluated as less correct than patients’ decisions when the outcome was a failure but equally as correct when the decision resulted in a success. Bayes factors are reported as per the built-in function within the ggstatsplot package in R.

Table 6

Evaluation justifications’ ratings.

JUSTIFICATION	PHYSICIAN SUCCESS	PHYSICIAN FAILURE	PATIENT SUCCESS	PATIENT FAILURE	TOTAL
Outcome	45	19	28	24	116 (16.5%)
Ethical concerns	16	29	3	0	48 (6.8%)
Others	108	123	139	153	523 (35.9%)
Unclear	8	4	6	0	18 (2.6%)
Total	177	175	176	177	705

Table 7

Comparison of effects between the target article and our replication.

DECISION-MAKERS	ORIGINAL EFFECT SIZE ESTIMATE (d_paired) AND 95% CONFIDENCE INTERVALS	REPLICATION EFFECT SIZE (d_independent) AND 95% CONFIDENCE INTERVALS	REPLICATION INTERPRETATION (LEBEL ET AL., 2019)
Patient	0.21 [–0.23, 0.66]	0.77 [0.62, 0.93]	Signal and same direction
Physician	0.53 [0.06, 0.99]	1.10[0.94, 1.26]	Signal and same direction
Aggregate of all scenarios	0.90 [0.37, 1.42]

[i] Note: The effect for the original is for paired samples, whereas our replication is for independent samples and should therefore be interpreted with caution.

Outcome importance: decision-maker and outcomes.
**Note:** The effect of outcome type (success vs. failure) and decision-maker type (physician vs. patient) on the perception of outcome importance.

Perception of responsibility: decision-maker and outcomes.
**Note:** The effect of outcome type (success vs. failure) and decision-maker type (physician vs. patient) on the perception of responsibility. When the outcome was a success, there were stronger perceptions that the decision-maker was responsible than when outcome was a failure.

Perceived social norms: decision-maker and outcomes.
**Note:** The effect of outcome type (success vs. failure) and decision-maker type (physician vs. patient) on the perceptions of social norms—whether others would perform the same action. When the outcome was a success, there were stronger perceptions others would perform the same action as the decision-maker than when outcome was a failure.

ROLE	SRIRAJ AIYER	HOI CHING KAM AND KA YUK NG	NATHANIEL A. YOUNG	JIAXIN SHI	GILAD FELDMAN
Conceptualization					X
Pre-registrations		X			X
Data curation
Formal analysis	X	X	X
Funding acquisition					X
Investigation		X			X
Methodology		X			X
Pre-registration peer review/verification		X		X	X
Data analysis peer review/verification	X	X	X
Project administration					X
Resources					X
Supervision				X	X
Validation	X		X
Visualization	X		X
Writing—original draft		X	X
Writing—review and editing	X				X

Outcomes Affect Evaluations of Decision Quality: Replication and Extensions of Baron and Hershey’s (1988) Outcome Bias Experiment 1

Figures & Tables

Table 1

Table 2

Table 3

Table 4

Table 5

Figure 1

Table 6

Table 7

Figure 2

Figure 3

Figure 4

Paradigm

My account