| QUESTION | HYPOTHESIS | SAMPLING PLAN | ANALYSIS PLAN | RATIONALE FOR TEST | INTERPRETATION GIVEN DIFFERENT OUTCOMES | THEORY THAT COULD BE SHOWN WRONG BY THE OUTCOMES |
|---|---|---|---|---|---|---|
| Is the sunk cost effect weaker for time than for money? | The sunk cost effect is weaker for time than for money. | Participants recruited online using the US American Amazon platform. | Chi-square test | We follow the statistical methods of the original paper. | Based on the criteria used by LeBel et al. (2018) we will examine the replicability of the findings of Soman (2001). | The sunk cost effect is weaker for time than for money and the facilitation of money-like accounting for sunk time costs strengthens the sunk cost effect. |
| Does the facilitation of money-like accounting for sunk time costs strengthen the sunk time cost effect? | Facilitation of money-like accounting by using education about economic approaches to time strengthens the sunk cost effect of time | Two-way between-subject ANOVA |
Table 1
Soman (2001): Summary of studies and hypotheses and a comparison of original and replication effects.
| HYPOTHESES | STUDY | DESCRIPTION | STATISTICAL TEST | ORIGINAL OR REPLICATION | EFFECT SIZEa [95% CI] | REPLICATION OUTCOMEb |
|---|---|---|---|---|---|---|
| Hypothesis 1: The sunk-cost effect is weaker in the domain of temporal costs than in the domain of monetary costs. | 1 (Theatre and concert tickets) | Two types of tickets are expressed in two different types of sunk cost domains—either time or money to investigate the relative strength of each domain. | Chi-square; difference between sunk time and sunk money conditions in rate of choosing a ticket | Original | ϕc = .61 [.43, .78] | signal – inconsistent, smaller |
| Replication | ϕc = .38 [.31, .45] | |||||
| 2 (Choosing a project) | The domain (time/money) and the existence of sunk cost (present/absent) are manipulated within a scenario, describing potential projects to work on to test the strength of the sunk cost effects across domains. | Chi-square; difference between sunk time and no sunk time conditions in rate of choosing a project | Original | ϕc = .02 [.00, .18] | signal – inconsistent, positive | |
| Replication | ϕc = .32 [.23, .42] | |||||
| Chi-square; difference between sunk money and no sunk money conditions in rate of choosing a project | Original | ϕc = .32 [.12, .52] | signal – consistent | |||
| Replication | ϕc = .23 [.14, .33] | |||||
| Hypothesis 2a: If the absence of a sunk time cost effect is due to difficulties associated with the accounting of time, then the facilitation of accounting should cause the effect to reappear. [Alternative hypothesis] Hypothesis 2b: If the absence of a sunk time cost effect is due to the fact that individuals behave rationally when evaluating past time investments, then the facilitation of accounting should not cause the effect to reappear. [Null hypothesis] | 5 (Education and opportunity costs) | The level of opportunity cost (high/low) and education (present/absent) were manipulated to evaluate the strength of sunk cost effects. | ANOVA; opportunity cost main effect | Original | = .09 [.00, .23] | no signal – inconsistent |
| Replication | = .00 [.00, .01] | |||||
| ANOVA; education main effect | Original | = .17 [.04, .32] | no signal – inconsistent | |||
| Replication | = .00 [.00, .01] | |||||
| ANOVA; opportunity cost by education interaction | Original | = .00 [.00, .02] | no signal – consistent | |||
| Replication | = .00 [.00, .01] |
[i] a We provide additional detail regarding the calculation of effect sizes in the supplementary materials “Effect sizes calculation”.
b We classified each effect using the criteria set out by LeBel et al. (2019).
Table 2
Original versus replication methodological comparison.
| ORIGINAL | REPLICATION | REASON FOR CHANGE | |
|---|---|---|---|
| Participants | Undergraduate students from Hong Kong University of Science and Technology and University of Colorado. | Participants from CloudResearch/Amazon MTurk. | Larger more diverse sample. Addressing sample concerns and allowing for exploratory analyses comparing effects across studies. |
| Study 1, 2 and 5 were done separately with different participants. | Study 1, 2 and 5 were done in the same survey with the same participant. | ||
| Delivery | Paper questionnaires | Online questionnaire using Qualtrics | |
| Questions | The original studies did not use any comprehension checks or instructional manipulation checks. | We used comprehension and instructional manipulation checks in our replication. | To ensure that participants read and understood the materials. |
| Materials | In Study 5, a class on opportunity cost was delivered to those in the education condition. | A passage about opportunity cost along with questions about that passage as instructional manipulation checks were presented. | To adjust to an online sample, we used a passage that participants read instead of a class. |
| Scale | In Study 5 the preference scale was originally from 1 to 9 and presented as such. | We adjusted the presentation of the scale to 4/0/4 instead of 1 to 9. | Avoid biasing participants in a certain direction. |
| Order of studies | Study 1 –> Study 2 –> Study 5 | Randomized the order of studies 1 and 2 only, but not study 5. Study 5 is presented last at the end of the experiment. | To address potential impact of presentation order. |
[i] Note. Effect size for Study 1 and 2 is ϕc and for Study 5 – ; see “Effect sizes calculation” section in the supplementary materials.
Table 3
Power analysis.
| STUDY | SMALLEST EFFECT SIZE FROM ORIGINAL | POWER TO DETECT SMALLEST EFFECT SIZE FROM ORIGINAL | SMALLEST EFFECT SIZE DETECTABLE WITH 80% POWER | ||
|---|---|---|---|---|---|
| N = 515 | N = 1030 | N = 515 | N = 1030 | ||
| Study 1 | .61 | 99%+ | 99%+ | .12 | .09 |
| Study 2 | .32 | 99%+ | 99%+ | .17 | .12 |
| Study 5 | .31 | 99%+ | 99%+ | .13 | .09 |
[i] Note. Effect size for Study 1 and 2 is ϕc and for Study 5 – ; see “Effect sizes calculation” section in the supplementary materials.
Table 4
Comparison of the Soman’s (2001) and the current sample.
| SOMAN (2001) | REPLICATION | |
|---|---|---|
| Sample size | Study 1: 122 Study 2: 206 Study 5: 72 | 821 |
| Geographic origin | Study 1: Hong Kong Study 2: US American Study 5: US American | US American Amazon Mechanical Turk workers |
| Gender | Undisclosed | 427 males, 386 females, 8 other/did not disclose |
| Median age (years) | Undisclosed | 42 |
| Average age (years) | Undisclosed | 44.03 |
| Standard deviation age (years) | Undisclosed | 12.79 |
| Age range (years) | Undisclosed | 20-82 |
| Medium (location) | Study 1: Physical survey Study 2: Physical survey Study 5: Physical survey | Computer (online) |
| Compensation | Study 1: Credit Study 2: Undisclosed Study 5: Undisclosed | Nominal payment |
| Year | 2001 | 2023 |
| Sample source | Undergraduate students | General population |
Figure 1
Summary of results comparing Soman’s original studies to the current replication effort.
Note. Bold indicates statistically significant correlations.
Table 5
Results of linear (DV: Preference) and generalized linear (DV: Binary choice) models from the additional within-subjects analysis.
| PREDICTORS | PREFERENCE | TWO-ALTERNATIVE FORCED CHOICE | ||||
|---|---|---|---|---|---|---|
| ESTIMATE | 95% CI | p | ODDS RATIO | 95% CI | p | |
| Opportunity cost | .09 | –.46, .63 | .758 | .97 | .38, 2.46 | .951 |
| Study | .15 | –.23, .54 | .437 | 1.37 | .75, 2.57 | .311 |
| Education | .22 | –.17, .60 | .273 | 1.21 | .64, 2.30 | .552 |
| Opportunity cost × study | –.46 | –1.23, .31 | .240 | .65 | .19, 2.23 | .497 |
| Opportunity cost × education | .48 | –.30, 1.25 | .228 | 2.00 | .56, 7.23 | .285 |
| Study × education | .08 | –.47, .63 | .770 | 1.04 | .45, 2.43 | .922 |
| Opportunity cost × study × education | .31 | –.79, 1.41 | .578 | 1.67 | .31, 9.11 | .551 |
Figure 2
Study 2: Predicted probabilities from logistic regression analyses.
Note. The main effect of sunk domain is plotted based on predicted probabilities from the sunk cost present condition, while the main effect of sunk presence is plotted using marginal standardization across levels of sunk domain.
Table 6
Summary of additional analyses.
| HYPOTHESIS | QUESTION ADDRESSED | ANALYSIS USED | CONSISTENT WITH REPLICATION ANALYSIS (YES/MIXED/NO) | DETAILS |
|---|---|---|---|---|
| The sunk cost effect is weaker for time than for money. | Does the likelihood of picking the option associated with sunk costs (rocket engine in Study 2) vary significantly between levels of one independent variables (sunk cost presence or sunk domain) given a change in the other (i.e., an interaction effect)? | 2x2 logistic regression on both Soman’s original data as well as the replication data. | Yes | Both re-analyses and replication analyses are not in-line with the hypothesis: the replication analyses showed comparable to larger effect size for time than for money, whereas the reanalyses show no support for domain differences. |
| Facilitation of moneylike accounting by using education about economic approaches to time strengthens the sunk cost effect of time (tested only in the time domain). | What are the differences between Study 1 and the high versus low opportunity cost conditions in Study 5 (i.e., study by opportunity cost interaction, in the no education condition in Study 5)? | Two linear models: one linear model with preference ratings as the dependent variable and one generalized LM with 2-alternative ticket choice as the dependent variable. The models included three independent variables: study (Study 1 vs Study 5), opportunity cost (low vs high), education (no education vs education), and all their interactions. | Yes | Both no support for interactions show that, at least in the time domain, neither the opportunity cost, nor the education manipulations, made a difference. Although this is aligned with the replication analyses in our sample, it is not in-line with Soman’s (2001) conclusion. |
| Are differences between Study 1 and the high versus low opportunity cost conditions in Study 5 affected by education (study by opportunity cost by education interaction)? | Yes |
Table 7
All analyses re-run, split by whether Study 1 or Study 2 was presented first.
| ANALYSIS | STATISTICAL TEST AND FACTORS | FULL SAMPLE N = 821 | STUDY 1 FIRST N = 393 | STUDY 2 FIRST N = 428 | ||||
|---|---|---|---|---|---|---|---|---|
| ES | p | ES | p | ES | p | |||
| Replication analyses | ||||||||
| Study 1: Forced choice | Chi-square | .38 | < .001 | .41 | < .001 | .36 | < .001 | |
| Study 1: Preference | Independent samples t-test | –.79 | < .001 | –.74 | < .001 | –.84 | < .001 | |
| Study 2: Time domain | Chi-square | .32 | < .001 | .36 | < .001 | .29 | < .001 | |
| Study 2: Money domain | Chi-square | .23 | < .001 | .28 | < .001 | .19 | < .001 | |
| Study 5: Preference | between-groups ANOVA | opportunity cost | .00 | .284 | .01 | .118 | .00 | .969 |
| education | .00 | .150 | .01 | .058 | .00 | .850 | ||
| opportunity cost × education | .00 | .308 | .00 | .359 | .00 | .527 | ||
| Study 5: Forced choice | Generalized Linear Model | opportunity cost | .97 | .951 | 1.40 | .619 | .68 | .570 |
| education | 1.21 | .552 | 1.75 | .207 | .82 | .693 | ||
| opportunity cost × education | 2.00 | .285 | 1.22 | .825 | 3.69 | .185 | ||
| Additional analyses and checks | ||||||||
| Study 2 re-analysis | Logistic Regression | sunk domain | 1.17 | .414 | 1.01 | .960 | 1.35 | .282 |
| sunk presence | .20 | < .001 | .15 | < .001 | .26 | < .001 | ||
| sunk type × sunk presence | .53 | .142 | .49 | .291 | .55 | .297 | ||
| Study 1 versus Study 5: Analysis of within subject effects | Linear model | opportunity cost | .09 | .758 | .35 | .370 | –.23 | .566 |
| study | .15 | .437 | .03 | .901 | .29 | .308 | ||
| education | .22 | .273 | .44 | .118 | .05 | .852 | ||
| opportunity cost × study | –.46 | .240 | –.38 | .486 | –.54 | .342 | ||
| opportunity cost × education | .48 | .228 | .42 | .457 | .65 | .234 | ||
| study × education | .08 | .770 | .33 | .412 | –.18 | .647 | ||
| opportunity cost × study × education | .31 | .578 | –.01 | .986 | .62 | .424 | ||
| Generalized Linear Model | opportunity cost | .97 | .951 | 1.40 | .619 | .68 | .570 | |
| study | 1.37 | .311 | 1.23 | .650 | 1.43 | .436 | ||
| education | 1.21 | .552 | 1.75 | .207 | .82 | .693 | ||
| opportunity cost × study | .65 | .497 | .93 | .939 | .49 | .436 | ||
| opportunity cost × education | 2.00 | .285 | 1.22 | .825 | 3.69 | .185 | ||
| study × education | 1.04 | .922 | 1.24 | .722 | .94 | .928 | ||
| opportunity cost × study × education | 1.67 | .551 | 1.29 | .832 | 2.02 | .597 | ||
[i] Note. Reported effect sizes (ES) are: Chi-square – ϕc, Independent samples t-test – Cohen’s d, ANOVA –, Generalized Linear Model and Logistic Regression – Odds Ratios, Linear model – β.
Figure 3
Percent of incorrect responses on comprehension questions across the entire sample.
| ROLE | NIKOLAY PETROV | YIN KAN CHAN, CHEUK NAM LAU, TIN HO KWOK. LOK CHING CHOW, WAI YAN LO | WENKAI SONG | GILAD FELDMAN |
|---|---|---|---|---|
| Conceptualization | X | X | ||
| Data curation | X | X | X | |
| Formal analysis | X | X | ||
| Funding acquisition | X | |||
| Investigation | X | X | ||
| Preregistration verification | X | X | ||
| Data analysis verification | X | |||
| Methodology | X | X | ||
| Project administration | X | |||
| Resources | X | X | X | |
| Software | X | X | ||
| Supervision | X | |||
| Validation | X | |||
| Visualization | X | X | ||
| Writing – original draft | X | X | ||
| Writing – review and editing | X | X | X |