Reliability of Decision-Making and Reinforcement Learning Computational Parameters

Participants

Fifty-four healthy participants were recruited from the UCL Institute of Cognitive Neuroscience Subject Database. Four participants were excluded for failing to complete the second session (final N = 50: 32 females [64%]; age range = 19–38; mean age = 25.16, SD ± 5.48 years; mean education = 17.38, SD = ±3.24 years). Participants reported no current or past psychiatric or neurological disorder, cannabis use in the past 31 days, alcohol consumption in the past 24 hours, or any other recreational drug use in the week prior to participation. Participants provided written informed consent and were compensated at the end of their second session with a flat rate of £30 and a bonus of up to £20 based on task winnings. The study was approved by the UCL Psychology and Language Sciences Research Ethics Committee (Project ID Number: fMRI/2013/005) and was performed in accordance with the Declaration of Helsinki.

Sample size was determined by an a priori power analysis in G*Power (Faul et al., 2007). The power analysis was based on the smallest effect size of interest, r = 0.4, since reliability below this threshold is conventionally considered poor (Fleiss, 2011). Detecting an effect size of this magnitude, at the one-tailed 0.05 alpha level with 90% power, requires 47 participants.

Study procedure and tasks

Participants completed a battery of computerized tasks, including a restless four-armed bandit (Daw et al., 2006; Seymour et al., 2012) and an individually calibrated gambling task (Charpentier et al., 2017) over two sessions (mean test-retest interval = 13.96 days, SD = 0.20). On each trial in the bandit task, participants chose one out of four bandits and received one out of four possible outcomes: reward, punishment, neither reward nor punishment or both reward and punishment (200 trials total). Win and loss probabilities fluctuated independently over time and between boxes (Figure 1a–b; Supplemental Materials). In the gambling task, participants chose between a 50–50 gamble and a sure option without receiving feedback. Trials were classified as either mixed (50% chance to win or lose money gamble, or a sure option of 0 points) or gain-only (50% chance to win or receive nothing gamble, or a variable sure gain; Figure 1c; Supplemental Materials). An initial training phase (50 mixed and 40 gain-only trials) was used to create individually calibrated offers (centred on indifference points) in a second phase (64 mixed and 56 gain-only trials). Calibration failed for one participant resulting in N = 49 participants for this task. Both tasks lasted around 15 minutes.

Figure 1

Data analysis

Data were processed in Matlab (R2019b) and analysed in SPSS (v25, IBM Corp, Armonk, NY) and R (v. 4.2.1). Computational modelling was performed with the hBayesDM package for R (v. 3.6.0; https://github.com/CCS-Lab/hBayesDM) (Ahn et al., 2017), which uses hierarchical Bayesian modelling in Stan (v.2.21.2). For all analyses, p < 0.05 (two-tailed) was considered statistically significant. Cohen’s d_z (within-subject) effect sizes are reported for practice effects (Lakens, 2013).

Model-agnostic task analyses

Model-agnostic measures for the bandit task included the mean probability of repeating a choice after win-only, loss-only and no outcomes (‘neither’). A repeated-measures analysis of variance (ANOVA) was conducted with the within-subjects factors outcome (win, loss, neither) and session (session 1, session 2) to assess basic behaviour and practice effects.

Model-agnostic measures for the gambling task included the mean probability of gambling on mixed and gain-only trials. It was predicted that gambling would be higher on mixed trials, which was assessed using a repeated-measures ANOVA with within-subjects factors gamble (mixed, gain-only) and session (session 1, session 2).

As a supplementary analysis, bandit and gamble model-agnostic measures were additionally derived from trial-by-trial mixed-effects logistic regressions for each session since previous studies have suggested that hierarchically predicted values can improve the precision of reliability (Brown et al 2020; Waltmann et al., 2022). Further details on all model-agnostic measures are in Supplemental Materials.

Computational modelling

The bandit task data was fit with seven different RL models previously described in detail (Aylward et al., 2019; Supplemental Materials). Three PT models were fit to the gambling task (Ahn et al., 2017; Charpentier et al., 2017; Kahneman & Tversky, 1979; Sokol-Hessner et al., 2009). Modelling was conducted on the second phase of the gambling task (i.e., on individually calibrated trials). The models were fit for each session separately, using separate hierarchical priors (group-level parameters), as this has shown to provide more accurate fits (Valton et al., 2020), and we wished to avoid artificially inflating reliability estimates. We also estimated model fits under a single hierarchical prior (session 1 and session 2 data together) as a sensitivity analysis (Supplemental Materials). Model comparison was performed with leave-one-out information criterion (LOOIC) where the winning model was the one with the lowest LOOIC. Several model validation checks were completed for the winning models, including examination of MCMC convergences, parameter recovery and recapitulation of real data (Daw, 2011; Kruschke, 2015; Wilson & Collins, 2019; Supplemental Materials; Figure S1-S8).

Reliability analysis

Test-retest reliability was assessed with ICCs (ratios of intra-individual to inter-individual variability (Koo & Li, 2016; McGraw & Wong, 1996)), with values of <0.40 interpreted as poor, 0.4–0.6 as fair, 0.6–0.75 as good, and >0.75 as excellent reliability (Fleiss, 2011). A two-way mixed-effects model based on single-measures and absolute-agreement ICC was used (fixed effect: testing time-interval, random effect: subject), equivalent to ICC(A,1) (McGraw & Wong, 1996).

We also specified an additional model that considers data from both sessions jointly, taking the hierarchical structure into account and estimating reliability directly within the model, which has recently been shown to improve the estimation of reliability (Brown et al 2020; Haines et al., 2020; Waltmann et al., 2022). For model-agnostic measures, this was achieved by estimating a single mixed-effects logistic regression with random intercepts and slopes, accounting data for both sessions jointly by including session as a second-level grouping factor with subjects nested within session (Brown et al 2020; Waltmann et al., 2022). This allows extracting variance components from the logistic regression to calculate a one-way random-effects, absolute-agreement, single-measure ICC: ICC(1) in the McGraw and Wong (1996) convention. This was calculated for all bandit and gamble model-agnostic measures. For computational parameters, model-calculated Pearson’s r correlations were estimated between parameters from session 1 and 2 by fitting all the data from both sessions together and embedding a correlation matrix between sessions in the winning RL and PT models, using an identical approach described in detail previously (Haines et al., 2020; Pike et al., 2022). This essentially results in a multivariate prior where the correlation between group-level parameters across session are also considered and the uncertainty of point estimates are accounted for in the model-calculated reliability estimate. Separate subject-specific parameters are estimated for each session as well, but these were not extracted to calculate reliability on as previous studies have suggested that this overestimates reliability (Waltmann et al., 2022).

Posterior predictive performance

To assess to what extent an individual’s future behaviour can be predicted using a generative model fit to their own task performance two weeks earlier, we calculated the probability of participants’ choices on each trial (i.e., the softmax output), given their session 2 data and model parameter estimates from session 1. Probabilities were averaged across trials for each individual.

Since hierarchical parameter estimation produces ‘shrinkage’, effectively pulling parameter estimates from different individuals closer to each other (which improves estimation accuracy), it is possible that future performance may also be predicted above-chance using other participants’ parameter estimates from session 1 (e.g., participant A’s parameter estimates from session 1 predicting participant B’s session 2 choices). We therefore assessed whether using an individual’s model parameter estimates from session 1 predicted the same individual’s choices on session 2 better than using all other subjects’ model parameter estimates. To construct the latter measure, for each subject, we predicted trial-by-trial choices on session 2 based on parameter estimates from every other participant’s session 1 model, and averaged the probabilities across all participants. Additionally, we compared subjects’ own session 1 parameters in predicting their session 2 behaviour to the mean session 1 model parameter priors in predicting future behaviour.

Data accessibility

All script code and data are available on OSF at https://osf.io/n7czx/.

Four-armed bandit task: model-agnostic results

Basic behaviour and practice effects

As expected, there was a main effect of outcome type on behaviour (F_(2,98) = 117.39, p < 0.001, ${\eta }_p^2=0.71$; Figure 2a). The probability to repeat a choice was significantly greater after wins compared with both losses and outcomes on which neither wins nor losses occurred, and greater after neither compared with losses (all p < 0.001). There was no significant main effect of testing session (F_{(1, 49)} = 0.01, p = 0.91, ${\eta }_p^2 <0.001$). There was a significant outcome-by-session interaction (F_(2,98) = 3.12, p = 0.049, ${\eta }_p^2 =0.06$), reflecting slightly increased repeated choices after wins and decreased repeated choices after losses. However, the difference in the tendency to repeat a choice between session 1 and session 2 did not reach significance following any of the outcome types (loss: t₍₄₉₎ = 1.45, p = 0.15, d_z = 0.21; win: t₍₄₉₎ = 0.87, p = 0.39, d_z = 0.12; neither: t₍₄₉₎ = 0.54, p = 0.59, d_z = 0.08), and therefore we do not interpret this result further.

Figure 2

Four-armed bandit task: modelling results

Model comparison indicated that the winning (most parsimonious) model was the five-parameter ‘Bandit4arm_lapse’ (nomenclature from the hBayesDM package) model, with reward and punishment learning rate parameters, reward and punishment sensitivity parameters and a lapse parameter (final parameter captures random responding; Table S2), consistent with previous reports (Aylward et al., 2019). Three individuals were excluded due to difficulties in obtaining mean parameter estimates, as multiple peaks were evident in the posterior distribution of at least one parameter (Figure S2). The Bandit4arm_lapse model was therefore re-fit without these participants. Excluding these participants did not affect test-retest reliability inference.

Practice effects

On session 2 there were significant increases in the reward sensitivity (t₍₄₆₎ = 3.00, p = 0.004, d_z = 0.44) and lapse parameters (t₍₄₆₎ = 8.88, p < 0.001, d_z = 1.29), but not on any of the other parameters (reward learning rate: t₍₄₆₎ = 1.28, p = 0.21, d_z = 0.19; punishment learning rate: t₍₄₆₎ = 1.74, p = 0.09, d_z = 0.25; punishment sensitivity: t₍₄₆₎ = 1.28, p = 0.21, d_z = 0.19; Figure 3a). However, there were no significant practice effects when the data was fit under a single hierarchical prior (Table S3).

Figure 3

Posterior predictive performance

Parameter estimates from session 1 predicted task performance on session 2 substantially better than chance (mean = 42%, chance = 25% accuracy; t₍₄₆₎ = 9.10, p < 0.001; Figure 4a), indicating that the model could predict future choices by using a generative model fit to the same participants’ data two weeks earlier. Using an individual’s parameter estimates to predict their own future choices was significantly better than when that prediction was based on the average of the other participants’ session 1 estimates (t₍₄₆₎ = 3.20, p = 0.003; Figure 4b). However, there was no significant difference between using an individual’s own session 1 parameter estimates compared with the session 1 mean prior parameter in predicting future behaviour (t₍₄₆₎ = 1.04, p = 0.30; Figure 4c).

Figure 4

Gambling task: model-agnostic results

Basic behaviour and practice effects

As expected, propensity to gamble was significantly higher on mixed gambles (F_{(1, 48)} = 13.71, p = 0.001, ${\eta }_p^2 =0.22$). There were no significant main (F_{(1, 48)} = 0.76, p = 0.40, ${\eta }_p^2 =0.02$) or interaction (F_{(1, 48)} = 1.07, p = 0.31, ${\eta }_p^2 =0.02$) effects of session on the propensity to gamble (session differences: probability to gamble on mixed trials t₍₄₈₎ = 0.23, p = 0.82, d_z = 0.03; probability to gamble on gain-only trials t₍₄₈₎ = 1.51, p = 0.14, d_z = 0.22; Figure 5a).

Figure 5

Test-retest reliability

Model-agnostic measures on the gambling task exhibited fair-to-good reliability (Figure 5b; Table 2) and did not change substantially when examined hierarchically in separate mixed logistic regressions (Table S4). Calculating reliability within a joint mixed logistic regression numerically improved the reliability of the gambling model-agnostic measures (Table 2).

Table 2

Reliability of model-agnostic and computational measures of the gambling task. All measures are significant at p < 0.05. Brackets represent the 95% confidence interval.

MODEL-AGNOSTIC P(GAMBLE) MEASURES	ICC(A,1)	ICC(1)	PEARSON’S R
Summary statistics (Figure 5)
Mixed trials	0.63 (0.43–0.78)	0.63 (0.43–0.78)	0.63 (0.42–0.77)
Gain-only trials	0.59 (0.38–0.75)	0.59 (0.38–0.75)	0.60 (0.39–0.76)
Model-calculated reliability from joint hierarchical logistic regression
Mixed trials		0.73
Gain-only trials		0.72
PROSPECT THEORY MODEL	ICC(A,1)	ICC(1)	PEARSON’S R
Model estimated separately per session (Figure 6)
Loss aversion	0.68 (0.50–0.81)	0.68 (0.50–0.81)	0.72 (0.55–0.83)
Risk aversion	0.78 (0.55–0.89)	0.78 (0.64–0.87)	0.83 (0.71–0.90)
Inverse Temperature	0.80 (0.64–0.89)	0.80 (0.67–0.88)	0.84 (0.74–0.91)
Model-calculated reliability from joint hierarchical Bayesian model
Loss aversion			0.87 (0.77–0.94)
Risk aversion			0.90 (0.83–0.95)
Inverse Temperature			0.91 (0.85–0.96)

Gambling task: modelling results

The winning model was the PT model (‘ra_prospect’ in the hBayesDM package) with loss aversion, risk aversion and inverse temperature parameters (last parameter represents choice consistency; Table S5), consistent with previous reports (Charpentier et al., 2017). A loss aversion parameter above 1 represents overweighting of losses to gains, while a risk aversion parameter less than 1 indicates aversion to risk. Neither test-retest nor practice effects were substantially altered when the model was fit under a single hierarchical prior (Table S6).

Practice effects

There were significant session effects on all PT model parameters (on session 2: decreased loss aversion: t₍₄₈₎ = 2.17, p = 0.04, d_z = 0.31; decreased risk aversion: t₍₄₈₎ = 4.04, p < 0.001, d_z = 0.58; increased inverse temperature: t₍₄₈₎ = 3.07, p = 0.004, d_z = 0.44; Figure 6a).

Figure 6

Posterior predictive performance

PT model parameters from session 1 predicted future choices at session 2 significantly above chance (mean = 68%, chance = 50% accuracy; t₍₄₈₎ = 12.08, p < 0.001; Figure 7a). Predicting future performance at session 2 was significantly higher when based on participants’ own parameter estimates from session 1 compared with model parameter estimates of other participants from session 1 (t₍₄₈₎ = 8.38, p < 0.001; Figure 7b). This was also true when comparing between participants’ own session 1 parameter estimates with the mean session 1 prior for parameters (t₍₄₈₎ = 6.28, p < 0.001; Figure 7c).

Figure 7

Four-armed bandit RL model reliability

Reward and punishment learning rates from the bandit task demonstrated good reliability while reward and punishment sensitivity showed fair reliability, suggesting that this task may be more suitable for assessing learning rates than sensitivity. Elevated punishment learning rates (faster learning in the face of negative outcomes) and lapse values have been associated with greater mood and anxiety symptoms, representing potential measurable mechanistic treatment targets (Aylward et al., 2019). However, the present study suggests that the lapse parameter, which exhibited poor reliability as assessed by the bandit task, may not be a suitable target. This parameter measures responding not captured by the model (including goal-directed and random exploration), and the sources of this ‘noise’ might differ across sessions. It is therefore perhaps unsurprising that this parameter was unreliable. Crucially, the lapse parameter showed poor recoverability, which places an upper limit on its potential reliability. Some of this poor recoverability may be explained by limited lapse variation, especially in session 1 (Figure S3). The distribution of the group-level standard deviation lapse parameter was biased towards smaller values here such that the lapse parameter did not vary greatly between individuals (Figure S5). This suggests that the lapse parameter could be replaced with a constant and inference on this parameter is not advised.

Although no prior studies have specifically investigated ICC properties of the current RL model, one previous study found similarly poor reliability of the lapse parameter across six months in a go/no-go RL model in adolescents (Moutoussis et al., 2018). In contrast to our results, this study also reported poor reliability of both reward and punishment learning rates. These differences may arise for a multitude of reason, such as using different tasks (an orthogonalised go/no-go task versus a restless bandit task), testing time-windows (six months versus two weeks), populations (adolescents versus adults), or models. It is not possible to delineate these diverging results without systematically comparing these factors in one study. Importantly, however, we provide evidence that it is possible to achieve at least moderate reliability for some canonical RL parameters.

Interestingly, the model-agnostic outcome measures of the bandit task exhibited similar reliability to the computational measures. Model-agnostic measures of cognitive tasks have often been reported to exhibit poor-to-moderate reliability (Enkavi et al., 2019; Hedge et al., 2018; Plichta et al., 2012; Rodebaugh et al., 2016). It has been argued that this may be due to their inability to capture the generative process underlying task performance (Huys et al., 2021; Price et al., 2019). Our results suggest that it should not be assumed that computational parameters will always provide greater reliability than non-computational ones. However, the model-agnostic outcome measures are only a proxy of the processes the bandit task assesses, as it is difficult to compute model-agnostic equivalents of some parameters, such as reward/punishment sensitivity. Indeed, models make explicit and falsifiable predictions of the components driving behaviour, which can be refined and used to simulate artificial data to generate new predictions. Thus, computational modelling is a more rigorous and preferable method for assessing behaviour than model-agnostic measures, which unlike computational methods, lack the mechanistic insights into the underlying processes generating behaviour.

Gambling PT model reliability

While reliability of parameters ranged from poor-to-good in the bandit task, parameters from the gambling task showed good-to-excellent test-retest reliability. These were also higher than the reliability of the model-agnostic measures, suggesting that computational models may offer advantages in psychometric properties here. In particular, the risk aversion parameter, which has previously been associated with anxiety (Charpentier et al., 2017), exhibited excellent reliability (ICC = 0.78), providing promise for use in clinical research. These results show higher reliability than previous studies (loss aversion r ≈ 0.25–0.61, risk aversion r ≈ 0.50–0.60, inverse temperature r ≈ 0.30–0.60; Chung et al., 2017; Glockner & Pachur, 2012; Scheibehenne & Pachur, 2015). These studies all used different estimation procedures, including hierarchical Bayesian, and employed both longer and shorter testing time-windows than the current study, suggesting that these factors may not fully explain the differences. It is possible that our results instead stem from different PT model specifications, as well as different task designs. Indeed, a strength of the gambling task is that we calibrated offers to each individual’s indifference point (Charpentier et al., 2017). A similar approach of dynamically updating parameter values to each individual during task performance has previously been suggested as a solution to unreliable cognitive tasks (Palminteri & Chevallier, 2018). This may allow for removing any potential state influences across participants to extract more trait-like measures of the variables of interest (here risk/loss aversion).

Predictive accuracy

We also examined how well the models predicted future task performance, which provides complementary perspective on reliability, unique to computationally-informed measures. Notably, for the PT model participants’ own parameter estimates from the first session were better at predicting their future performance compared with using parameter estimates from all other participants and from model priors. Individuals’ own RL parameters only provided an advantage in predicting future performance when compared with other’s parameter estimates but not model priors. It is likely that the RL model did not perform as well on this metric as the PT model because the RL model does not provide as close a fit to behaviour, potentially due to some participants performing at chance level. Overall, this indicates that individuals may also reliably differ in the cognitive mechanisms underlying their decisions, and offers reassurance that hierarchical estimation procedures are suitable for inter-individual inferences (Brown et al., 2020; Daw, 2011; Scheibehenne & Pachur, 2015). In other words, individuals show relatively unique computational decision-making profiles, particularly in the PT model. This is consistent with two previous studies using a different PT model and gambling task (Glockner & Pachur, 2012; Scheibehenne & Pachur, 2015).

Implications

The RL parameters showed relatively modest reliability, suggesting that these processes are more vulnerable to state influences or measured with less precision than PT parameters. A consequence of this is that larger sample sizes may be required to examine effects, as effect sizes would be expected to be lower, relative to PT tasks (Fleiss, 2011). Interpreting the marked difference in reliability between the PT and RL models is not straightforward, as these models measure complementary aspects of cognition. The bandit task is a learning paradigm that requires constant updating of optimal choices. It is possible that in the first session individuals had not yet stabilized on a cognitive strategy and were still learning the task structure, as indicated by lower evidence of the winning model in session 1 compared with session 2 (Table S2). It would be interesting to explore if an initial baseline session, would improve test-retest reliability.

It should also be noted that we observed substantially greater reliability of RL (good-to-excellent) and PT (excellent) parameters when estimated within the joint generative model, in line with previous studies (Brown et al., 2020; Waltmann et al., 2022). Similarly, calculating the reliability of model-agnostic measures directly from a joint logistic regression improved model-agnostic reliability estimates, although in the range of good reliability. Similar to the hierarchical estimation approach of separate sessions, joint estimation regularises estimates toward the group mean for that session, improving the accuracy of point estimates. However, joint estimation additionally considers the potentially correlated structure. While this can increase the within-subject precision, it can consequently inflate the reliability of point estimates extracted from joint models (hence why reliability was not calculated on the point estimates in these models; Waltmann et al., 2022). Estimating reliability directly within the model, however, takes the uncertainty around the point estimates into account, providing unbiased reliability estimates (Waltmann et al., 2022). This suggests that the increased precision (and here thus improved reliability) may stem from the greater information (including more data points) available in joint modelling. This analysis approach may therefore be preferred when the study design allows for it.

Limitations

A potential limitation of our study is the sample tested, as the reliability of tasks in healthy individuals may differ from that in clinical groups. Similarly, our results only speak to reliability over two weeks. Thus, it is possible that longer time periods may produce lower reliability, which should be assessed in future studies. Reliability over two weeks is particularly informative for interventional studies such as randomised controlled trials (e.g., for rapid-acting antidepressants or for early markers of response for traditional antidepressants/psychotherapies). This time-window is also in line with other reliability studies aiming to establish reliability of measures for e.g., individual differences (e.g., Hedge et al., 2018; Nord et al., 2017), based on the assumption that measures should remain relatively stable over a short time-period.

Conclusion

In summary, we show that commonly-used computational parameters derived from an RL ‘restless’ bandit task and a calibrated gambling task exhibit fair-to-excellent reliability. Specifically, learning rates showed good reliability and sensitivity parameters showed fair reliability from the RL model, while loss aversion had good reliability and risk aversion and inverse temperature displayed excellent reliability from the PT model. These models can further be used to predict future behaviour in the same individuals, especially PT model parameters, indicating that the decision-making processes assessed in these tasks represent relatively consistent and unique characteristics of an individual. These findings take us one step closer to translating computational measures of behaviour into clinical application.