Figure 1
Hypotheses. (a) A computational cycle of active inference (black) and potential perturbations at different stages in the cycle (red). These perturbations can give rise to hopelessness – a belief that any taken action will lead to undesired states – and an increased influence of Pavlovian relative to instrumental modes of behavior (teal), both of which are associated with suicidality. (b) The brain network that we hypothesize to support the proposed perturbations: norepinephrine modulates belief updates (blue) while serotonin is involved in mediating the effects of stressor controllability (pink). Acute stress leads to increases in the learning rate, which is associated with Amy-LC connectivity (Uematsu et al., 2017; Jacobs et al., 2020), whereas environmental volatility – here assuming state-action prediction errors (SAPEs) as a proxy for environmental change – drives decay of previously learned associations and is mediated by dPFC-LC connectivity (Sales et al., 2019; Clewett et al., 2014). LC projections to the ACC mediate action-dependent state transition belief updates (Tervo et al., 2014; Sales et al., 2019), which are encoded in the ACC (Akam et al., 2021; Holroyd and Yeung, 2012). Finally, controllability of aversive outcomes, which depends on the inferred probabilities of achieving the desired outcomes, reduces aversiveness by inhibiting amygdala activation via the vmPFC-DRN-Amy circuit (Maier and Seligman, 2016; Kerr et al., 2012).
Figure 2
Avoid/Escape Go/No-Go task design and model specification. (a) Following Millner et al. (2018; 2019) the task has 4 cues corresponding to the 2 × 2 (Go/No-Go x Avoid/Escape) factorial task structure, with 2 possible outcomes: aversive or neutral. For modelling purposes, the task was divided into 3 discrete time points. At the start of a trial (t = 1) the agent is in one of the four hidden states (s1–4) and no observations are available (o1). Next, the agent is taken to t = 2, where a cue (and in the case of Escape condition also an aversive sound) is presented corresponding to one of four possible hidden states (s5–8) and observations (o2–5). At t = 2 the agent chooses what action to take (Go or No-Go) which then leads to one of four possible states (s9–12) and observations (o6–9): Go response + silence (s9, o6), Go response + aversive sound (s10, o7), No-Go response + silence (s11, o8), No-Go response + aversive sound (s12, o9). (b) The main model structures. The likelihood of observations, A, was implemented to have deterministic mappings between states and observations due to the salience of the aversive stimulus and the cues. As no learning was required, A in the generative model and in the generative process were identical. State transitions from t = 2 to t = 3 for instrumental (Go/No-Go) policies B were probabilistic, captured by the y parameter. For the objective transition probabilities y was set to 0.8, meaning that correct response by the agent led to the neutral state 80% of the time. For the generative model, y was initialized with 0.5 to correspond to the agent having a uniform prior over the two possible transitions. The zero probabilities for the other transitions reflect the assumption that the agent understands the task structure and does not expect to end up in a Go state after choosing No-Go and vice versa. State transition probabilities from t = 2 to t = 3 for the Pavlovian policy, B0, were implemented to allow only for No-Go responses in the Avoid and Go responses in the Escape conditions (with Go responses in the Avoid and No-Go responses in the Escape conditions having 0 probabilities). The strength of the belief that the Pavlovian policy will lead to the desired states is captured by the z parameter. Prior over outcomes (C) assumed that the agent does not like outcomes 4, 5, 7 and 9 (all of which involve the aversive stimulus). The strength of this preference of neutral outcomes is captured by parameter c. The prior over initial states D was assumed to be uniform for states 1–4. The other states have zero probability, which reflects the assumption that the agent understands the task structure and does not expect to be in states 5–12 at the beginning of a trial. Finally, prior over policies E was also assumed to be uniform across the available Go, No-Go and Pavlovian policies (π). See S1 Appendix: full mathematical details of the model for more implementation details.
Figure 3
Summary of the proposed computations, possible neural correlates and parameters of interest for STB. Within the proposed model there are four areas of relevance for STB: learning rate, belief decay rate, stress reactivity and perceived controllability of a stressor. Stress weight parameter, k, controls the boost in the learning rate in response to stress. Increasing this parameter would result in increased learning from stressful outcomes. Stress sensitivity parameter, c, captures individual sensitivity to stress, which then also affects the learning rate. Controllability threshold, w0, is a midpoint in the logistic function that translates the beliefs about state transitions into an estimate of stressor controllability. In other words, w0 regulates how positive state transition beliefs have to be for a stressor to be deemed sufficiently controllable. Finally, belief decay threshold, m, regulates how large state-action prediction errors (SAPEs) have to be before significant belief decay takes place. Note that for the decay rate and the controllability there are other parameters (gradients, gw, g, and minimum and maximum decay values λmin, λmax) that we could inspect, but for simplicity here we focus on the midpoint values w0 and m as the exact parameterization of these effects is somewhat arbitrary and the midpoints are sufficient for exploring the general direction of different manipulations.
Figure 4
Model simulations: a single (healthy control) participant with low stress weight (k = 0.1). (a-c) average choice accuracy before reversal, after reversal and overall, respectively, for Go-to-Avoid (GA), No-Go-to-Avoid (NGA), Go-to-Escape (GE) and No-Go-to-Escape (NGE); the four colors denote different cues used in the task. The results in (a) reproduce active-escape bias reported by Millner et al. (2018) in the general population. (b) and (c) are additional predictions about performance after reversal and overall, respectively. (d) Decay parameter values for different SAPEs throughout the task. Note that SAPEs for aversive outcomes are larger which leads to smaller decay parameter, and thus to larger belief decay (see Eq. (1)). (e) Performance across all trials. The top 3-row panel shows the sequence of cue presentation (middle row), executed action (non-grey squares: bottom row – No-Go, top row – Go) and trial outcome (white – neutral, black – aversive); each column corresponds to a single trial. Actions are represented implicitly by either black or white color. If for a given trial the top square is either black or white, it means that the Go action was selected, if the bottom square is either black or white then the No-Go action was selected. The main panel shows trajectories of correct action probabilities, which gradually increase as the task progresses, but drop sharply once the Go/No-Go cue meanings are reversed on trial 100. The response to this environmental change can be seen in the decreased decay parameter (black line), which drives faster forgetting of previously learned contingencies and allows the agent to adapt. Note that decay parameter trajectory here is scaled to be between 0 and 1 and smoothed out using moving average with a window size of 5 trials. (f-i) Trajectories of underlying beliefs about state transitions and policy probabilities. These plots reflect the straightforward relationship between belief strength and policy probability: as the probability of an instrumental Go/No-Go action leading to the desired state increases (solid/dash-dotted colored lines) the probability of choosing Go/No-Go policy tracks that increase (solid/dash-dotted gray), and probabilities of Pavlovian policies (solid black) decrease as a result. The vertical dashed lines in all of the plots denote the reversal.
Figure 5
Model simulations: a single (STB) participant with a high stress weight (k = 1). (a-c) average choice accuracy before reversal, after reversal and overall, respectively, for Go-to-Avoid (GA), No-Go-to-Avoid (NGA), Go-to-Escape (GE) and No-Go-to-Escape (NGE); the four colors denote different cues used in the task. The results in (a) reproduce increased active-escape bias in suicidality reported by Millner et al. (2019), and predict that this bias would be even larger after a reversal in cue meanings (b panel). (d) Decay parameter values for different SAPEs throughout the task. Note that now aversive outcomes produce smaller SAPEs, due to increased expectation of aversive states. (e) Performance across all trials. The top 3-row panel shows the sequence of cue presentation (middle row), executed action (non-grey squares: bottom row – No-Go, top row – Go) and trial outcome (white – neutral, black – aversive); each column corresponds to a single trial. Actions are represented implicitly by either black or white color. If for a given trial the top square is either black or white, it means that the Go action was selected, if the bottom square is either black or white then the No-Go action was selected. The main panel shows trajectories of correct action probabilities. Compared to the healthy control in the previous figure, the trajectories are noisier, especially after the reversal on trial 100. Decay rate trajectory (black line) is also nosier, which is partly responsible for the poor adaptation after the reversal. Note that decay parameter trajectory here is scaled to be between 0 and 1 and smoothed out using moving average with a window size of 5 trials. (f-i) Trajectories of underlying beliefs about state transitions and policy probabilities. Compared to the healthy control, the belief trajectories are noisier, but even more importantly, beliefs about the instrumental transitions to neutral states are on average weaker (cf. hopelessness), which leads to increased probability of the Pavlovian policy. The vertical dashed lines in all of the plots denote the reversal.
Figure 6
Model simulations: exploration of the hypothesis space. Each column shows the effects of varying on of the parameters: k – stress weight (while m = 1.3, c = 8, w0 = 0.6), m – belief decay threshold (while k = 0.7, c = 8, w0 = 0.6), c – stress sensitivity (while k = 0.6 m = 1, w0 = 0.5) and w0 – controllability threshold (while k = 0.9, m = 1.3, c = 8). (a-d) the mean of beliefs that the neutral state will be reached averaged across 4 contexts and 2 possible actions. (e-h) The mean probability of choosing the Pavlovian policy. (i-l) Active-escape bias (the difference between choice accuracy on GE and NGE trials). The solid and dashed red lines denote the expected active-escape bias in healthy control group and suicidality group, respectively (based on Millner et al. (2018; 2019) findings). (m-p) Mean choice accuracy across all 4 contexts.
Figure 7
Model simulations: trajectories of beliefs and policies under different parameter manipulations. (a) low belief decay, m = 2, (b) low controllability, w0 = 0.8, (c) high stress weight, k = 1.1, (d) high stress sensitivity, c = 20. The other parameters were set to the same values as in Figure 6. All panels show trajectories of NGE/NE cue: where the cue is NGE before the reversal (the vertical dashed line) and GE after the reversal. Less variable rigid negative beliefs and Pavlovian policy in (a) could be associated with planful suicide attempts, whereas more variable beliefs and sudden increases in Pavlovian policy in (b-d) could be associated with more impulsive suicide attempts (Schmaal et al., 2020; Bernanke et al., 2017).