References
- Alloy, L. B. , Abramson, L. Y. , Walshaw, P. D. , & Neeren, A. M. (2006). Cognitive vulnerability to unipolar and bipolar mood disorders. Journal of Social and Clinical Psychology, 25(7), 726–754. https://doi.org/10.1521/jscp.2006.25.7.726
- Alloy, L. B. , Kelly, K. A. , Mineka, S. , & Clements, C. M. (1990). Comorbidity in anxiety and depressive disorders: A helplessness/hopelessness perspective. In J. D. Maser & C. R. Cloninger (Eds.), Comorbidity of anxiety and mood disorders. Washington, DC: American Psychiatric Press.
- Ambrose, R. E. , Pfeiffer, B. E. , & Foster, D. J. (2016). Reverse replay of hippocampal place cells is uniquely modulated by changing reward. Neuron, 91(5), 1124–1136. https://doi.org/10.1016/j.neuron.2016.07.047
- American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (DSM-5). Washington, DC: Author.
- Arnaudova, I. , Kindt, M. , Fanselow, M. , & Beckers, T. (2017). Pathways towards the proliferation of avoidance in anxiety and implications for treatment. Behaviour Research and Therapy, 96, 3–13. https://doi.org/10.1016/j.brat.2017.04.004
- Aupperle, R. , & Martin, P. P. (2010). Neural systems underlying approach and avoidance in anxiety disorders. Dialogues in Clinical Neuroscience, 12(4), 517–531.
- Aylward, J. , Valton, V. , Ahn, W.-Y. , Bond, R. L. , Dayan, P. , Roiser, J. P. , & Robinson, O. J. (2019). Altered learning under uncertainty in unmedicated mood and anxiety disorders. Nature Human Behaviour, 1. https://doi.org/10.1038/s41562-019-0628-0
- Bach, D. R. (2015). Anxiety-like behavioural inhibition is normative under environmental threat-reward correlations. PLoS Computational Biology, 11(12), e10046 46. https://doi.org/10.1371/journal.pcbi.1004646
- Bandura, A. , & Adams, N. E. (1977). Analysis of self-efficacy theory of behavioral change. Cognitive Therapy and Research, 1(4), 287–31 0. https://doi.org/10.1007/BF01663995
- Barlow,D. H. (2002). Anxiety and its disorders: The nature and treatment of anxiety and panic. New York, NY: Guilford Press.
- Bellemare, M. G. , Dabney, W. , & Munos, R. (2017). A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 449–458). N.P.: JMLR.
- Berenbaum, H. (2010). An initiation-termination two-phase model of worrying. Clinical Psychology Review, 30(8), 962–975. https://doi.org/10.1016/j.cpr.2010.06.011
- Browning, M. , Behrens, T. E. , Jocham, G. , O’Reilly, J. X. , & Bishop, S. J. (2015). Anxious individuals have difficulty learning the causal statistics of aversive environments. Nature Neuroscience, 18(4), 590–596. https://doi.org/10.1038/nn.3961
- Butler, G. , & Mathews, A. (1983). Cognitive processes in anxiety. Advances in Behaviour Research and Therapy, 5(1), 51–62. https://doi.org/10.1016/0146-6402(83)90015-2
- Butler, G. , & Mathews, A. (1987). Anticipatory anxiety and risk perception. Cognitive Therapy and Research, 11(5), 551–5 65. https://doi.org/10.1007/BF01183858
- Chow, Y. , Tamar, A. , Mannor, S. , & Pavone, M. (2015). Risk-sensitive and robust decision-making: A CVAR optimization approach. In Advances in neural information processing systems (1522–1530). Montreal, Quebec: NeurIPS.
- Clark,D. A. , & Beck,A. T. (2011). Cognitive therapy of anxiety disorders: Science and practice. New York, NY: Guilford Press.
- Davey, G. C. , Jubb, M. , & Cameron, C. (1996). Catastrophic worrying as a function of changes in problem-solving confidence. Cognitive Therapy and Research, 20(4), 333–344. https://doi.org/10.1007/BF02228037
- Daw, N. D. , Niv, Y. , & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12), 1704–1711. https://doi.org/10.1038/nn1560
- Dayan, P. (1993). Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4), 613–624. https://doi.org/10.1162/neco.1993.5.4.613
- Dugas, M. J. , Freeston, M. H. , & Ladouceur, R. (1997). Intolerance of uncertainty and problem orientation in worry. Cognitive Therapy and Research, 21(6), 593–606. https://doi.org/10.1023/A:1021890322153
- Dymond, S. , Dunsmoor, J. E. , Vervliet, B. , Roche, B. , & Hermans, D. (2015). Fear generalization in humans: Systematic review and implications for anxiety disorder research. Behavioral Therapy, 46(5), 561–582. https://doi.org/10.1016/j.beth.2014.10.001
- Friston, K. J. , Stephan, K. E. , Montague, R. , & Dolan, R. J. (2014). Computational psychiatry: The brain as a phantastic organ. The Lancet Psychiatry, 1(2), 148–158. https://doi.org/10.1016/S2215-0366(14)70275-5
- Fung, B. J. , Qi, S. , Hassabis, D. , Daw, N. , & Mobbs, D. (2019). Slow escape decisions are swayed by trait anxiety. Nature Human Behaviour, 1, 702–708. https://doi.org/10.1038/s41562-019-0595-5
- Gagne, C. , Dayan, P. , & Bishop, S. J. (2018). When planning to survive goes wrong: Predicting the future and replaying the past in anxiety and PTSD. Current Opinion in Behavioral Sciences, 24, 89–95 . https://doi.org/10.1016/j.cobeha.2018.03.013
- Gallagher, M. W. , Bentley, K. H. , & Barlow, D. H. (2014). Perceived control and vulnerability to anxiety disorders: A meta-analytic review. Cognitive Therapy and Research, 38(6), 571–584. https://doi.org/10.1007/s10608-014-9624-x
- Gallagher, M. W. , Naragon-Gainey, K. , & Brown, T. A. (2014). Perceived control is a transdiagnostic predictor of cognitive–behavior therapy outcome for anxiety disorders. Cognitive Therapy and Research, 38(1), 10–22. https://doi.org/10.1007/s10608-013-9587-3
- Garcia, J. , & Fernandez, F. (2015). A comprehensive survey on safe reinforcement learning. Journal of Machine Learning, 16, 1437–1480.
- Gaskett, C. (2003). Reinforcement learning under circumstances beyond its control. In Proceedings of the International Conference on Computational Intelligence, Robotics and Autonomous Systems. Washington, DC: IEEE.
- Gläscher, J. , Daw, N. , Dayan, P. , & O’Doherty, J. P. (2010). States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, (66), 4 585–595. https://doi.org/10.1016/j.neuron.2010.04.016
- Harlé, K. M. , Guo, D. , Zhang, S. , Paulus, M. P. , & Angela, J. Y. (2017). Anhedonia and anxiety underlying depressive symptomatology have distinct effects on reward-based decision-making. PLoS One, 12(10), e0186473. https://doi.org/10.1371/journal.pone.0186473
- Hunter, L. E. , Meer, E. A. , Gillan, C. M. , Hsu, M. , & Daw, N. D. (2019). Excessive deliberation in social anxiety. bioRxiv, 522433. https://doi.org/10.1101/522433
- Huys, Q. J. , Daw, N. D. , & Dayan, P. (2015). Depression: A decision-theoretic analysis. Annual Review of Neuroscience, 38, 1–23. https://doi.org/10.1146/annurev-neuro-071714-033928
- Huys, Q. J. , & Dayan, P. (2009). A Bayesian formulation of behavioral control. Cognition, 113(3), 314–328. https://doi.org/10.1016/j.cognition.2009.01.008
- Huys, Q. J. , Eshel, N. , O’Nions, E. , Sheridan, L. , Dayan, P. , & Roiser, J. P. (2012). Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Computational Biology, 8(3), e1002410. https://doi.org/10.1371/journal.pcbi.1002410
- Jacobson, N. C. , & Newman, M. G. (2014). Avoidance mediates the relationship between anxiety and depression over a decade later. Journal of Anxiety Disorders, 28(5), 437–445. https://doi.org/10.1016/j.janxdis.2014.03.007
- Kaelbling, L. P. , Littman, M. L. , & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1–2), 99–134. https://doi.org/10.1016/S0004-3702(98)00023-X
- Kessler, R. C. , Sampson, N. A. , Berglund, P. , Gruber, M. J. , Al-Hamzawi, A. , Andrade, L. , … Wilcox, M. A. (2015). Anxious and non-anxious major depressive disorder in the World Health Organization World Mental Health Surveys. Epidemiology and Psychiatric Sciences, 24(3), 210–226. https://doi.org/10.1017/S2045796015000189
- Konstantellou, A. , Campbell, M. , Eisler, I. , Simic, M. , & Treasure, J. (2011). Testing a cognitive model of generalized anxiety disorder in the eating disorders. Journal of Anxiety Disorders, 25(7), 864–869. https://doi.org/10.1016/j.janxdis.2011.04.005
- Lally, N. , Huys, Q. J. , Eshel, N. , Faulkner, P. , Dayan, P. , & Roiser, J. P. (2017). The neural basis of aversive Pavlovian guidance during planning. Journal of Neuroscience, 37(42), 10215–10229. https://doi.org/10.1523/JNEUROSCI.0085-17.2017
- Lee, S. W. , Shimojo, S. , & O’Doherty, J. P. (2014). Neural computations underlying arbitration between model-based and model- free learning. Neuron, 81(3), 687–699. https://doi.org/10.1016/j.neuron.2013.11.028
- Lejuez, C. W. , Read, J. P. , Kahler, C. W. , Richards, J. B. , Ramsey, S. E. , Stuart, G. L. , … Brown, R.. A. (2002). Evaluation of a behavioral measure of risk taking: The balloon analogue risk task (BART). Journal of Experimental Psychology: Applied, 8(2), 75–84. https://doi.org/10.1037//1076-898x.8.2.75
- Leotti, L. A. , & Delgado, M. R. (2011). The inherent reward of choice. Psychological Science, 22(10), 1310–1318. https://doi.org/10.1177/0956797611417005
- Leotti, L. A. , & Delgado, M. R. (2014). The value of exercising control over monetary gains and losses. Psychological Science, 25(2), 596–604. https://doi.org/10.1177/0956797613514589
- Leotti, L. A. , Iyengar, S. S. , & Ochsner, K. N. (2010). Born to choose: The origins and value of the need for control. Trends in Cognitive Sciences, 14(10), 457–463. https://doi.org/10.1016/j.tics.2010.08.001
- Ly, V. , Wang, K. S. , Bhanji, J. , & Delgado, M. R. (2019). A reward-based framework of perceived control. Frontiers in Neuroscience, 13, Article 65. https://doi.org/10.3389/fnins.2019.00065
- MacLeod, A. K. , & Byrne, A. (1996). Anxiety, depression, and the anticipation of future positive and negative experiences. Journal of Abnormal Psychology, 105(2), 286–289. https://doi.org/10.1037/0021-843X.105.2.286
- Maner, J. K. , Richey, J. A. , Cromer, K. , Mallott, M. , Lejuez, C. W. , Joiner, T. E. , & Schmidt, N. B. (2007). Dispositional anxiety and risk-avoidant decision-making. Personality and Individual Differences, 42(4), 665–675. https://doi.org/10.1016/j.paid.2006.08.016
- Marr,D. C. (1982). Vision: A computational investigation into the human representation and processing of visual information. Cambridge, MA: MIT Press.
- Mattar, M. G. , & Daw, N. D. (2018). Prioritized memory access explains planning and hippocampal replay. Nature Neuroscience, 21(11), 1609–1617. https://doi.org/10.1038/s41593-018-0232-z
- Momennejad, I. , Otto, A. R. , Daw, N. D. , & Norman, K. A. (2018). Offline replay supports planning in human reinforcement learning. eLife, 7, e32548. https://doi.org/10.7554/eLife.32548
- Momennejad, I. , Russek, E. M. , Cheong, J. H. , Botvinick, M. M. , Daw, N. D. , & Gershman, S. J. (2017). The successor representation in human reinforcement learning. Nature Human Behaviour, 1(9), 680–6 92. https://doi.org/10.1038/s41562-017-0180-8
- Morimura,T. , Sugiyama,M. , Kashima,H. , Hachiya,H. , & Tanaka,T. (2012). Parametric return density estimation for reinforcement learning. arXiv preprint arXiv:1203.3497.
- Moutoussis, M. , Shahar, N. , Hauser, T. U. , & Dolan, R. J. (2018). Computation in psychotherapy, or how computational psychiatry can aid learning-based psychological therapies. Computational Psychiatry, 2, 50–73. https://doi.org/10.1162/CPSY_a_00014
- Niv, Y. , Daw, N. D. , Joel, D. , & Dayan, P. (2007). Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology, 191(3), 507–520. https://doi.org/10.1007/s00213-006-0502-4
- Norbury, A. , Robbins, T. W. , & Seymour, B. (2018). Value generalization in human avoidance learning. eLife, 7, e34779. https://doi.org/10.7554/eLife.34779.001
- Oud, B. , Krajbich, I. , Miller, K. , Cheong, J. H. , Botvinick, M. , & Fehr, E. (2016). Irrational time allocation in decision-making. Proceedings of the Royal Society B: Biological Sciences, 283(1822), 20 151439. https://doi.org/10.1098/rspb.2015.1439
- Paulus, M. P. , & Yu, A. J. (2012). Emotion and decision-making: Affect-driven belief systems in anxiety and depression. Trends in Cognitive Sciences, 16(9), 476–4 83. https://doi.org/10.1016/j.tics.2012.07.009
- Pittig, A. , Treanor, M. , LeBeau, R. T. , & Craske, M. G. (2018). The role of associative fear and avoidance learning in anxiety disorders: Gaps and directions for future research. Neuroscience & Biobehavioral Reviews, 88, 117–140. https://doi.org/10.1016/j.neubiorev.2018.03.015
- Ramírez, E. , Ortega, A. R. , & Del Paso, G. A. R. (2015). Anxiety, attention, and decision making: The moderating role of heart rate variability. International Journal of Psychophysiology, 98(3), 490–496. https://doi.org/10.1016/j.ijpsycho.2015.10.007
- Russek, E. M. , Momennejad, I. , Botvinick, M. M. , Gershman, S. J. , & Daw, N. D. (2017). Predictive representations can link model-based reinforcement learning to model-free mechanisms. PLOS Computational Biology, 13(9), e100576 8. https://doi.org/10.1371/journal.pcbi.1005768
- Sutton, R. S. (1991). Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4), 160–163. https://doi.org/10.1145/122344.122377
- Sutton,R. S. , & Barto,A. G. (2018). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
- Symmonds, M. , Bossaerts, P. , & Dolan, R. J. (2010). A behavioral and neural evaluation of prospective decision-making under risk. Journal of Neuroscience, 30(43), 1 4380–114389. https://doi.org/10.1523/JNEUROSCI.1459-10.2010