The Intensity of Internal and External Attention Assessed with Pupillometry

1. Introduction

Attention allows for selectively processing the most relevant information within the rich visual world. Besides the selection of relevant information from the external world (e.g. looking for an apple in the grocery store), attention can also select information from temporarily internally stored representations in visual working memory (VWM; e.g. remembering you need to buy a banana as well). As such, attention shifts to selectively enhance items on a perceptual (i.e. external attention) or memory level (i.e. internal attention) to guide behavior (Chun et al., 2011; Gresch et al., 2023; Griffin & Nobre, 2003; Hautekiet et al., 2023; Heuer et al., 2020; Landman et al., 2003; Oberauer, 2019; Olivers & Roelfsema, 2020; Souza & Oberauer, 2016; van Ede, 2020; van Ede & Nobre, 2023; Verschooren & Egner, 2023).

One aspect of attention that is relatively unexplored concerns how intensely it is deployed (Miller et al., 2019; Unsworth & Miller, 2021). A possible reason for this lacuna in the literature is that traditional behavioral outcomes do not allow to determine the intensity of attention on a trial-by-trial basis, as such outcomes necessitate the calculation of summary values across conditions (e.g. accuracy in % correct or mean response times). The limitation of these behavioral measures is further exacerbated if internal attention is assessed, since it can be difficult to explicitly disentangle its effects from external attention (Koevoet, Strauch, Van der Stigchel, et al., 2023). Importantly, the ultimate quality of VWM representations is shaped by the integrative outcome of the intensity of external and internal attention. VWM encoding is strongly modulated by attention (e.g. Awh et al., 2006; Blom et al., 2016; Koevoet, Naber, et al., 2023) and dual-tasks have been shown to disrupt VWM prioritization (e.g. Janczyk & Berryhill, 2014; Lin et al., 2021). Thus, encoding and prioritization consume attentional resources and having more of such resources available for the task leads to improved VWM precision.

Pupil size holds promise as a physiological measure to effectively capture and delineate between the intensities of external and internal attention. Here, internal attention exclusively refers to the prioritization of material stored in VWM (Chun et al., 2011; van Ede & Nobre, 2023). We have recently proposed that different pupillary responses capture distinct operations of VWM (Koevoet, Strauch, Van der Stigchel, et al., 2023; Strauch, Wang, et al., 2022). For instance, the pupil orienting response and a later pupil dilatory component index encoding (external attention) and prioritization (internal attention), respectively. The pupil orienting response occurs within 200–700 ms following the onset of a visual stimulus and manifests as a brief dilation followed by a prominent constriction (Koevoet, Strauch, Van der Stigchel, et al., 2023; Lynn, 2013; Nieuwenhuis et al., 2011; Strauch, Romein, et al., 2022; Strauch, Wang, et al., 2022; Wang & Munoz, 2015). Specifically the pupil orienting constriction is linked to the depth of sensory processing, and thus the intensity of external attention (Barbur & Thomson, 1987; Binda & Murray, 2015; Naber et al., 2013; Strauch, Wang, et al., 2022). Building on this notion, our previous findings demonstrate that stronger orienting constrictions predict the amount and the precision of information committed to VWM (Koevoet, Naber, et al., 2023).

Pupillometric studies have leveraged the pupil light response (PLR) to study what is externally or even internally attended (Hustá et al., 2019; Unsworth & Robison, 2017; Zokaei et al., 2019). The PLR entails pupil dilations and constrictions in response to external dark and bright visual stimuli, respectively. Through the PLR, pupil size reveals what is externally attended: Attending dark items leads to larger pupils than when attending bright items (Binda & Gamlin, 2017; Binda et al., 2014; Mathôt & Van der Stigchel, 2015; Mathôt et al., 2013; Strauch, Romein, et al., 2022). Remarkably, Zokaei et al. (2019) recently showed that pupil size also reveals the focus of internal attention. In this study participants encoded bright and dark items during a VWM task. Through an auditory retro-cue, participants were instructed to internally attend to a specific brightness stored in memory. Crucially, whenever participants shifted internal attention toward a dark item, the pupil dilated more than whenever a bright item was prioritized internally. The magnitude of this PLR-modulated prioritization effect predicted participants’ working memory precision, directly linking it to behavior. Additionally, Unsworth and Robison (2018) found that internally prioritizing information in working memory elicited a pupil dilation, even irrespective of brightness (Unsworth & Robison, 2018). This response likely reflects (mental) effort akin to dilations during (V)WM maintenance (Beatty, 1982; Kahneman, 1973; Koevoet, Strauch, Van der Stigchel, et al., 2023; Strauch, Wang, et al., 2022), but this effect was specific to the internal prioritization process. The observed dilation during prioritization therefore reveals that shifting internal attention requires effort, making pupillometry a potentially valuable outcome measure for quantifying the intensity of such shifts. Note that we here consider the intensity of internal attention to be the sum of sub-processes such as internal orienting, selection and enhancement of stored representations (see van Ede & Nobre, 2023) because pupil dilation cannot delineate between these sub-processes. However, the precise relationship between behavioral outcomes and the brightness-independent dilation response when deploying internal attention remains unexplored.

Here, we reanalyzed data from a retro-cuing VWM task (Wilschut & Mathôt, 2022) to address three main questions. First, we aimed to replicate our previous finding which showed that more pronounced pupil constrictions during encoding predict more precise VWM representations (Koevoet, Naber, et al., 2023). Second, we asked whether pupil size captures the intensity of internal attention. Building on the link between the PLR-based internal prioritization effect and behavior (Unsworth & Robison, 2018; Zokaei et al., 2019), we hypothesize that this general pupil dilation response reflects the intensity of internal attention and, consequently, predicts the precision of VWM representations (Koevoet, Strauch, Van der Stigchel, et al., 2023). Third, if the pupil response components indeed reflect distinct VWM operations, both the constriction during encoding and the dilation during prioritization should predict the precision of VWM representations. We predict that both of these components shape the quality of stored material on a trial-by-trial basis.

2. Methods

2.1 Procedure

Thirty-one participants took part in the experiment reported in Wilschut and Mathôt (2022) (Figure 1; see the original paper for the full details). The original experimental procedure was approved by the ethics committee of the department of psychology at the University of Groningen (study approval code: PSY-2021-S-0321).

Figure 1

Briefly, participants encoded three different colored circles (2000 ms; 1.7° radius), maintained these colors for 1000 ms, and the spatial location of one color was then retrocued (100% valid) for reproduction (100 ms). After another delay period (2500 ms), one of five different task-irrelevant probes was presented for 100 ms: Exactly the same as the retro-cued colors (e.g. green), a color from the same category (e.g. lighter green), a different color (e.g. yellow), a color on the opposite side of the color wheel (e.g. purple) or no probe. These probes were used to investigate how different types of colors would affect pupil responses and bias eventual VWM reports in the original paper (Wilschut & Mathôt, 2022). Lastly, after a final delay period (2000 ms), participants reported the color of the retro-cued stimulus on a color wheel with a mouse click which ended the trial. Participants were instructed to respond as precisely as possible, and were awarded points based on their precision (for details see Wilschut & Mathôt, 2022).

The experiment consisted of a total of 190 trials equally distributed across probe conditions (38 per condition) in a mixed order, which were preceded by five practice trials. Throughout the experiment, gaze position and pupil size were measured monocularly using an EyeLink 1000 desktop mount (SR Research, Mississauga, ON, Canada) at 1000Hz.

3. Results

3.3 Internal and External Attention Jointly Predict Precision

The analyses above established that the intensity of both external and internal attention are captured by pupillary responses, but how do these forms of attention shape the quality of VWM representations? To address this directly, we analyzed whether pupillary dynamics predicted VWM precision on a trial-by-trial basis (Figure 2). To this end, pupil constriction during encoding and pupil dilation during prioritization were entered into an LME to predict the absolute hue error of the response at the end of the trial (also see Supplementary Materials). We found that both pupil constriction, β =.001 ± .0004, t = 2.45, p = .014, and pupil dilation amplitudes significantly predicted the error in hue at the end of the trial, β =.002 ± .0006, t =2.68, p =.007. Specifically, stronger pupil constriction during encoding and more pronounced pupil dilation elicited by internal prioritization predicted more precise responses (i.e. smaller absolute errors). Trial number did not significantly predict the precision of VWM reports, β =.005 ± .0028, t =1.66, p =.097. The model was determined using AIC-based backward selection, favoring a model that included neither baseline pupil size nor the interaction between encoding constrictions and prioritization dilations (Supplementary Materials). The fact that baseline pupil size and the interaction term did not meaningfully explain additional variance shows that 1) baseline pupil size was not significantly predictive of VWM precision when controlling for trial number and 2) external and internal attention may not interact to shape VWM precision. Together, these analyses indicate that pupil size indexes the intensity of external and internal attention, and that these components idiosyncratically predict the quality of VWM representations.

Figure 2

Errors significantly differed between the probe conditions, ts > 2.2, ps < .024 (bottom right in Figure 1). More specifically, whenever participants were probed with the exact color that was retro-cued, the absolute error was smaller compared with the other conditions. This is not surprising, since participants could refresh their representation of the color shortly before having to reproduce it. Note that the predictive pupil effects reported above are not driven by the probe conditions because probe types were considered in the analysis.

4. Discussion

Here, we found that the intensities of internal and external attention independently contribute to the quality of visual working memory content. Using openly available data (Wilschut & Mathôt, 2022), we show that the intensities of both types of attention are captured by distinct pupillary responses.

Recent studies employing subtle gaze biases (<1°) to track the focus of internal attention support the idea that internal attention is deployed at different intensities (de Vries & van Ede, 2023; Liu et al., 2023; Liu et al., 2022; van Ede et al., 2020; van Ede et al., 2019). The extent of these gaze biases toward internally attended content is associated with faster responses, indicating varying intensities of internal attention (van Ede et al., 2020; van Ede et al., 2019). Our study extends such findings by demonstrating that the intensity of internal attention, as measured through pupillary responses, enhances the precision of stored representations.

Turning to external attention, we report that smaller pupils during encoding lead to more precise VWM representations, complementing our previous work (Koevoet, Naber, et al., 2023). Our findings are consistent with the idea that distinct pupillometric response components inform about different operations of VWM such as encoding (external attention) and prioritization (internal attention) (Koevoet, Strauch, Van der Stigchel, et al., 2023; Strauch, Wang, et al., 2022). As such, relatively early pupil constrictions reveal encoding, while dilation responses to retro-cues reflect prioritization. Our approach overcomes inherent limitations of behavioral measures that cannot separate and quantify contributions of external and internal attention.

Using pupillary responses, we show that encoding and prioritization dynamically shape the quality of VWM representations. The two pupil response components explained idiosyncratic variation in VWM precision, and did not interact significantly. This shows that internal and external attention may idiosyncratically enhance stored material in a ‘more is better’ fashion – possibly without interacting strongly. These distinct pupillary response components serve as indices of the intensities of internal and external attention. Using (de)convolutional modeling techniques (see below), future work may provide more detailed insights into the interplay between external and internal attention on a trial-by-trial level.

Why is it beneficial to deploy attention at differing intensities? Deploying attention is often considered costly (Just et al., 2003; Kahneman, 1973; Koevoet, Strauch, Naber, et al., 2023), and stronger, more intense deployments of attention should be more expensive. To optimize such costs, the intensities of external and internal attention should be dynamically adjusted to only be deployed as strongly as necessary for a task. In contrast, an all-or-none deployment of attention does not allow for such a cost-efficient mechanism. The current data cannot test this costs-perspective directly as only naturally occurring fluctuations of internal and external intensities were investigated (e.g. due to fluctuations in the efficiency of attentional shifting across trials or failures to engage in the task, see De Jong et al., 1999). Thus, future work could experimentally manipulate task demands to elucidate whether the intensity of attention can be flexibly adjusted to optimize costs based on the current goals.

Two other potentially relevant pupillary dynamics were not directly assessed here: Baseline pupil size and pupil dilation during maintenance. First, baseline pupil size reveals tonic fluctuations in arousal, and has been linked to VWM precision/performance previously (Galeano-Keiner et al., 2023; Koevoet, Naber, et al., 2023; Robison & Unsworth, 2019; Starc et al., 2017). Specifically, baseline pupil size is thought to be linked to VWM precision following an inverted-U curve (Aston-Jones & Cohen, 2005; Gilzenrat et al., 2010; Jepma & Nieuwenhuis, 2011; Koevoet, Naber, et al., 2023; Koevoet, Strauch, Van der Stigchel, et al., 2023; Teigen, 1994; Yerkes & Dodson, 1908). Baseline pupil size did not affect VWM precision when controlling for trial number in the current data (Supplementary Materials). This could have multiple reasons: 1) Participants were at the performance optimum around the peak of the inverted-U curve. This is feasible since responses were overall very precise. 2) Trials were relatively long (7.7s) leading to reduced predictive power of baseline pupil size or 3) encoding constrictions and prioritization dilations are stronger predictors of VWM precision, accounting for similar variance as baseline pupil size.

Second, memorizing more items causes increased pupil dilation until one’s (V)WM capacity is reached (Beatty, 1982; Granholm et al., 1996; Kahneman & Beatty, 1966; Kosachenko et al., 2023; Robison & Unsworth, 2019; Zhou et al., 2022). Note that this dilatory pupil effect is also elicited in tasks that do not employ retro-cues (i.e. no internal prioritization is facilitated). This component is difficult to isolate in the current data because a caveat to utilizing pupillary dynamics to capture distinct processes is the temporal sluggishness of the signal. When (cognitive) processes occur in close temporal proximity to one another, delineating between the effects of pupil responses during encoding, maintenance and prioritization becomes complex (Mathôt & Vilotijević, 2022). To fully capture encoding, maintenance and prioritization in a single task on a trial-by-trial basis, trials need to be relatively slow-paced, allowing for the pupil to resolve cognitive events. The sluggishness of the pupil response could also be accounted for using (de)convolutional modeling techniques (Denison et al., 2020; Knapen et al., 2016; Korn & Bach, 2016; Wierda et al., 2012). Future work could incorporate such methods and/or techniques to obtain a complete picture of all cognitive operations employed when flexibly using VWM.

Together, our findings show that the intensities of internal and external attention are captured by pupil size on a trial-by-trial basis. The intensity of internal and external attention independently predict the quality of VWM representations. Our approach holds promise to elucidate the intricate interplay between internal and external attention to effectively guide behavior in the rich visual world.

Data Accessibility Statements

Analyses scripts to reproduce the results are available via the Open Science Framework: https://osf.io/ek3ap/. The original analyses and data by Wilschut and Mathôt (2022) can be found here: https://osf.io/qksfh/.

Additional File

The additional file for this article can be found as follows:

Ethics and Consent

The original experimental procedure was approved by the ethics committee of the department of psychology at the University of Groningen (study approval code: PSY-2021-S-0321).

Funding Information

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement n° 863732).

Competing Interests

The authors have no competing interests to declare.