Individual Differences in Holistic and Compositional Language Processing

1.1 Backward transition probability

As they move through the world, both proficient adult speakers and young L1 learners pay attention to the distributions inherent in the language around them (Aslin, 2017). Through a process of statistical learning, language users keep track of the frequency with which certain syllables, morphemes, and words co-occur. Sensitivity to co-occurrence is a key component of native language acquisition; learners break down acquired multiword units (e.g., “gimme-it”), which are initially un-analyzed chunks, into their component parts (i.e., “give” + “me” + “it”) (Abbot-Smith & Tomasello, 2006; Arnon, 2009; Lieven et al., 2009). Usage-based and emergentist theories of linguistics posit that this sensitivity continues throughout a speaker’s life, with each exposure to language leaving a memory trace that updates their mental representation. This results in implicit knowledge of the frequency of not only single words, but also multi-word units (Arnon & Snider, 2010; Behrens & Pfänder, 2016; Carrol & Conklin, 2020; Divjak, 2019; Gries & Divjak, 2012).

Which units are then stored as wholes, and which are constructed from their parts? It appears that units are stored and retrieved redundantly at both the level of the “part” and the “whole” (Blumenthal-Dramé, 2017; Bybee, 2010; Hilpert, 2014; Yang et al., 2020). That is, smaller, decomposed units (e.g. morphemes, words) exist along a spectrum of relative cognitive prominence along with their complex forms (e.g. multi-morphemic words, multi-word units) (Hay & Baayen, 2002). A continuum emerges in terms of how accessible the representation of the whole is relative to the representation of the part (Hay & Baayen, 2005), and this continuum is likely affected by both participant-level and item-level variables. One item-level variable may be the relative frequency of parts to wholes (i.e., “worthless” is more frequent than “worth” so is often processed at the chunk level, but “tearless” is less frequent than “tear”, so it is more likely to be constructed compositionally) (Blumenthal-Dramé, 2017; Blumenthal-Dramé et al., 2017; Hay & Baayen, 2005). Additional factors are also known to affect chunk status both within and outside of the linguistic domain, including expertise, compositionality, and salience, among others (Gobet et al., 2016; Gobet & Clarkson, 2004; A. Goldberg & Suttle, 2010; Miller, 1956; Yang et al., 2022).

In the current experiment, we focus on the frequency-based relationships between words, and take a common metric of lexical co-occurrence to represent the relative frequency of parts to wholes: backward transition probability (BTP). This simple interlexical statistical metric measures the frequencies of one individual unit compared to the frequency of its co-occurrence in a larger whole; specifically, the likelihood of part two being directly preceded by part one. In the scope of the current paper, BTP is calculated as the likelihood of word 1 (W1, e.g., absolute) directly preceding word 2 (W2, e.g., silence), compared to all instances of W2, though the metric is not limited to lexical units and can be calculated at any grain size. For example, if absolute silence has a corpus frequency of 154, and silence appears 36,701 times as a noun in the same corpus, the bigram is given a BTP score of 154/36701 (approximately .004).

BTP between words (and particularly, in bigrams) has been found to affect reading times and eye movement behavior when reading (e.g., Demberg & Keller, 2008; Kapteijns & Hintz, 2021). The metric has been used to operationalize diverse theoretical concepts in linguistics, including probabilistic integration of unfolding words or phrases into previously-encountered contexts and structures (also called “retrodiction”) (Ferreira & Chantavarin, 2018; Onnis & Huettig, 2021; van Paridon & Alday, 2020), and sensitivity to phonotactic contingencies and lexical co-occurrence (McConnell & Blumenthal-Dramé, 2019; Perruchet & Desaulty, 2008). The measure has also been used to model language learning processes (McCauley & Christiansen, 2019; Pelucchi et al., 2009; Roete et al., 2020). BTP’s sibling metric, forward transition probability (FTP), has also received wide-spread attention, along with its log-inverse: “surprisal” (Boston et al., 2008; Frisson et al., 2005; Hale, 2016; Levy, 2008; Lowder et al., 2018; McDonald & Shillcock, 2003; Smith & Levy, 2013; Willems et al., 2016). Although the two metrics are similar, they have been found to be independently informative of real-time language processing (Onnis & Huettig, 2021).

It is reasonable to assume that BTP could be a predictive metric for sentence reading in general, i.e., at any point in a given sentence. Modifier-noun bigrams, however, are a particularly interesting stage upon which to observe this phenomenon for multiple reasons: First, they are easily extracted from corpora. At the same time, grammatical structure is kept constant (thereby avoiding potential influence from confounds like syntactic structures and parts of speech). There are also observable probabilistic preferences for one modifier over another even when there are multiple synonymous options (e.g., ‘vast majority’ vs. ‘large majority’). Additionally, modifier-noun NPs are right-headed in English, i.e., the head noun appears after or to the right of potential modifiers. Because the head noun is both obligatory and carries the main semantic weight of the phrase, orienting towards this important lexeme may take primacy over pre-empting potential modifiers. This is supported by previous research, which shows that BTP is a stronger predictor of reading times for modifier-noun bigrams than FTP and other corpus linguistic association measures (McConnell & Blumenthal-Dramé, 2019, 2021), particularly for speakers of English, as compared to a left-branching language like Korean (Onnis & Thiessen, 2013).

Therefore, we take BTP to operationalize the prominence of the wholes to the parts in the modifier-noun bigrams that make up the stimuli in this experiment. Specifically, we measure the BTP score between the modifier and the noun in bigrams like “absolute silence” (see Section 2.1 for more about stimuli). We assume that BTP represents the relative prominence of the chunk (the bigram) compared to its component parts (the individual words) because it is calculated via the frequencies of the bigram and the frequency of the noun. We hypothesized that comprehenders would emerge on a spectrum between a preference for the chunk level and a preference for a rather atomistic or compositional approach; this hypothesis is in line with theories of visual perception, which have found that factors like age, expectation, and even ability for creative thinking can affect global precedence biases (Behrmann et al., 2014; Lee et al., 2021; Staudinger et al., 2011; Summerfield & Egner, 2009; Zabelina & Ganis, 2018). This difference should be apparent in (self-paced) reading times to the noun in the bigram, as it is the point in which the comprehender can process and integrate the chunk (i.e., the bigram). Further, we postulated that there would be identifiable individual differences that modulate which processing style a particular comprehender prefers (McCauley & Christiansen, 2015).

1.2 Endogenous individual differences

Usage-based theories have widely postulated and collected evidence for the influence of experience, including age, years of formal education, print exposure, and even the kinds of texts readers have been exposed to (Dąbrowska, 2018, 2019; Verhagen et al., 2018). Age and reading experience have been established as important modulators of reading strategy in the context of modifier-noun compounds specifically; older readers and those with less reading experience, i.e., those readers who may have greater need for compensation, showed stronger effects of transition probability (McConnell & Blumenthal-Dramé, 2021). However, these experience-based, or “exogenous”, factors are not the sole determiners of individual differences; there are also a host of “endogenous”, i.e., inherent, factors that seem to play a role in language processing (Kerz & Wiechmann, 2021; Kidd et al., 2018; Kidd & Donnelly, 2020; Ryskin et al., 2020).

In the current experiment, we account for the known experience-based predictors of age, education, and reading experience while investigating the explanatory potential of five endogenous cognitive abilities that might be related to processing style. Of course, creating a binary distinction between endogenous and exogenous traits is a simplification; many (if not most) endogenous traits are likely affected by experience. However, this distinction highlights that the factors we focus on here are not primarily experience based. Specifically, the current experiment investigates two measures (global/local shifting and inhibition) that reflect processing preferences for the holistic or the atomistic level and two cognitive abilities known to affect language comprehension (verbal working memory and reaction speed). The fifth factor, personality, is an exploratory variable that has newly been gaining ground in linguistic research for its ability to capture underlying processing preferences and styles that may account for achievement in relevant abilities like statistical learning.

1.2.4 Personality traits

Recently, research has become increasingly interested in a potential new source of individual difference in language skills: personality traits. Personality is not only a socio-cultural construct but has also been deconstructed into five core cognitive tendencies in gathering, processing and sharing information (Goldberg, 1990; McCrae & John, 1992; Wiggins, 1996). The “Big Five” personality traits measure extraversion, openness, conscientiousness, agreeableness and neuroticism (also called emotional stability) (see Figure 1) and they have been found to correlate with the volume of brain regions associated with similar personality characteristics (DeYoung et al., 2010). The Big Five traits have been associated with differences in academic achievement and L2 learning (Grey et al., 2015; Komarraju et al., 2011; MacIntyre & Charos, 1996), and recently, in sensitivity to linguistic violations at multiple levels (Hubert Lyall & Järvikivi, 2021).

Figure 1

Of the five traits, we were particularly interested in openness, conscientiousness, and extraversion. Openness has been attributed to better implicit learning ability (Kaufman et al., 2010) and L2 speaker’s sensitivity to frequency effects (Kerz & Wiechmann, 2017), and the same statistical learning abilities may be called on to store and utilize BTP. Conscientiousness, (though also agreeableness and neuroticism) are considered to rank how ‘sociable’ or ‘conforming’ a person is (DeYoung et al., 2002; Roberts et al., 2014), which, based on the literature, we hypothesized might lead to higher motivation in an experimental context (Stevens & Ash, 2001). Finally, extraversion may be a source of individual difference in language processing because extraverted people may be more likely to have larger or more diversified social networks, in turn making them more able to predict upcoming input, though it is unclear if this prediction could occur at the level of interlexical co-occurrence (Lev-Ari, 2019). However, we refrained from making specific or directional hypotheses on the effect of these variables, given their highly exploratory nature and lack of previous literature.

To assess these continuums, participants completed the 60-question BFI-2 inventory (Soto & John, 2017), a well-established tool in psychology, in which they were asked to self-report how much they identified with statements like “I am someone who is outgoing, sociable” (extraversion) or “I am someone who is curious about many things” (openness) on a scale of 1 to 5.

2.1 Stimuli

Stimuli were full sentences that were composed of three parts: semantically neutral onsets, modifier-noun bigrams and three-word spillover regions (see Table 1 for example). Bigrams were extracted from the Corpus of Contemporary American English (COCA) equally across 20 bins of log bigram frequency (Davies, 2008). They were then inspected by hand and removed if they were emotionally arousing (violent, religious or sexual), specialist terms from specific domains like sports or medicine (e.g., ‘underactive thyroid’), featured a modifier that was identical in form to a verbal participle (e.g., ‘cooked crab’, ‘galloping horse’), or were either idioms or compounds (e.g., ‘botanic gardens’, ‘soy cheese’). Additionally, bigrams were disqualified if either lexeme was longer than 12 letters, contained more than one base and/or more than two derivational and inflectional morphemes (e.g., ‘biodegradable’), or included a removable prefix (e.g., ‘inexperienced’), as any of this criteria could mean that the lexeme itself was a collocation at the morphological level.

Each bigram was then matched to a partner bigram with a maximally similar log-transformed bigram frequency (+/– 0.25). This allowed us to isolate the effect of the association between the individual words. After this step, we had two items with the same first word and nearly the same log bigram frequency (e.g., “absolute silence” and “absolute control”). However, we could still not rule out that if one was read faster than the other, it was an absolute effect of the second word (“silence” vs. “control”). Thus, we added two more items to the pair with the same second word and a maximally similar first word (e.g., “total silence” and “total control”), for a total of four matched items (Edmonds, 2013). The matched items were not extracted from the corpus but invented as semantic matches, so bigram frequency was not controlled across the matched condition.

The second word in the bigram (the noun) was designated the critical word, because this is the location in which the bigram can be fully identified, given that English noun phrases are right-headed. Log-transformed frequency of both the first word and the second word in the bigram were added to the data frame, as well as the bigram frequencies. All frequencies were extracted from COCA (2019 download version). BTP was calculated as bigram frequency divided by word 2 frequency using the same frequencies. Figure 2 shows the distribution of BTP scores across the stimuli, both in terms of raw values and the distribution after log-transformation (since the distribution is clearly Zipfian).

Figure 2

Items were then embedded in neutral sentences, which were counterbalanced across participants to ensure that the sentential environment did not systematically affect reading times. Table 1 exemplifies this, with participants being assigned to either List A or List B, and parts 1 and 2 being maximally spaced from each other within the course of the experiment (either all 1s before all 2s or vice versa). The same items were used in a previous experiment (McConnell & Blumenthal-Dramé, 2021).

Participants read the sentences above word-by-word in a moving window self-paced reading setup, which was based on the code by IbexFarm (Drummond, 2016). For this, participants saw lines representing the lengths of words in a sentence and were asked to press the spacebar to make the first word appear. The first word appeared over the line that corresponded to its length, and when participants pressed the spacebar again, the first word disappeared and the second word appeared. In this way, participants proceeded through all words in all sentences. Approximately one-third of sentences were followed by a comprehension question, which generally had 3 possible options (a small minority were in a yes/no format) and covered various levels of language processing from name recall to deeper syntactic understanding.

Participants read for about 45 minutes after several practice sentences and were instructed to read as normally as possible and to take breaks (e.g., to take a drink of coffee) only on the question screens. They were informed that the sentences were not related and did not form a story. After collecting data, 29 items were removed from the analysis because they were either unattested in COCA (this only affected items from the semantic match condition) or had the same first word as another item, which inadvertently happened due to the complexity of the matching process. 259 sentences per participant remained, regardless of the list they were assigned.

2.2 Participants

Participants were recruited online via Prolific (Damer & Bradley, 2018). Only participants who were monolingual English speakers without reading disabilities and answered 80% of SPR comprehension questions correctly were eligible; recruitment stopped when 100 such submissions were reached. Of them, 57 were female and 1 was non-binary or another gender. Ages ranged from 18 to 76 (median 30.5). 62 were UK nationals and 38 were US nationals. 46 had undergraduate education, 40 had a degree below a bachelor’s degree and 14 had a degree above a bachelor’s degree. If participants missed a task, which happened for technical reasons in a small number of cases, their score for the missing task was set to the average score across that task. This affected 4 cases for the Navon task and 2 for reading experience.

2.3 Individual difference measures

There was considerable variability between participants in all measures. No data was removed. Dependent variables were calculated as outlined in Table 2.

Figures 3 and 4 show the distribution of each of the individual difference assessments, with one value per participant. The mean, standard deviation, skewness and kurtosis of each task is also represented numerically in Table 3, along with the range of values on each task.

Figure 3

Figure 4

Internal consistency of each task was calculated using split-half reliability using the R package splithalfr (Pronk et al., 2022) using 5,000 random splits. Personality was excluded from this calculation, since the BFI-2 inventory has been independently validated (Soto & John, 2017). For the complex response time task, the Spearman-Brown corrected reliability estimate was 0.82, 95% CI [0.73, 0.91]. For the Flanker task, the estimate for the compatible condition was 0.96, 95% CI [0.93, 0.97] and for the incompatible condition was 0.94, 95% CI [0.89, 0.97]. For the Navon task, the estimate for the global condition was 0.87, 95% CI [0.82, 0.91] and for the local condition was 0.90, 95% CI [0.86, 0.93]. For the reading span task, the estimate was 0.86, 95% CI [0.81, 0.89].

2.4 Model specification

After data collection, the noun in the bigram was labeled the “critical word” because it is at this point that the bigram can be identified. Previous research has shown that so-called “spillover effects” can affect the reading times of subsequent words, so the two words directly following the critical word were coded as “spillover 1” and “spillover 2”, respectively. These three levels made up the “position” variable.

All numeric predictors were centered and scaled to ensure that individual difference assessments on different scales were comparable; additionally, the Big 5 personality traits were scaled to each participant’s mean response across all five tasks to account for participant differences in responding to a Likert scale (i.e., those who consistently rate themselves higher or lower on all dimensions). Response times, frequencies and transition probabilities were log-transformed (Smith & Levy, 2013). To ensure that only true reading times were included (i.e., none that were physiologically impossible or exaggeratedly long), we removed RTs outside 2.5 standard deviations from each participant’s mean (2.22%).

To pre-empt collinearity problems, the Spearman’s rank correlation between all individual difference assessments was consulted before model fitting (see Figure 5). If two terms were correlated above 0.4, only the one that was hypothesized to be most relevant to the research aims was included in the model. Although we collected both simple and complex reaction speeds as part of the reaction time measure, simple reaction speed was not considered in the model because it correlated strongly with complex reaction speed (ρ = 0.62) and it was deemed inferior in measuring cognitive processing speed because of how heavily it may rely on internet and hardware speed. Similarly, reading experience was not added to the model because it was strongly correlated with verbal working memory (ρ = 0.49), given that the focus of the current paper is on endogenous factors, and reading experience was intended primarily as a covariate. This does, however, align with proposals that verbal working memory may at least in part reflect language experience (Kidd et al., 2018; MacDonald & Christiansen, 2002). The exception to this threshold were individual subcomponents of the Big 5 personality traits, since these are well established. After model fitting, VIF was checked and confirmed that there were no multicollinearity problems.

Figure 5

The model formula included ten individual difference terms: complex response time, verbal working memory (“rst_mem”), inhibition (“flanker”), global/local shifting (“navon”), the five Big 5 personality traits, and age. Additionally, the model included the following known covariates: current word length, previous word length, position (word number) in sentence, position in experiment, place of origin (US or UK) and education level (high school or trade school, undergraduate and graduate education). Word length was used as a proxy for word frequency because word frequency is directly implicated in the calculation of BTP, the two measures are (negatively) correlated (ρ = –0.45), and previous research made the same methodological decision (McConnell & Blumenthal-Dramé, 2021).

All categorical predictors were sum coded. A linear mixed effects model was fit in Julia using the package MixedModels.jl (Bates et al., 2021) and JellyMe4.jl (Alday, 2023), and plotted in R using lme4 (Bates et al., 2015) and ggeffects (Lüdecke, 2018). Random effects were fit on three grouping variables: participant ID, first word in the bigram, and second word in the bigram. Due to the stimuli design, this controlled for the paired quads that were designed to minimize the effects of individual words. Random slopes were fit to word-level covariates. After model fitting, the principal components of the random effects were checked for overparameterization, but no problems were detected, so the random effects structure did not need to be edited.

A single step of forward model selection was performed after finalizing the random effects structure, in which a three-way interaction term by position was added only for those individual difference terms that showed significant two-way interactions with BTP. This was intended to account for hypotheses about the time course of effects while minimizing the number of three-way interactions that the model included.

The final model formula is below, written in Julia model syntax conventions (@formula)².

logRT ~ 1 + BTP_lz & navon_z & position + BTP_lz & flanker_z & position + BTP_lz * (complex_rt_z + rst_mem_z + navon_z + flanker_z + big5_O_cz + big5_C_cz + big5_E_cz + big5_A_cz + big5_N_cz + age_z) + position + trial_number_z + word_number_z + length_z + prev_length_z + origin + education +

After model fitting, model diagnostic plots and assumptions were checked and no violations were detected. No VIF scores were above 2. The full analysis code is available in the OSF repository associated with this paper: https://osf.io/zmh48/.