One Standard for All: Uniform Scale for Comparing Individuals and Groups in Hierarchical Bayesian Evidence Accumulation Modeling

LBA

In the LBA, each choice alternative is associated with an evidence accumulator. The initial amount of evidence in each accumulator prior to evidence accumulation is determined by the starting point parameter, which defines the range (0-starting point) from which the amount is sampled in each trial. In each trial, the drift rate is also sampled from a normal distribution with mean drift rate (v) and a standard deviation of the drift rate (sv). The LBA is acknowledged to be simplified because it assumes that evidence accumulation rate remains constant in a given trial (hence, “Linear Ballistic”). A decision is reached once the amount of evidence in an accumulator crosses the boundary, b, which is sometimes estimated indirectly through B, where b = starting point +B. The last parameter is non-decision time (t0) expressed in sec and describing the duration of the non-decisional processes such as early feature extraction and motor preparation.

When applying the model using the Bayesian hierarchical method, one estimates the parameters for each individual, separately, but constrains this estimation with a population level distribution. The process of model fitting starts with the definition of the model which includes the determination of the free (to be estimated) parameters and the fixed parameter that sets the scale. Following, one needs to specify two types of priors: base- level priors, which are the priors for each parameter at the individual participant’s level (typically, all participants’ parameters have the same priors), and hyper level priors which are the priors for the parameters that characterize the population. Estimation proceeds by posterior sampling using Markov Chains Monte Carlo (MCMC) method. Posterior sampling adequacy is assessed using the Potential Scale Reduction Factor (PSRF), that should fall below 1.1 (Brooks & Gelman, 1998).

The new method

In the new method, we de facto fixed sv at the population level. This was done indirectly using the hyper- level Bayesian priors. More specifically, we defined the Mu (population level mean) prior for sv, using a Beta distribution with α = β = 1 (creating a uniform distribution) and a tiny range from .999 to 1.001. This method de facto fixes the mean of the population sv at about 1, and thus sets the scale (one unit of evidence is equal to the population-level sv), but still allows for individual and group differences in sv. In the present work, we report two parameter recovery studies, providing the necessary proof of concept for the new method. In each parameter-recovery study, we simulated data using pre-determined parameter values, and then estimated the parameter values with the model, and checked whether the estimates are close enough to the pre-determined values.

To summarize, the goal of this study was to find a credible implementation of the notion to fix one parameter at the population level, which allows the EAM (here, LBA) to investigate individual differences or experimental effects while not being exposed to the aforementioned risks.

Method

The steps involved in Study 1 are summarized in Figure 1. We first simulated 21 data sets, that differed from one another in the combinations of mean (across participants) parameter values (Table 1). Based on pervious empirical work (Berkovich & Meiran, 2023; Berkovich & Meiran, 2024; Givon et al., 2023), the range of parameter values is realistic for the type of tasks that we study in our lab. Given the fact that each model fitting takes a very long time (just to give an idea, a single fitting for the 21 models took about one month to converge, despite using massive parallel processing), we limited the number of mean-combinations by not changing the starting point mean value, which was fixed across simulations at 2. Additionally, since research questions that concern t0 are not the focus of our studies, the value of t0 for all ”participants” in all models was set to 0.3, and its recovery was accordingly not tested. Not changing t0 and the fact that we used a particular range of parameter values consist of a limitation of the current study.

Figure 1

To simulate participants’ data, we first determined the “true” individual parameter values by drawing from normal distributions, with means as defined in Table 1, and with a standard deviation being 0.6 for starting point and boundary, 0.8 for v. true and v. false, and 0.4 for sv. Randomly drawn values were corrected if they fell outside an allowable range. The lower bound for starting point, boundary and v. false was 0.05, for v. true the lower bound was 1.3, and for sv the lower bound was 0.01. Additionally, if the difference between v. true and v. false in a specific model was smaller than 0.2 (favoring v. true), then the value of v. false was replaced by the value of v. true minus 0.2. Using these “true” individual parameter values, we simulated 21 datasets, each comprising 100 “participants”, each with 1,000 “trials”.

Each such simulated dataset was then fitted using Bayesian hierarchical LBA. The priors for individual estimates were the mean values that were used for simulation with noise added. We added noise to the mean values by adding a value that was drawn from a normal distribution with mean = 0 and SD = 0.1. Adding noise makes the priors more realistic in the sense of being “in the neighborhood” of the true value but not being exactly at it.

The population level prior means were the same as the individual level prior means except for sv which was defined using (uniform) Beta distribution with α and β values set to 1 and range from 0.999 to 1.001. The Sigma prior (population standard deviation) was uninformative (meaning that it was de factor determined by the data alone and not by the priors) and was defined with a (uniform) Beta distribution, with α = β = 1 with a range of 0–3.

Posterior sampling was accomplished with a burn in period of 1,000 samples per sample-chain and actual sampling of 12,000 samples and thinning equal to 12, which means keeping each 12^th sample. We used the default number of chains: number of free parameters times three.

The results for each one of the 21 data-sets thus comprised of a set of “true” parameter values (those used to simulate the data) for each one of the 100 “participants” and estimated parameters for these “participants”. Of interest was the degree of correspondence between the “true” and estimated parameter values across participants in each one of the data-sets.

Method for assessing parameter recovery

After the model has converged, we calculated PSRF for each participant (to assess model convergence). Then we computed the Pearson correlation and the ICC_2,1 between the “true” parameter values and the (estimated) recovered parameter values. The logic is that if Pearson correlation and the ICC_2,1 are high, we can deduce that the new method is credible and can be used to compare individuals/groups. This is true in the sense that if the LBA provides a reasonable approximation of the data generation process, then the estimated parameter values correctly represent that operation of this process. We used both Pearson correlation and ICC_2,1 to allow us to inspect both the stability of the relative score (Pearson correlation) and the absolute agreement (ICC_2,1). Since this procedure produced 21 correlations and 21 ICC’s, we averaged these values through Fisher’s Z-transformation. It is important to note that while we employed a Bayesian method for parameter estimation, the hypothesis testing was conducted using frequentist statistics. This approach was taken because we are not aware of a Bayesian method to test the ICC for significance. Consequently, we decided to utilize frequentist statistics for all our tests to maintain consistency.

Results

Most of the models converged successfully, with PSRF falling below 1.1 for 95%–100% of the ”participants” in each model. Only in the case of one model, PSRF fell below 1.1 for only 19% of the “participants”. Since for some simulated participants, PSRF was unsatisfactory and fell above 1.1, we report the correlations and the ICC_2,1s value after removing these “participants”. The parallel analysis that includes these “participants” did not change the picture too much and is reported in the Supplementary Materials.

We found that with the exception of v.false, all correlation and all ICC_2,1s were significantly different from zero (p < 0.05). For the v.false its ICC_2,1 was not significant in 8 out of the 21 models. The range of values is shown in Table 2. Additionally, the Pearson correlation results are shown in Figure 2a, and ICC_2,1 results are shown in Figure 2b.

Figure 2

It seems that all parameters, except for v.false, were recovered successfully at the individual (simulated) participant level, as reflected in both a reasonably high Pearson correlation and a reasonably high ICC_2,1.

We suspected that v.false failed to recover successfully not because of the new method, but that it might be a disadvantage of the classic method as well. In order to test if this is indeed the case, we ran another parameter recovery study, using the exact same method as before, but this time we fixed sv = 1 for all participants, as customarily done (i.e., the “classic” method). The results of this parameter recovery are reported below.

This time we found that for all “participants” in all models, PSRF fell below 1.1, except for one “participant” in one model. This “participant” has been removed from the analysis and the results of the analyses that include this “participant” are reported in the Supplementary Materials.

Once again, all Pearson correlations for all parameters were significantly different from zero (p < 0.05). This time, all ICC_2,1 but that of v.false met this criterion. The results of this analysis are reported in Table 3 and Figure 3.

Figure 3

Summary

The results of Study 1 demonstrate the applicability of the newly suggested method by a quite successful parameter recovery, including that of sv. However, in retrospect, we identified three shortcomings in our methodology. Specifically, (a) it is plausible that the successful parameter recovery was attained because the priors were unrealistically similar to the true population mean values. (b) It seems that the recovery by the new method, despite being successful, was less successful than when using the “classic” method. Additionally, (c) most real-world studies model at least one parameter as a function of groups or conditions, and this division is lacking in Study 1. (Nonetheless, individual differences in the parameter values were successfully captured). To address these shortcomings, we conducted Study 2, using a slightly different approach.

Method

To address the first drawback, regarding the too-close-to-true priors, we adopted a different method to create the priors in which we sampled the priors from a uniform distribution that ranged two standard errors below and above the true parameter mean. We believe that this change more accurately mirrors real world applications, characterized by well-informed guessing, where researchers have extensive experience with the task at hand.

To address the second and third shortcoming, we wanted to demonstrate the most important feature of the new method which allows one to find true parameter differences without the risk of wrongly attributing differences to parameters that do not truly differ. Therefore, we simulated two groups of 50 “participants”, each. In the first group, sv mean value was 1.5, and in the second group, sv mean value was 0.5. The population mean values in the rest of the parameters were identical across the two groups: Starting point = 2, Boundary = 2.5, v.true = 2.5, v.false = 0.8, t0 = 0.3. Then, we created the parameters for each participant, and the data, using the same method that was used in Study 1. To reveal the discrepancy between the new method and the classic method, we fitted the data twice: once using the new method, and a second time using the classic method. The idea was to demonstrate that the new method would reveal the true difference between groups (sv), but that the classic method might wrongly reveal non-existing groups differences in other parameters. This time, we also decided to assess model fit using the Deviance Information Creation (DIC; Spiegelhalter et al., 2014), in addition to just assessing parameter recovery success. Using DIC allowed us to compare between the new and the classic methods in terms of model fit, with a lower DIC indicating better fit. Following a reviewer’s comment, we add that the DIC may be inaccurate in complex models such as the LBA (Pooley & Marion, 2018). Nonetheless, DIC helped us (albeit tentatively) compare the two methods.

Results

Modeling the data using both the classic and the new method was pretty successful in terms of convergence, with PSRF falling above 1.1 for only one “participant” in the classic method (2.45), and for three “participants” in the new method (2.01, 2.08, 2.23). As before, those “participants” were removed from further analyses, but the results of the analysis including them did not change the picture too much and are reported in the Supplementary Materials.

In both methods, all correlation and all ICCs were significantly different from zero (p < 0.05). See Tables 4 and 5 and Figures 4 and 5.

Figure 4

Figure 5

While comparing the two models using DIC, we found that the new method generates a better fit than the “classic” method in 58 out of the 96 ”participants” (60.41%). The summation of the DIC for each method separately yields the same conclusion, indicating a hugely lower (better) value for the new method (194,665.3) compared to the “classic” method (195,714.6), indicating decisive superiority of the new method. Considering that DIC differences of even 6 are considered to be meaningful, the current difference of 1,049.3 points probably reflects a meaningful discrepancy, even when considering DIC’s limitations.

Finally, and most importantly, we conducted a series of t-tests to compare the mean posterior values between the two groups. When using the new method, we found that the groups were significantly different from each other only in sv, (t(95) = 14.302, p < 0.001), in the correct direction (Mean of first group = 2.120, Mean of second group = 0.772), while the rest of the t-tests were non-significant (p > 0.05). In other words, the new method correctly detected the true group difference, and did not wrongly detect other differences. In a sharp contrast, the classic method (wrongly) found significant group differences in all the parameters (see supplementary materials for a full description). In other words, the classic method failed detecting the true group difference (because it existed in the fixed parameter) and, as a result, wrongly found a group difference in the remaining parameters.

Summary

In summarizing the results of Study 2, it appears that the new method overcomes the limitations of the classical method. Specifically, when groups differ in sv, the classic method wrongly found differences in other parameters, while the new method accurately revealed the true differences between the groups. Additionally, the new method recovered the parameters of the individual participants successfully while employing Pearson correlation and ICC_2,1. However, this time we found that the differences between Pearson correlation and ICC_2,1 were substantial, suggesting that (both models) were better at recovering differences between individuals and groups than recovering absolute values. The reason for this discrepancy that was found in Study 2 but not in Study 1, remains unclear. Nevertheless, we noticed that the recovered parameter values in Study 2 were systematically higher than the real values, as in the Supplementary Materials.

It is important to emphasize that the exact value of a parameter seems to often lack practical significance, since in most cases researchers are interested in differences. This implies that the discrepancy between the Pearson correlation and the ICC_2,1 result, while not being fully understood, does not seem to have much relevance.

Another noteworthy aspect regarding the results of Study 2 is the wide confidence interval of the ICC_2,1, occasionally encompassing zero, despite the significance. Apparently, since the p-values and the confidence interval are calculated using different methods, these results may occur.