Cognitive Human Factors and Forensic Document Examiner Methods and Procedures: Writing Characteristics, Visual Context, and Handwriting Examination Decision Accuracy

Methods

Materials and Equipment

All eye tracking protocols were conducted with Tobii Pro X2-60® stand-alone eye tracking systems, using Tobii Studio© software (version 3.4.5, Tobii Technology, Stockholm, Sweden). All eye tracking protocols were performed using Dell Precision 7710 laptop computers (Intel Core i7-6820HQ CPUs, 2.7GHz, 8GB RAM, 64-bit Windows 7 operating system, Osprey capture cards).

Signature stimuli. Genuine and simulated signatures were produced by approximately 100 writers on Wacom® Intuos 3 digitizing tablets with Wacom® Intuos 3 inking pens and Neuroscript MovAlyzeR® software. This enabled capture of handwriting speed and pressure data for each signature.

Our FDE subject matter experts classified signature types as either text-based (in which each allograph of the name was clearly written) or stylized (in which one or fewer allographs are legible) [4]. We further classified signatures as either high- or low-complexity using Found and Rogers’ method of evaluating the number of turning points, line intersections, and retrace strokes in a signature [5]. The final 56 stimulus signatures selected for all the eye-tracking experiments in our larger study represented the range of writings FDEs might encounter in casework. These signatures were scanned into a computer using an Epson Expression model 11000XL scanner (600dpi) and Adobe Photoshop© (v. 17). Images were saved as 8-bit grayscale 1024 x 768-pixel jpeg files (10 maximum quality) to enable their display on the eye tracking systems.

Overall, we created five different presentation formats in which we varied the spatial location of the questioned and known signatures. The signatures were presented against a dark gray visual noise background to minimize eye strain from the white screen background. This also reduced the amount of ambient light from the display, allowing participants’ pupils to dilate more fully, improving the eye-tracker’s ability to detect the infrared light reflected from their retinas.

We created two versions of the eye-tracking experiments, in which we counterbalanced the presentation of the questioned signature to control for possible confounding variables. We did this to ensure that the effects we were observing (the number of fixation counts in known signatures when they were adjacent to the questioned signatures) were due to the placement of the questioned signature and not to characteristics of the writing. Questioned signature placement in experiment version 1 was determined by even-odd split so that in all odd-numbered questioned/known comparisons (i.e., 1, 3, 5…19) the questioned signature appeared to the left of the knowns, and in all even-numbered comparisons (i.e., 2, 4, 6…20) the questioned signature appeared to the right of the knowns. In experiment version 2 the placement was reversed so that in the odd-numbered comparisons the questioned signatures appeared to the right of the knowns, and in the even-numbered comparisons appeared to the left of the knowns. Figures 1, 2, 3 present examples of the stimuli for each of the five eyetracking protocols.

Figure 1

Figure 2

Figure 3

Procedures

Eye-tracking protocols were administered under low light conditions in which window coverings were drawn and overhead lights extinguished. The research area was illuminated by a single lamp placed away from the data collection area and shaded to eliminate any glare on the eye- tracker screen. Figure 4 demonstrates the configuration of the eye-tracking equipment during a typical data collection session.

Figure 4

Participants viewed the eye tracker screen from 57 cm away so that the visual angle of the screen was 331 degrees x 271 degrees (W x H). The width of a typical questioned signature subtended a visual angle of approximately 281 degrees. Participants were calibrated to a 9-point reference grid, which provided a resolution of subject gaze to better than 0.5 degree of visual angle. Figure 5 demonstrates the binocular position of the participant’s gaze as detected from the reflection of infrared light from the retina.

Figure 5

Recall that here we are reporting a subset of findings from our larger study. Thus, after calibration, participants completed a total of 20 questioned/known signature comparisons. Participants completed a practice trial demonstrating the format of the signature presentation and the kind of responses required before beginning the experiment.

The comparisons were presented in five sets (trials), consisting of four comparisons per trial. After each comparison, we asked participants to give an opinion about whether the questioned signature was genuine (written by the same person who wrote the known signatures) or simulated (written by different people). We also asked them to indicate their Opinion Strength on the commonly used 9 level scale ranging from Inconclusive to Identification/Elimination. Finally, we asked them to give the certainty of their decision on a 20-point Likert-type scale ranging from 1 (not at all certain) to 20 (extremely certain). The scale levels were unmarked, and Decision Certainty was recorded by mouse click as the experimenter scrolled the mouse over the scale positions until the participant indicated that the cursor had reached their level of certainty. Participants were free to take rest breaks between each of the five trials if they chose.

All 20 comparisons were presented in the Questioned Signature Position format where the position of the questioned signature varied from left to right, as seen in Figure 1 above. Questioned Signature Position stimuli were counterbalanced as described above. Within these 20 comparisons, 10 were presented in the questioned-before-known sequence, and the other 10 comparisons were presented in the known-before-questioned sequence, per the Presentation Sequence format demonstrated in Figure 2 above.

Results and Discussion

Overall decision accuracy. These analyses are based on 1,698 independent observations of the 20 signatures described above. Of these observations, 763 (44.9%) were based on genuine signatures, while 935 (55.1%) were based on freehand simulated signatures.

Overall Decision Accuracy for these signature comparisons was 87.9%. We conducted an initial chi-square analysis to determine whether significant differences in FDE Decision Accuracy occurred for genuine versus simulated signatures (Ground Truth). A greater number of accurate calls were made for simulated signatures (865 of 935, or 92.5%) than for genuine signatures (638 of 763, or 83.6%). This difference was statistically significant, x² (1, N = 1,698) = 41.23, p < .001, indicating that the participants were reliably able to produce accurate decisions, although accuracy was greater for the signatures produced by freehand simulation than for genuine signatures.

We conducted a second chi-square analysis to investigate whether FDE Decision Accuracy varied according to Signature Type. Of the 934 decisions for text-based signatures, 783 (83.8%) were accurate, while of the 764 stylized signature decisions, 710 (92.9%) were accurate. This difference was statistically significant, x²(1, N = 1,698) = 32.78, p < .001. This suggests again that the participants were able to reliably reach accurate decisions for both signature types, although it is possible that some freehand simulations were easier to identify as such due to the skill level of the simulators.

We conducted a third chi-square analysis to determine whether FDE Decision Accuracy varied according to Signature Complexity. Of the 849 decisions on high-complexity signatures, 808 (95.2%) were accurate. Of the 849 decisions on low-complexity signatures, 685 (80.7%) were accurate. This difference was statistically significant, x² (1, N = 1,698) = 83.93, p < .001. Accuracy rates for both levels of signature complexity were high, but the lower decision accuracy for low complexity signatures highlights the importance of the amount of detail present in the writing samples.

We conducted a fourth chi-square analysis to determine whether FDE Decision Accuracy varied according to Questioned Signature Position. Of the 849 comparisons when the questioned signature appeared at the left, 747 (88.0%) of FDE decisions were accurate. Of the 849 comparisons when the questioned signature appeared at the right, 746 (87.9%) of FDE decisions were accurately called. This difference was not statistically significant (p = .941). This suggests that alternating the visual flow of the examination from the left-to-right (questioned to known) reading pattern typical of U.S., Canadian, and Australian readers to a right-to-left (known to questioned) visual flow does not impact the accuracy of the examination outcome.

We conducted a final chi-square analysis to determine whether FDE Decision Accuracy in these 20 comparisons varied according to whether the questioned or the known signature was presented first (Presentation Sequence). Of the 849 comparisons in which the questioned signature was presented before the questioned/known comparison, 688 (81.0%) of FDE decisions were accurate. Of the 849 comparisons in which the known signatures were presented before the questioned/known comparison, 805 decisions (94.8%) were accurate. This difference was statistically significant, x2 (1, N = 1,698) = 75.94, p < .001. This finding is particularly interesting because it appears contrary to current training and practice standards recommending that examiners study the questioned writing before the known exemplars. Recall, however, that the presentation sequence was counterbalanced such that all signatures were presented in both the known-before-comparison and the questioned-before-comparison format; thus, this difference indicates that the order of presentation did affect Decision Accuracy.

Utilization of known signatures. We conducted independent groups t-tests to compare the mean number of fixations in the each of the six known signatures of the comparisons. We were interested in learning whether there was a difference in FDE utilization of the known signatures related to characteristics of the signatures (Signature Type, Signature Complexity, Ground Truth, Decision Accuracy) or the context of the presentation of the questioned and known signatures (questioned-before-known vs. known- before-questioned presentation, and questioned signature position).

We also performed a series of binomial logistic regression analyses to identify whether differences in mean fixation counts in each of the six known signature areas of interest predicted Signature Type (text-based or stylized), Signature Complexity (high or low), Ground Truth (genuine or freehand simulation), Decision Accuracy (accurate or misleading), Signature Presentation Sequence (questioned-before-known or known-before-questioned), or Questioned Signature Position (questioned at the left or questioned at the right).

Ground Truth and Decision Accuracy analyses. Section A of Table 1 presents the results for the six independent groups t-tests comparing the mean number of fixations on each known signature, to investigate whether the average number of fixations was different for genuine and simulated signatures.

The p values indicate that the mean fixation count for all known signatures was statistically significantly different depending on whether the questioned signature was genuine or simulated (Ground Truth). Section B of the table demonstrates that the mean fixation count for all known signatures was statistically significantly different depending on whether the FDE’s decision was accurate or misleading (Decision Accuracy). Fixation counts in the known signatures were higher when signatures were genuine, as well as when participants’ decisions were misleading. This suggests that participants may have referenced the knowns to a greater extent when the questioned and known signatures were more similar, possibly leading to over-interpretation of the writing features and misleading opinions.

Figure 6 presents the distribution of the mean fixation counts by known signature.

Figure 6

Significant predictors of Ground Truth and Decision Accuracy. We combined the variables Known Signature 1, Known Signature 2, Known Signature 3, Known Signature 4, Known Signature 5, and Known Signature 6 in a binomial logistic regression model to test how accurate the six factors together were in predicting whether the Ground Truth was genuine or simulated (the outcome variable). All logistic regression analyses were conducted using the enter method. Table 2 presents the regression coefficients for these analyses. This allowed us to investigate which of the known signatures in the comparisons may have been more influential.

The overall regression model was statistically significant (x² (6) = 18.66, p = .005). Although statistically significant, the variables in the model did not improve the amount of variability explained (Nagelkerke r² = 2.1%). Results indicated that the overall model fit was poor (–2 Log Likelihood = 1232.35). The model was statistically reliable in classifying 63.4% of the observations. Wald statistics indicated that when the variables were combined, only Known Signatures, which received the highest number of fixations, and Known Signature 6, which received the lowest number of fixations, predicted the category of Ground Truth. Although the means for the number of fixations in each of the six signatures were significantly higher when the questioned signatures were genuine, those in Known Signature 1 and Known Signature 6 were high enough to suggest that participants were utilizing all available information to a greater extent when the signatures were genuine than when the questioned signatures were simulated.

We combined the same variables in a binomial logistic regression model to test how accurately the six factors together were in predicting whether the decision was accurate or misleading (the outcome variable).

The overall model was statistically significant (x² (6) = 18.66, p = .005). Although statistically significant, the variables in the model did not improve the amount of variability explained (Nagelkerke r² = 2.1%). Results indicated that the overall model fit was poor (-2 Log Likelihood = 1232.35), although the model was statistically reliable in classifying 87.8% of the observations. Wald statistics indicated that when the variables were combined, none predicted the category of Decision Accuracy. The number of fixations in each of the six signatures was significantly higher when the decision was misleading, but none of the known signatures appeared to have had more influential characteristics than the others.

Utilization of known signatures in Signature Type and Signature Complexity analyses. We conducted independent groups t-tests to compare the mean number of fixations in the each of the six known signatures of the comparisons. Here we were interested in learning whether there was a difference in FDE utilization of the known signatures related to Signature Type (text-based or stylized) or Signature Complexity (high or low). We also performed a series of binomial logistic regression analyses to identify whether differences in mean fixation counts in each of the six known signature areas of interest predicted Signature Type or Signature Complexity.

Section A of Table 3 presents the results for the six independent groups t-tests comparing the mean number of fixations on each known signature for text-based and stylized signatures.

The p values indicate that the mean fixation counts for all known signature were statistically significantly different depending on whether the signatures were text-based or stylized (Signature Type). Higher means among the text-based signatures suggest that participants may have fixated on a greater number of legible allographic features than were available in the stylized signatures. Section B of the table demonstrates that there were no significant differences in fixation counts among the known signatures according to signature complexity (Signature Complexity).

We conducted a 2 (Signature Type) x 2 (Signature Complexity) factorial analysis of variance (ANOVA) to further investigate whether these combined factors provided additional information about these relationships. We found that fixation counts were indeed higher for both high and low complexity text-based signatures (p < .001). We also found that fixation counts were lowest among high complexity, stylized signatures (p < .001). This supports the idea that the number of legible allographs is related to the number of fixations. It also suggests that pictorial differences among high complexity stylized signatures may have been relatively more salient to the participants, requiring less attention to accurately discern. Figure 7 presents the distribution of the mean fixation counts by known signature.

Figure 7

Significant predictors of Signature Type and Signature Complexity. We combined the fixation count variables (Known Signatures 1-6) in a binomial logistic regression model to test how accurate the six factors together were in predicting whether the Signature Type was text-based or stylized (the outcome variable). Table 4 presents the regression coefficients for these analyses.

The overall model was statistically significant (x² (6) = 34.33, p < .001), indicating that the variables improved the amount of variability explained (Nagelkerke r² = 2.7%). Results indicated that the overall model fit was poor (-2 Log Likelihood = 2302.55). The model correctly classified 53.1% of cases. Wald statistics indicated that when the variables were combined, the only predictor variable to reach significance was Known Signature 4. The odds ratio for Known Signature 4 was small, indicating low impact on the outcome.

We combined the same variables in a binomial logistic regression model to test how accurately the six factors together predicted Signature Complexity (the outcome variable). The overall model was not statistically significant (x² (6) = 8.02, p = .237), indicating that the variables did not improve the amount of variability explained (Nagelkerke r² = 0.06%). The model correctly classified only 51.61% of cases. Wald statistics indicated that when the variables were combined, none predicted whether the signatures were high or low complexity.

Utilization of known signatures in Signature Presentation Sequence and Questioned Signature Position analyses. We conducted independent groups t-tests to compare the mean number of fixations in the each of the six known signatures of the comparisons. Here we were interested in learning whether there was a difference in FDE utilization of the known signatures related to Signature Presentation Sequence (questioned- before-known or known-before-questioned) or Questioned Signature Position (questioned at the left or questioned at the right). We also performed binomial logistic regression analyses to identify whether differences in mean fixation counts in each of the six known signature areas of interest predicted Signature Presentation Sequence or Questioned Signature Position. Section A of Table 5 presents the results for the six independent groups t-tests comparing the mean number of fixations on each known signature for questioned-before-known or known-before-questioned sequences.

The p values indicated that the mean fixation count for every known signature was statistically significantly different depending on whether the presentation sequence was questioned-before- known or known-before-questioned (Presentation Sequence).

We followed this analysis with a paired groups t-test to investigate whether the total number of fixations on the known signatures was related to the lower number of known signature fixations on the comparison stimulus (known-before-questioned sequence). We found that the mean total fixation count for the knowns (11.89) was significantly lower than that for the questioned/known comparisons (51.74), t (843) = -19.29, p < .001, (95% CI = -43.91, -35.80). A bivariate Pearson’s Product Moment correlation showed that the total fixation count for the known signatures that were viewed alone was significantly positively correlated to the number of fixations on the comparison stimulus (r² = .212, p < .001). This indicates that if the number of fixations on the knowns alone was high, the number of fixations on the questioned/known comparison was also high, suggesting that differences were related to the participants’ individual examination habits.

We did a similar analysis to investigate whether this pattern was also present when the questioned signature was presented first, followed by the questioned/known comparison stimulus. The mean fixation counts for the questioned signature alone and the questioned signature on the comparison stimulus were not statistically significantly different (p = .175). Here, too, we found a significant positive correlation between the total number of fixations on both stimuli in this comparison (r² = .228, p < .001), suggesting a pattern of individual examination habits similar to that in the known-before-questioned sequence.

Section B of the table demonstrates that all mean fixation counts except known signature 6 were significantly different depending on whether the questioned signature was presented at the left or the right of the known signatures (Questioned Signature Position). Figure 8 presents the distribution of the mean fixation counts by known signature.

Figure 8

Significant predictors of Presentation Sequence and Questioned Signature Position. The results of these analyses are presented in Table 6. We combined the variables Known Signature 1, Known Signature 2, Known Signature 3, Known Signature 4, Known Signature 5, and Known Signature 6 in a binomial logistic regression model to test how accurately the six factors together predicted whether the presentation sequence was questioned-before-known or known-before-questioned (the outcome variable). The overall model was statistically significant (x² (6) = 111.42, p < .001). The variables in the model improved the amount of variability explained (Nagelkerke r² = 8.5%). Results indicated that the overall model fit was poor (-2 Log Likelihood 2242.51). The model correctly classified 59.4% of cases. Wald statistics indicated that the only significant fixation count predictor of Questioned/Known Presentation Sequence was Known Signature 4, but the odds ratio was small, indicating low impact on the outcome.

We performed a second binary logistic regression to investigate whether the observed differences in fixation count among the known signatures were related to whether the questioned signature was located on the left or on the right (the outcome variable). The overall model was statistically significant (x² (6) 615.13, p < .001). The variables in the model improved the amount of variability explained (Nagelkerke r² 40.5%). Results indicated that the overall model fit was poor (-2 Log Likelihood 1739.79). The model was statistically reliable, correctly classifying 79.3% of cases. Wald statistics indicated that four of the six variables in the model (Known Signature 1, Known Signature 2, Known Signature 3, and Known Signature 4) were significant predictors of questioned signature location, but the odds ratios for all were small, indicating low impact on the outcome.

Predictors of Decision Accuracy: Combined Signature Characteristic and Fixation Count Analyses. Finally, we combined all the predictor variables (Signature Type, Signature Complexity, Presentation Sequence, Questioned Signature Position, and all Known Signature fixation count variables) in a binomial logistic regression model to investigate which factors together predicted whether the participant decisions were accurate or misleading (the outcome variable). The results of this analysis are presented in Table 7.

The overall model was statistically significant (x² (10) = 255.79, p < .001), indicating that the variables improved the amount of variability explained (Nagelkerke r² = 26.8%). Results indicated that the overall model fit was poor (–2 Log Likelihood = 995.22). The model correctly classified 87.8% of cases. Wald statistics indicated that when the variables were combined, only Known Signature 6 significantly predicted Decision Accuracy. Among the remaining variables, Signature Type, Signature Complexity, and Presentation Sequence were also significant predictors of the outcome. The large odds ratios for Signature Type (3.65) and Presentation Sequence (2.73) indicated substantial change in the likelihood of Decision Accuracy. Odds ratios near 1.00 indicated that Signature Complexity and the number of fixations in Known Signature 6 had little impact on the outcome, even though both reached statistical significance in the model.

In Summary

Ground truth. Our participants were reliably able to produce accurate decisions, although accuracy was greater for freehand simulation signatures than for genuine signatures. Genuine signatures attracted a greater number of fixations among the known signatures, and fixations were greater among the knowns when the decisions were misleading than when they were accurate. Participants may have referenced the knowns to a greater extent when the questioned and known signatures were more similar, leading them to over-interpret the writing features and contributing to the number of misleading opinions.

Decision accuracy. When the Known Signature fixation count variables were combined, none predicted Decision Accuracy. Although the means for the number of fixations in each of the six signatures were significantly higher when the decision was misleading, none of the characteristics of the known signatures appear to have been more influential than the others.

The mean number of fixations in each of the six known signatures was significantly higher when the questioned signatures were genuine. Fixation counts in Known Signatures 1 and 6 were high enough to suggest that participants were utilizing all available information in the known signatures to a greater extent when the questioned signatures were genuine than when they were simulated.

Signature type. Participants reliably reached accurate decisions for both signature types, although it is possible that some freehand simulations were easier to identify as such due to the skill level of the simulators. Higher mean fixation counts among the text-based signatures suggested that participants may have accessed a greater number of legible allographic features than there were available in the stylized signatures.

Signature complexity. Accuracy rates for high and low signature complexity were high, but the lower decision accuracy for low complexity signatures highlighted the importance of the amount of detail present in the writing samples. Fixation counts were lowest among high complexity simulated signatures. In this case, pictorial differences among high complexity stylized signatures may have been relatively more salient to the participants, requiring less attention to accurately discern.

Questioned signature position. Alternating the visual flow of the examination from the left- to-right (questioned-to-known) reading pattern typical of U.S., Canadian, and Australian readers, to a right-to-left (known-to-questioned) visual flow did not impact the accuracy of the examination outcome.

Presentation sequence. Current training and practice standards recommend that examiners should study the questioned writings prior to examining the known exemplars. Here we found that the questioned-before-known presentation sequence attracted a higher number of misleading decisions than did the known-before-questioned sequence. We can reasonably conclude in this case that the order of presentation did affect Decision Accuracy because of the counterbalanced experimental protocols. Presentation sequence was counterbalanced such that all 20 signature comparisons were presented in both the known- before-comparison and the questioned-before- comparison format.

The known-before-questioned presentation is contrary not only to current document examination practice recommendations, but also to the recommendations of several highly respected statisticians, who cite statistical models that predict a greater likelihood for cognitive bias if the known signatures are viewed first. Although our results are informative, no single study should be considered dispositive. Training and practice in any field should be informed by a body of methodologically sound research, leading to reliable and evidence-based best practices.

Study Limitations and Future Research Directions

Good experimental research is conducted in a carefully controlled environment, under standardized conditions that allow the researchers to isolate the effects of the factors that they are studying. The experimental environment necessary to ensure the quality of the data provided by the X2-60 eye-tracking units did not, and cannot, approximate the laboratory environment in which most handwriting examinations occur. We anticipate that the results of the comparisons performed on the eye-tracking systems would be somewhat different if our participants had access to the tools and techniques available to them in a document examination laboratory.

The experimental nature of this study may also have an impact on the opinion strength expressed by our participants. Although we have data about opinion certainty and confidence for these comparisons, we have chosen not to include them here for space considerations. We will report them more fully in the future.

It is important that readers recognize the difference between experimental protocols and live casework. We must note that forensic document examiner decisions in practice are not forced- choice decisions. As part of our efforts to study various aspects of “sufficiency,” examiners were also asked to make several decisions about writing samples that contained minimal information on which to base an opinion. In practice, it is unlikely that examiners would form opinions based on such minimal information. Our ground truth study should therefore not be considered a proficiency test of any kind. Future studies should take place in conditions that more closely approximate the working environment of a laboratory.

Conclusion

Avoiding the grave consequences of erroneous conclusions in live casework requires effort, resources, and commitment by all parties in the legal process. Although the experimental nature of this research limits to an extent the generalizability of the results, experimental studies such as this inform the field in many useful and important ways. Research provides the empirical support to assert that proposed practices are the best practices. Research forms the basis for developing empirically validated and rigorous document examination protocols, measures, and education and training programs that consistently and comprehensively address the knowledge and skills required to establish expertise in forensic fields.

Eye-tracking methodology, physiological data, information about how examiners use the evidential features in handwriting examinations, knowledge about the decision making process all help to achieve these goals by increasing the transparency of the examination process, and improving the quality of performance of attorneys, judges, and experts.