The discipline of forensic document examination in the United States has primarily utilized a series of opinion terms since an effort to standardize terms was first introduced by McAlexander et al1 and followed by the first published US industry standard, based largely on that original effort,2 There have been various discussions in recent years about modifying the reporting process and some examiners or laboratories have enacted changes. However, the purpose of this treatise is not to dwell on the specific terminology, but rather on the user reaction to instances in which multiple opposing opinions also differing in their degree of qualification, confidence level, or likelihood are proffered. These authors have witnessed instances in which differing opinions have been misrepresented in litigation. The specific instances were not part of open hearings so confidentiality prevents specific citations. In general discussion with other document examiners, the authors consider this problem to be far reaching.
Background
Regardless of the form in which opinions are expressed, there are three basic classes of opinions namely associative, dissociative, and indeterminate. Within the first two classes are multiple terms used to differentiate levels of qualification/ confidence/likelihood. Unqualified opinions are designed to express the highest level of confidence/likelihood. The third class of opinion is the indeterminate opinion – the neutral position.
Seaman Kelly conducted a study of forensic document examiners who had provided indeterminate opinion testimony in court.3 Forty-seven forensic document examiners from Australia, Canada, Israel, South Africa, United Kingdom and the United States submitted their testimony record that included the number of times the examiner testified to an indeterminate opinion, the court jurisdictions, retained by plaintiff/prosecutor or by defense, and the type of case. A total of five hundred and nineteen testimonies were given in court. The breakdown of court jurisdictions where indeterminate opinion testimony was presented are; 245 in criminal court-jury trials, 178 in civil court-jury trials, one UN tribunal, one immigration trial, twelve military court hearings, 62 juvenile court trials, eight bench trials, two administrative hearings, three Daubert hearings, and seven depositions. Forensic document examiners testified for the prosecutor or plaintiff 355 times and for the defense 164 times.
An indeterminate opinion was the only opinion offered as testimony in 103 of the 519 testimonies. The remaining 416 testimonies involved a range of opinions including indeterminate opinions.
The tasks listed by the participants of the poll included 411 tasks involving handwriting, hand printing, signatures; 64 tasks involving the examination of typewriters, printers, copiers, inks, paper, and rubber stamp impressions; 34 tasks involving the examination of indented writing using the ESDA; four tasks involving watermark examinations; and six ftasks involving fracture match examinations.
The participants were asked to list limitations, if any, that led to the indeterminate opinions. The limitations that provided the basis for the opinions were:
Handwriting-little to undefined formation, lack of individuality, or writing is a scribble
Disguised or distorted writing
Lack of adequate or comparable exemplars
Poor copy quality
Poor carbon copy quality
Simulated signature
Graffiti hand printing and numerals
Tracing/cut and paste
Bias Against Indeterminate Opinions
An indeterminate opinion, by its very definition, is one of neutrality. However, this does not mean that an indeterminate opinion is void of significant handwriting characteristic similarities or dissimilarities within the comparison process. Neither is this opinion indicative of confusion or lack of proficiency by the examiner. This opinion is expressed by the examiner for several reasons such as severe limitations as noted above, or by contradictory data none of which is compelling in nature.
There does appear to be an overall bias against indeterminate opinions by users. One National Institute of Justice report, quoted in The Report of the Expert Working Group for Human Factors in Handwriting Examination4 stated:
The findings suggest that jurors tend to over-value some attributes of forensic science expert testimony and under-value other aspects. The most persistent finding is that jurors rely heavily on the ‘experience’ of the testifying expert and the expert’s asserted certainty in conclusions.
The Expert Working Group Report also noted two studies supporting the position that jurors prefer unambiguous testimony that is strongly worded. Ultimately the Expert Working Group Report expresses the cautionary statement:
These findings are a concern because if jurors and others give greater credence to strong opinions that might not be as well reasoned or well founded as more complex, qualified opinions, they may make incorrect decisions on culpability or liability.
Experience has shown that efforts are made by some attorneys to cast indeterminate opinions as the inability of an examiner to complete a task properly. However, the opinion is supported by the evidence when examinations denote severe limitations or conflicting data to the extent that neither the dissociative evidence nor the associative evidence is compelling, et al. As such, it is the data that is the basis for this opinion and not any inability on the part of the examiner. Part of the examiner’s experience in this discipline is to both recognize limitations and accept their results even when the ultimate opinion becomes, “I don’t know”.
Bias involving the summary rejection or minimalization of indeterminate opinions is not only a concern with jurors, but also extends to the judiciary. It can be far easier to form a ruling from information with yes or no opinions than to have to assess the complications of data or rely on more indirect or less analytical (and less definitive) evidence. There are instances in which one must take the time and energy to properly consider the more complex yet accurately assessed data of a more qualified or indeterminate opinion.
However, we would be remiss to not also acknowledge that there are forensic examiners that overly rely on indeterminate opinions. The central theme that users should take the time to consider the intricacies of the data also applies to the forensic document examiner.
Value of Indeterminate Opinions
One may question the value of indeterminate opinions. However, the value should never sway the opinion provided by the examiner. The opinion should solely be based on the evidence.
That said the authors would like to address criticism made as to value. Dror and Langenburg wrote that an indeterminate opinion in a latent print examination is a “broad and imprecise decision” because it has no value.5 However, what is not addressed is whether an indeterminate opinion and the sound foundational basis on which the opinion is reached could have significant value in a matter in which another examiner has opined to a stronger opinion that is not supported by the evidence.
The value of indeterminate opinions, or any opinions for that matter, are case specific. One cannot paint value of such evidence with broad strokes and examiners should give no thought whatsoever as to the impact of the opinion, as that would, in and of itself, be a form of significant bias.
Biedermann, et al, appear to understand the case specific aspect of value by arguing that indeterminate opinions “are no less definite than identification or exclusion decisions.”6
Bias Against the More Qualified Opinions
The bias witnessed in the rejection or minimalization of indeterminate opinions also extends to instances in which opposing associative and dissociative opinions are expressed with varying levels of qualifications. The authors have witnessed instances in which attorneys attempt to trivialize the more qualified opinion over the less qualified or unqualified opinion as some form of proof that their expert is the better expert. Sadly, it has also been the experience of the authors that some judges are receptive to this practice. In one case, a Final Judgment noted:
The Court also heard testimony from an expert called by the Defense, XXXXXXX, who provided a forensic document report that asserted there was a high probability that XXXXX signed the mortgage and only a mere indication that XXXXXXX signed the mortgage.7 (Emphasis added)
Merlino indicates that opinions of lower confidence have higher error rates.8 While this has been studied, the issue referenced in this treatise is different and one should not confuse the two or attempt to use the Merlino study to support lesser opinion bias. The basis for the difference between the Merlino study and the subject matter of this treatise is that the bias of lesser opinion concerns those that have a core class distinction between associative, dissociative, or indeterminate opinions, where the Merlino study addresses opinions within one class of opinion. Merlino’s point is supported by evidence and promotes the reasoning and value behind qualified opinions and their varying degrees of confidence/likelihood.
Countering the Bias
Document examiners can take proactive steps to minimize the chances of the injection of bias based on their reports.
Qualifications – One common differentiation noted by the authors for instances in which there are multiple differing opinions is the qualifications of the examiners. As part of any report, one should consider inclusion of detailed curriculum vitae with generous references to the published training industry standard.
Methodology and Industry Standards – Another common differentiation noted by the authors for instances in which there are multiple differing opinions is the methodologies used, or not used, within the examination. As such, it becomes even more important to provide substantive detail of the procedures utilized, the instrumentation utilized, the reasons for the tests, and the data obtained from each test to include images of the data where possible. The authors have also found that detailed paragraph to paragraph reference to the published industry standard and their application in the examination has proven effective in thwarting any possible attempt at instituting bias.
Limitations – It is also vital that any limitations to an examination is expressed in detail to include the type of limitations, the effect of the limitations, and how each limitation was applied to the assessment and the opinion. The limitations that apply to one’s examination will invariably apply to the opposing examiner. Detailing these limitations will assist in substantiating the qualifications applied.
Data – The most important aspect to any assessment of handwriting comparisons is the data generated from the examination. Forensic document examiners assess data in reaching their opinions and in assessing limitations. The authors have found that greater detail of the data and explanation of its assessment within the report have been effective measures of preventing the potential of injecting bias.
Alternative hypotheses – The authors have noted that the inclusion of a detailed section within a report of applying data to alternative hypotheses has many benefits, one being the opportunity to address potential differing opinions proactively. A report inclusive of this section may address the specific data, limitations, and reasons for the rejection of each individual alternative hypothesis. Care is recommended to ensure that there is great detail in this portion of the report, equal to that of the basis for the actual opinion. In this way, the forensic document examiner can fully address all data, tests, and observations in full and proper context, making future misrepresentations exceedingly more difficult.
Presentation in Court – The authors have noted that the triers of fact respond favorably to well-prepared court presentations that are provided. These presentations should be geared to the non-technical person and walk the fine line between being able to connect with the juror who may have a third grade education and the juror sitting next to him with a PhD. This is not a simple task. The authors recommend that examiners not get stuck on just one form of presentation. As the examination data is case specific, so should the presentation of this data. The authors have found that numerous images each containing one nugget of information has proven fruitful in this endeavor for both its impact and simplicity of understanding, thereby avoiding the problem of information overload based on its small step by small step process. Ultimately, the presentation should shadow the recommendations given for report content to include graphic illustrations of the limitations and why they are so impactful. The authors contend that court presentations of indeterminate or highly qualified conclusions routinely require a greater level of explanation that translates to a more detailed, and often longer, presentation. As with reports the experience of the authors have shown that triers of fact respond favorably to detailed comparison of the examination to the procedures as outlined in industry standards.
Education – The authors urge readers to strive to educate those within the justice system as to the potential of the aforementioned bias and its potential devastating results to the true goal of justice. Specifically, we urge forensic specialists from all disciplines to take every opportunity to discuss this issue with jurists and attorneys when such opportunities arise. We also urge readers to speak up clearly when confronted with this bias. We cannot afford to be meek, nor can we be satisfied with a reactive approach. We need to be proactive and effectively communicate before another problem arises.
Summary
It is important for users of our reports, in particular the judiciary, to be cognizant of potential bias when assessing multiple differing opinions from testifying experts. While it may be an easier road to rely on the opinion that is most definitive, that does not mean that it is the most accurate. Understanding how one reaches an opinion must be the core basis for any assessment of competing hypotheses. The way to approach this assessment is to understand the procedures as noted in industry standards, to review the data collected (or note the lack of said data), and finally to study the method under which the various experts have applied and assessed the data. Necessary detail consists of qualifications, the methodologies used and how they coincide with industry standards, the raw data obtained, and how the assessment process was used to reach the opinions. If any of these details are missing from a report, one should question the expert concerning the details.
Forensic document examiners can proactively address the potential of bias from users by providing thorough reports and court presentations that detail procedures as per industry standards, the data obtained, the process of data assessment, and the limitations in the examination. Supportive illustrations and detailed assessment of alternative hypotheses will also assist in this proactive approach.
Qualified and indeterminate opinions are just as much the gold standard of opinions for examination as unqualified opinions when each are based on sound foundational assessments of the data.
Notes
[1] Note: While there is some debate over the use of the terms conclusions, findings, results, or opinions, this treatise will utilize the term opinion for no other reason than to maintain internal uniformity.