Computer-Scored Test Answer Marks, aka Bubble Marks: How Individual Are They?

Introduction

Recent research by three computer scientists has focused on the use of computer technology to assess authorship of “bubble marks” on documents.1 These answer marks, also sometimes called grid marks, are made on standardized tests, surveys, and ballots. In using these forms, a writer typically makes a pencil mark (often within a printed oval or circle) to indicate the answer to a question. The research conducted by these computer scientists has attempted to assess whether the answer marks on standardized documents are truly anonymous (as in the case of surveys and ballots) and whether the author of such marks can be determined (as related to numerous recent and on-going allegations of cheating on standardized tests). The report of this research was presented in 2011 and prompted the research presented here.

Over the years, many document examiners have inspected standardized tests on which handwritten answer marks are used. In these cases, there is commonly a concern about whether cheating has occurred. This cheating can take many forms including, but not limited to, use of impersonators (that is, a student sending in a proxy or impersonator to take the exam on his/ her behalf), swapping of tests and answer sheets by two persons in the same room, making of after-the-fact changes to specific answers or completing unanswered blanks on an answer sheet, in order to increase a student’s or school’s score. A survey of newspaper articles written over the last few years reveals numerous reports of alleged large-scale cheating on standardized tests across the country.

Typically when a document examiner receives one or more standardized answer sheets for examination, the document contains at least some general handwriting, possibly one or more signatures, and often handwritten numeric calculations. Examination of this writing is a typical, straightforward process discussed in many textbooks on document examination and covered by relevant standards in the field. Furthermore, considerable research has been conducted regarding identification of signatures, hand printing, and numerals. However, the literature on document examination contains little if any discussion of the limited subject of handwritten “bubble marks” on standardized tests as well as those on ballots and surveys. Yet sometimes the question about a test answer sheet, survey, or ballot can only be resolved by the evidence in the marks themselves.

Huber and Headrick discuss the examination of various marks (described as dingbats and iconic symbols) as follows:

While answer marks like those studied in this project are simple and unlike more complex letters or numerals, they are markings made with a writing instrument on paper by human hand. They are essentially handwritten forms and, as such, are handwriting. They can be compared to simplified and unreadable signatures or to circular letter designs (for example, an “o” or “O”). While each mark is limited in amount, on most answer sheets the writer is required to make many such marks. This repetition allows an examiner to determine whether there is a pattern to a writer’s answer marks and the manner and extent to which the writer varies his or her mark.

The Study Conducted

For the purpose of examining answer marks by different writers, an answer sheet was first designed using a readily-available online form. The answer sheet for the study, prepared using the Catpin Bubble Test Generator, has three columns and allows for answers to 50 total questions (20 in each of the first two columns and 10 in the third column. The answer for each question is to be marked in one of four ovals. (See attached illustration.)

Each participant was given and asked to complete three answer sheets, marked A, B and C respectively. For the first (answer sheet A), a set of 50 questions was created. The survey was anonymous, although the first ten questions were about the participant’s age category, handedness, writing education, and writing instrument preference. The remaining questions were general and of no significance to the study itself. For example, questions 15 and 16 were as follows:

• What kind of book do you prefer to read?

• a. Biographies

• b. Cookbooks

• c. Mysteries and thrillers

• d. None of the above

• What is your favorite season?

• a. Spring

• b. Summer

• c. Autumn

• d. Winter

For answer sheet A, the participants were given the following printed instructions:

For answer sheet B, the participant was given no questions but was simply asked to fill in one oval for each of the 50 numbered items. Then, (s)he was asked to erase specific marks and to make a new mark in a different oval for each of those items.

The instructions given for this section were as follows:

For the third part of the study, the answer marks on answer sheet C had already been completed by the author, using a pattern of marking answers in a sequence of B, C, D, B, C, D, etc. The participant was then asked to erase 14 specific marks on the answer sheet and change each of these 14 answers to “A.” These instructions are as follows:

The answer sheets were all printed in color with an ink jet printer on 20 lb. white copy paper. In addition to a set of three answer sheets, the participants were provided with sharpened No. 2 pencils with attached erasers.

Examination Methods and Techniques used

A macroscopic and microscopic examination was conducted of each answer sheet. Each page was examined with grazing light on both front and back. In addition, the pages were examined with incident light. Individual marks were examined with grazing light from various angles, reflected light, incident light and transmitted light.

For the purpose of determining the direction of the stroke forming the mark, examination with transmitted light was found to be the most effective technique. Each mark was examined microscopically with transmitted light and at various degrees of enlargement from 10X to 140X. Generally the higher degree of enlargement was the most useful for determining the stroke direction of the mark, and a lesser degree of enlargement was more useful for studying the shape of the mark. In addition, examination of the sheet from the back was useful in determining both the shape of the mark and the extent to which the mark indented into the paper.

In determining the direction of a stroke forming a mark – a key component of the examination, the most useful examinations were conducted with transmitted light and high-power magnification (over 100X). In addition, use of a transmitted light box with a bright light is necessary. Typically, pencil graphite will abrade and accumulate on the leading edge of paper fibers, revealing the direction in which the pencil was moving. While not being entirely analogous, this is somewhat akin to wind-driven snow or sand piling up along a fence line.

In order to make an assessment of the stroke direction, the examiner needs to examine the mark overall as well as in quadrants. The best evidence of stroke direction is often found near the edges of the mark rather than in the middle where there will usually be numerous overlapping lines. In addition, the nature of the mark being made must be considered. That is, even though a stroke is near the top of an oval, it may be at the bottom of a circular movement. This is because, in an attempt to completely fill an oval, some writers will make a series of small overlapping circular movements. There is no assurance that the line of separation between the top and bottom of a circular stroke forming the mark is actually in the middle of the printed oval. The examiner should carefully follow the stroke as much as possible and verify the stroke direction in several locations within the mark.

On each answer sheet (A, B and C for each participant) an examination was conducted to determine whether the writer had a pattern in making answer marks and whether the marks had the level of individuality and complexity required to make them identifiable as the writing of a single person. Particularly studied were:

1. the direction of the stroke forming themark (clockwise, counterclockwise, a combination of clockwise and counterclockwise, or back and forth)

2. the shape of each mark

3. the conformity to the printed oval

4. the presence of spurs or tails extending outside the mark

5. the writing pressure (or how much the mark indented into the paper)

6. the presence of voids within the mark and whether there is a pattern to these voids

7. the density of the graphite layer and/or its visual appearance

When completing answer sheet A the writer was not asked to erase any marks. However, the presence of any erasures was noted and the “new” marks were examined to determine if they were different in any way from the unchanged marks.

On answer sheet B, the page was also examined to ascertain whether the requested erasures and rewriting actually had been done and, if so, whether the “new” marks were different in any way from the unchanged marks.

On answer sheet C, the page was examined to ascertain whether the requested erasures and rewriting had been done and, if so, whether the “new” marks, which were known to have been by a different writer from the “original” marks, were significantly different in any way from the marks made by the primary writer (in this case, the author). The “new” marks on answer sheet C were also compared to the marks by this writer on documents A and B to determine their consistency.

The Participants

Answer sheets were completed by a total of 98 participants from Houston, Victoria, Rockport, and Weimar, Texas, Tulsa, Oklahoma, Bella Vista, Arkansas, and Columbus, Ohio. While all 98 completed page A, some writers did not complete either page B or page C of the project. As a result, there were only 94 completed answer sheets for section B and 93 completed answer sheets for section C.

The composition and preferences of the 98 participants were as follows:

Age

• 6 participants were age 14 or under;

• 10 participants were between ages 15 and 24;

• 25 participants were between ages 25 and 34;

• 57 participants were age 35 or over.

Handedness:

• 85 were right handed;

• 11 were left handed;

• 2 reported being ambidextrous.

Location of writing education:

• 12 learned to write in Houston, Texas;

• 21 learned to write in Texas but not in Houston;

• 58 learned to write in the United States but outside of Texas;

• 7 learned to write outside of the United States.

Who taught the participant to write:

• 5 were taught by their fathers;

• 25 were taught by their mothers;

• 4 were taught by another relative;

• 64 were taught at school by a teacher.

Preferred writing style:

• 33 mostly write in hand printing;

• 24 mostly write in cursive;

• 40 use a combination of hand printing and cursive;

• 1 mostly writes in some other form.

Preferred writing instrument:

• 22 prefer using a pencil;

• 59 prefer using a ball point pen;

• 11 prefer using a fiber tip pen;

• 6 prefer using another kind of writing instrument.

Results

Shape

Most of the participants made an answer mark that was circular in movement and commonly oval in shape. That is, they made an effort to conform to the shape of the printed mark. A smaller number (6) used a pattern of straight lines that were vertical, diagonal, or horizontal in nature. Twelve participants also varied between straight lines and more circular oval forms, with the oval form being predominant for these writers.

The conformity of the mark to the printed oval

Sixteen participants made erratic marks. In many instances, this feature involved a writer often extending the mark beyond the edges of the printed oval. However, in one instance, the participant made erratic, “messy” marks that stayed well within the oval.

The remainder of the participants made marks that typically did not extend beyond the printed oval and in fact were consistent in general shape and quality. For the purpose of this determination, if a writer only extended a mark slightly outside the printed line on three or four instances within a group of 50 marks, (s)he was not considered to have a pattern of extending beyond the printed line.

Presence of spurs or tails extending outside the mark

Fifteen participants were found to have a spur or tail within a few marks.

The Writing Pressure

For this examination, the answer sheet was examined on the back with grazing light. If the mark almost could not be detected with this examination, the mark was designated as light pressure. If the mark indented heavily into the paper surface, it was designated as heavy pressure.

Fifty-one of the writers used light to light/medium pressure in making the answer marks. Thirteen of the writers used heavy pressure. The remaining 34 writers used a medium pressure.

While most writers had some variation in pressure from one mark to the next, they were consistent in the general amount of pressure. That is, if a writer typically made marks with heavy pressure, (s)he may have made a few that were of a more moderate pressure but none that were light; if a writer typically made marks with very light pressure, (s)he may have made a few with slightly heavier pressure but none that were markedly heavy.

There were two exceptions to this finding. In both instances, the writer made some marks with light pressure and others with heavy pressure. However, each writer did not vary randomly but rather made marks with similar pressure within groups or as part of a series (for instance, making one column heavy and the next light).

Illustrated above are marks made by a single participant on the same answer sheet. Note the used of different pressure in different columns.

Density of the graphite layer and/or its visual appearance

For marks made by a single writer, the density or darkness of the graphite typically remained consistent within one answer sheet. However, on answer sheets C (on which the changed marks were by the participant and the unchanged marks were by the author), there was sometimes a clear difference between the darkness and the sheen of the marks made by the two different writers.

Erasures and new marks made after erasures on answer sheet C

On answer sheet C, an answer mark had been placed on each line prior to the sheet being provided to the participants. These marks were all written by the author, and the marks were oval in shape, relatively constant in design, and clockwise in direction. On each page, the marks were similar in the amount of pressure used to write the marks, although there was variation from one page to the next. This particularly reflects the fact these sheets were prepared in several groups or writing events, with the different groups being prepared under different conditions (such as in stacks or not, with different pencils, on different writing surfaces).

Once the participant received his or her answer sheets, (s)he was asked to erase 14 of the answers on sheet C and mark a different answer for each of these erased items. Ninety-three participants completed this portion of the project.

In 89 instances, there was at least some difference between the marks made by the author and those made by the study participant (or second writer). Many of the answer sheets had multiple differences between the first and second writers. These differences are as follows:

On 64 answer sheets for part C (69% of those completed), the changed marks were counterclockwise while the unchanged marks were clockwise in direction; on the remaining 29 answer sheets in this part of the study, the changed marks were clockwise in direction, were indeterminate in direction, were a zigzag design, or included new marks in both clockwise and counterclockwise directions. Essentially, on 69% of the answer sheets in this portion of the study, the new marks made by the participant could be clearly differentiated from those written by the author based on a fundamental difference in the direction of the stroke used to form the mark.

However, it is important to note that this percentage would be significantly lower if the original marks had been counterclockwise in direction, since the more-commonly-used stroke direction in making marks is counterclockwise.

On 58 answer sheets (62.4%) the pressure in the changed marks was different from that in the unchanged marks;

On 17 answer sheets the darkness of the marks (that is, the density of the graphite and the sheen of the mark) was different on the changed marks from that of the unchanged marks;

On 16 answer sheets, the shape of the changed marks differed from that of the unchanged marks;

On 4 answer sheets the size of the changed marks differed from that of the unchanged marks.

As was noted earlier, it is likely that writing pressure and density of the graphite are not entirely independent factors. Therefore, these two characteristics cannot be considered as two separate features of equal weight. In addition, the differences in shape and size of the marks found in this study were not profound and are supplemental rather than primary evidence of a different writer. On the other hand, a pattern of difference in the direction of the stroke forming an answer mark is significant.

When certain marks on an answer sheet have been erased and changed, and the new or changed marks are persistently written in the opposite stroke direction from the unchanged marks, this fundamental difference between changed and unchanged marks is persuasive evidence of a different writer. The evidence is even more persuasive when supplemented by the same pattern of differences in pressure, shape, size, darkness or some other feature.

Transmitted light photographs show the difference between the stroke direction of the unchanged mark (left) and that of the changed mark (right)—marks by two different writers on a single answer sheet.

Answer Sheet 118C, answer marks 41–50. The marks in 41–44 are original, unchanged marks. Those now appearing in 45–50 have been changed by a different writer. Note the difference in the pressure used by the two writers, as seen on the back of the page. The entries in red boxes are those by the second writer.

Conclusion

The majority of writers had a pattern and consistency to the manner in which they made answer marks in this project. And when a writer had a relatively wide range of variation (and essentially followed little if any pattern in making marks), it quickly became apparent when examining the subtle features of the marks as described above.

There were certain features that were repeated in the marks made by numerous writers. The most common pattern in writing these marks was a carefully-drawn circular mark written in a counterclockwise direction, filling the printed oval and only occasionally veering outside the printed line. When comparing answer marks by different writers who used this style, there was no evidence on which to base an opinion that they were in fact by different writers because they lack distinctive features.

None of the marks produced during this study had the kind of individuality needed to determine authorship, to the exclusion of all other writers. On the other hand, in many instances there was clear evidence that two sets of answer sheets were not by the same writer. While there were often supplemental features such as writing pressure, shape, size, and density or darkness of the marks that support a conclusion of a different writer on two sets of answer marks, the most significant feature that provided strong evidence of different writers is a difference in the direction of the stroke forming the marks. When numerous changes were made on an answer sheet, and there is a pattern of fundamental differences between the changed marks and the unchanged marks, the evidence of two writers is clear. As was noted by Dan Purdy in Scientific Examination of Questioned Documents, “if two documents were not prepared by the same writer, this can often be established by the presence of significant or irreconcilable differences.”3 That is the case with answer marks as well.