Figure 1
The Zooniverse project homepage.
Figure 2
Date range of illustrations for each collection.
Figure 3
Distribution of illustrations across languages in the International Children’s Digital Library collection.
Table 1
Project statistics for Picturing Stories.
| CATEGORY | TOTAL |
|---|---|
| Weeks | 42 |
| Participants (All)ᵃ | 10,647 |
| Participants (w/ID)ᵇ | 6,861 |
| Top Scientists (%)ᶜ | 17.6 |
| Single Visitors (%)ᵈ | 29.7 |
| (lower bound) | (24.8) |
| Books/Illustrations: IA | 2,506/27,878 |
| Books/Illustrations: ICDL | 882/16,165 |
| Captions (Total/Avg. Length in words) | 79,375/20.7 |
| All Annotations | 1,350,626 |
[i] ᵃ Participants (All) are calculated by including both logged-in and anonymous contributors and may slightly overestimate unique individuals due to session-based IDs. They cannot be aggregated across projects for this reason.
ᵇ Participants (w/ID) are calculated using a persistent User ID, which tracks logged-in accounts across projects. However, it excludes anonymous contributors and therefore represents a lower-bound estimate of unique participants.
ᶜ Top Scientists is the percentage of participants who are responsible for completing 80% of all annotations.
ᵈ Single Visitors is the percentage of all participants who only complete five or fewer annotations. Lower bound uses only logged-in User IDs for a more conservative estimate.
Table 2
Our annotation workflow across nine separate tasks.
| TASK | CATEGORY | QUESTION | POSSIBLE ANSWERS |
|---|---|---|---|
| T0 | Illustration | Is there an illustration on this page? | Yes, No |
| T1 | Characters | Who are the characters on the page? Check all that apply. | Man, Woman, Girl, Boy, Animal, Animate Object (e.g., a teddy bear or spoon who talks), Mythical/Imaginary Being, Other, None |
| T2 | Setting | Where is the scene set? | Indoors, Outdoors, Both/Ambiguous |
| T3 | Objects | What are the main objects in the scene? Check all that apply. | Nature (trees, flowers, lakes), Architecture (buildings, walls, furniture), Vehicles (cars, boats, planes), Toys, Reading and Art (books, musical instruments, paintings), Household items and tools (pots, pans, hammers, sticks), Other, None |
| T4 | Intensity | How emotionally intense is this scene? What is the energy like? | 1- Very calm (still, contemplative, restful), 2- Mildly calm, 3- Neutral, 4- Mildly intense, 5- Very intense (frantic, tense, active) |
| T5 | Actions | Choose an entity type in the image. What behaviors do you see? | sitting, standing, walking, talking, looking, pointing, gesturing, holding hands or embracing (someone), being embraced, holding (something not someone), giving, playing, working, eating/drinking, reading, listening, crying, sleeping, moving actively (running, jumping, dancing, climbing, flying, etc), animal noise, violent action |
| T6 | Family | Does the image appear to depict a family? For there to be a family there must be adults AND children present who appear to be related based on their actions or dress. | Yes/No |
| T7 | Family | What are the family members doing in this image? Check all that apply for all family members. | Embracing/Physical closeness; Comforting/Caregiving; Playing; Recreational Activity (Walking, Picnic, Outing); Teaching/Learning; Communicating/Pointing; Reading; Eating; Household Work; Outdoor Labor; Co-present but not interacting; I cannot tell from the image |
| T8 | Captions | Please write a caption for this illustration. Make sure to include key characters, objects, actions, and aspects of the setting. Write 1–2 full sentences and be as objective as possible. | “Three small children rest on hay: a boy holding a rake stands between a seated girl holding a flower and another boy lying down.” || “Three children play in the field, one is laying, one is sitting and holding a flower, the third is standing and holding a stick.” |
Figure 4
Sample screenshots of annotation workflows (T0–T4) moving left to right beginning in the upper-left. The illustration would be present for each panel.
Table 3
Inter-annotator agreement across task types, reported separately for the two collections as IA/ICDL. Agreement is computed per Zooniverse subject among the annotators who answered that task, restricted to subjects with at least three independent annotators. The Type column gives the answer type together with the total number of choices available. Full agreement is the proportion of subjects on which every annotator gave the identical answer. Majority is the proportion on which the modal answer was given by at least half of the annotators. Jaccard index is the mean, over annotator pairs, of the size of the intersection of their answer sets divided by the size of their union. Containment is the mean, over annotator pairs, of the size of the intersection divided by the size of the smaller set. Ordinal distance is an average deviation index.
| TASK | NAME | TYPE | N SUBJECTS | FULL AGREEMENT | MAJORITY | JACCARD | CONTAINMENT | ORDINAL DISTANCE |
|---|---|---|---|---|---|---|---|---|
| T0 | Illustration | Single (2) | 31040/35390 | 85.3/66.5 | –/– | –/– | –/– | –/– |
| T1 | Characters | Multi (9) | 24289/8928 | 43.9/39.9 | 84.6/81.4 | 0.73/0.69 | 0.85/0.80 | –/– |
| T2 | Setting | Single (3) | 24289/8928 | 74.7/68.2 | 98.2/98.1 | –/– | –/– | –/– |
| T3 | Objects | Multi (8) | 24289/8928 | 19.7/24.9 | 65.6/66.6 | 0.57/0.56 | 0.77/0.72 | –/– |
| T4 | Intensity | Ordinal (5) | 24289/8928 | 15.3/12.3 | 74.3/69.7 | –/– | –/– | 0.49/0.53 |
| T5 | Actions | Multi (21) | 11702/7329 | 2.6/5.1 | 18.5/25.8 | 0.41/0.44 | 0.77/0.75 | –/– |
| T6 | Family | Single (2) | 3865/1543 | 60.3/72.3 | –/– | –/– | –/– | –/– |
| T7 | Family Behavior | Multi (12) | 1694/712 | 4.6/4.6 | 34.0/35.4 | 0.37/0.39 | 0.61/0.64 | –/– |