Dynamics of Engagement in Citizen Science: Results from the “Yes, I do!”-Project

Tine De Moor; Auke Rijpma; Montserrat Prats López

doi:10.5334/cstp.212

Figures & Tables

Table 1

Measures of engagement composed on the basis of Aristeidou et al. 2017; Cox et al 2015; Ponciano and Brasileiro 2014; Sauermann and Franzoni 2015.

	Measure	Definition	Formula	Required data
PA	Project appeal	“Total number of volunteers who have contributed to the project divided by project active period squared.” (Cox et al., 2015: 32–33)	$PA = \frac{\sum people who have contributed}{{(project active period)}^{2}}$	Project level data: Project active period is project duration excluding (if any) periods of project inactivity: Date project start (PA, PC, SE) Date project end (PA, PC, SE) If applicable: Periods of project inactivity Number of people who have submitted at least one contribution to the project (listed in order of contribution) (PA, PC) Number of submitted contributions (listed in order of submission) (G) Individual level data: Date on which volunteer joins the project (A) Date of volunteer’s first contribution (SE) Date of volunteer’s last contribution (A, SE) (= date on which volunteer leaves the project) Number of days on which volunteer submits a contribution (A) (= active days) Number of inactive days between each submitted contribution (V) Number of hours logged in the project for each active day (D)
A	Activity Ratio	“Ratio of days on which the member was active and executed at least one task in relation to the total days they remained linked to the project.” (Aristeidou et al. 2017: 248, based on Ponciano and Brasileiro 2014: 253–254)	$A = \frac{\sum V' s active days}{date V leaves project] - date V joins project]}$
V	Variation in periodicity	“Standard deviation of the multiset of number of days elapsed between each pair of sequential active days.” (Aristeidou et al. 2017:248, based on Ponciano and Brasileiro 2014: 253–254)	$V = sd (\sum V' s inactive days)$
D	Daily devoted time	“The averaged hours the volunteer remains executing tasks on each day he/she is active.” (Ponciano and Brasileiro 2014: 253–254)	$D = \frac{\sum V' s active hours}{\sum V' s active days}$
G	Distribution of effort	“A measurement of equality in the distribution of classifications” (Gini coefficient) (Cox et al. 2015: 32–33; Sauermann and Franzoni 2015)	$G = 1 - \sum_{n}^{k = 1} (X_{k} - X_{k - 1}) (Y_{k} + Y_{k - 1})$ where X_k is the cumulated number of volunteers and Y_k the cumulated number of contributions
PC	Public contribution	“Median number of classifications per registered volunteer divided by project active period squared.” (Cox et al. 2015: 32–33)	$P C = \frac{median (contributions per volunteer)}{{(project active period)}^{2}}$
SE	Sustained engagement	“Median time interval (in weeks) between a registered user’s first and last recorded classification divided by project active period squared.” (Cox et al. 2015: 32–33)	$SE = \frac{median (date V last - date V first contribution)}{{(project active period)}^{2}}$

Table 2

Activity overview for the total of the “Yes, I do!”-project.

	Total	Rest	Periphery	Outer core	Core
N participants	182	44	75	42	21
N scans	32,503	46	380	2,544	29,533
Days in project	32,089	45	7,184	12,274	12,586
Active days in project	7,253	45	249	1,200	5,759
Average scans per day	2.0	1.0	1.7	2.6	4.4
Average break	38.0	–	64.6	23.4	6.8

Online tool for checking and correcting entries. Source: Vele Handen/Picturae.

Table 3

Proposed measures to track engagement over time.

	Measure	Definition	Dynamics	Formula	Required data
RN	Ratio of new volunteers	Percentage of new volunteers in relation to the total number of volunteers registered in the same period (day, week, month) during a project.	Figure 2	$RN = \frac{Number of new volunteers in period t}{Total number of volunteers in period t}$	Individual level data: Total number of active volunteers in every period of the project (day or week or month) (RN). Number of active volunteers that join at each period (day or week or month) of the project (RP). Number of volunteers who were already members of the platform before the project (RP). Number of submitted contributions in each period (day or week or month) of the project (LA, TI). Task’s type of each submission (TI). Number of insertions, deletions, substitutions, and/or transpositions made in the data of each submission (AG). Date of each submitted data contribution (LA, TI, FD). Date of each submitted data correction (verification or validation) (FD).
RP	Ratio of platform members	Percentage of volunteers who were members of the platform before the project in relation to the total number of volunteers of the project in the same period (day, week, month) during the project.	Figure 4	$RP = \frac{Volunteers in period t from platform}{Total number of volunteers in period t}$
LA	Level of activity	Number of submitted contributions made over time.	Figure 3	$LA = \sum Number of contributions in period t$
TI	Task interest level	For each type of task, the number of submitted contributions is tracked over time and compared to other tasks’ types as a share of the total number of contributions.	Figure 7	$TI = \frac{\sum Contributions made in task x in period t}{Total number of contributions in period t}$
FD	Feedback delay	Mean of the number of days between the submission of data and submission of corrected (verified or validated) data from the quality control.	Figure 8	$\begin{array}{l} FD = mean (date corrected submission \\ - date volunteer s' submission) \end{array}$
ACC	Accuracy	Optimal String Alignment (OSA) distance.	Figure 9	See next section “Quality”
AG	Accuracy growth	Growth in accuracy estimated using regression.	Figure 10 Table A1	AG in $ACC = a + AG * log N entry + u$

Table 4

Example records for the measurement of transcriptions’ accuracy.

groom address 1	groom address 2	groom address final	Sim (1, final)	Sim (2, final)
aan de Slijpstenen, in de Botertom	Slijpstenen in de Botertom	Slijpstenen, in de Botertom	0.794	0.963
over de Balmstraet	over de Balonsbaen	over de Balonsbaen	0.722	1
‘s Gravenland	‘S Gravenland	‘s Gravenland	1	0.923

Number of active citizen participants and new active citizen participants per month (entering at least one scan in that month; left panel) and ratio of new volunteers to total volunteers per month (RN, right panel).

Total number of entries per week. Vertical reference lines mark the events and workshops listed in tables 5 and 6.

Total entries per month by type of volunteer: Those who had registered at velehanden.nl before the project started (pre-registered), volunteers who registered during the project (newly registered), and volunteers who did not make their registration date public (unknown).

Number of scans entered per volunteers (left panel) and Lorenz curve (Gini: 0.91) (right panel). The Lorenz curve traces the cumulative percentage of scans entered (L(p)) against the cumulative percentage of volunteers, ordered by number of entered scans (p).

Table 5

Outreach events of the “Yes, I do!”-project.

Event	Date	Description
De Moor	13/03/2014	Lecture: Introduction to the project, to the topic, conversation with audience (participating and interested volunteers) about future plans
Prak	22/10/2014	Lecture on migration to Amsterdam in history
Yntema	07/03/2015	Lecture on breweries in Amsterdam and guided tour
Deneweth	11/06/2015	Lecture on builders and buildings in Amsterdam
Van Oostrom	23/09/2015	Lecture on marriage and literature in history
Symoens	12/11/2015	Lecture on universities and students in history

Project page where volunteers could choose the period (difficulty) of the scans to transcribe. Source: Vele Handen/Picturae.

Monthly shares of types of transcribed scans tracked throughout the project. Blue lines trace the share of the total scans for the panel’s period; grey lines in the background show the shares for the other periods to facilitate comparisons.

Table 6

Overview of paleography workshops offered throughout the project.

Paleography workshop	Date	Number of Participants
Basic paleography course	10/06/2014	19
Crash course 17th-century handwriting	27/11/2014	9
Crash course 17th-century handwriting	18/02/2015	12
Crash course 17th-century handwriting	15/04/2015	6
Crash course 17th-century handwriting	13/05/2015	10
Advanced course 17th-century handwriting	03/11/2015	7

Mean delay (upper chart) and total entries per week (bottom chart) over time.

Accuracy over time (monthly average accuracy measured by the optimal string alignment (OSA, see page 11) string distance metric between the volunteers’ transcription and the final correction of the groom’s address), by volunteer group (core: >1,000 scans; not core: ≤1,000 scans).

Starting accuracies (left panel) and improvement rates (right panel measured as OSA-improvement) against number of entries. Points indicate the mean estimate of the improvement rate or starting accuracy, and the vertical lines indicate 80 percent confidence intervals.

Dynamics of Engagement in Citizen Science: Results from the “Yes, I do!”-Project

Figures & Tables

Table 1

Table 2

Figure 1

Table 3

Table 4

Figure 2

Figure 3

Figure 4

Figure 5

Table 5

Figure 6

Figure 7

Table 6

Figure 8

Figure 9

Figure 10

Paradigm

My account