Introduction
Citizen science, in which members of the public participate in scientific research, is a well-established methodology within environmental and natural sciences. It has subsequently grown popular within health research (Wiggins and Wilbanks 2019), with projects such as the United Kingdom (UK) Zoe Health Study (health-study.zoe.com) involving 4.6 million participants. In recent years, this interest has extended to mental health-related research. A 2023 systematic review of mental health-related citizen science (MHCS) included nine academic publications (Todowede et al. 2023), and a 2024 report highlighted 45 exemplar projects across six continents (Batty et al. 2024).
Citizens have varying degrees of involvement in citizen science projects. Batty and colleagues (2024) identify two categories within MHCS according to involvement level: contributory projects and co-created projects.
Contributory projects are typically initiated and designed by researchers, with citizens contributing, collecting and/or analysing data. Projects may involve contributing personal data, for example, by monitoring moods (Ball et al. 2014) or playing online games to gather data on mental health–related mechanisms such as risk-taking or decision-making (https://brainexplorer.net). Citizens also collect non-personal data, for example, on geographical and social spaces that may impact mental health (Pykett et al. 2020), or they are trained to analyse data, for example, screening and assessing mental health research papers to contribute to “living” systematic reviews (Cipriani et al. 2023).
Co-created projects are jointly designed and carried out by citizens and researchers. Examples include CoAct (coactproject.eu), in which citizens and researchers co-created a chatbot to educate people about local mental health support networks, and Depression Detectives, which involved citizens discussing gaps in research with scientists and subsequently running their own study (Beange and Collins 2022).
Claims have been made that citizen science has the potential to democratise health research processes by increasing the number and diversity of participants who contribute knowledge to health research (Smith et al. 2019). This could yield better knowledge, more empowered communities, and improved community health, and provide health and social benefits for citizen scientists themselves (Den Broeder et al 2018). However, the principle of participation is already well established within health (Israel et al., 2026) and mental health research (Güell et al. 2023), including at co-creation level, using methodologies such as Participatory Action Research or Community-Based Participatory Research. The question is: What might a citizen science approach bring to mental health research that existing methodologies do not already provide?
The 2023 systematic review of MHCS found that citizen science could complement other participatory approaches, and recommended that public participation at all stages of the citizen science research process should be encouraged (Todowede et al. 2023). A later consultation with mental health citizen scientists identified greater co-creation of research as the primary desired long-term impact of MHCS (Todowede et al. 2025). However, the review and consultation did not provide ways of distinguishing between citizen science and existing participatory approaches, which may also achieve aims of greater co-creation. The necessity for this discussion has been raised (Bonhoure et al. 2019) but not yet been undertaken. The priority of greater coproduction identified in Todowede and colleagues’ 2025 consultation reflects a trend towards more co-creational models of citizen science across disciplines (Tøttrup et al 2025). Such a trend may be important to mitigate more questionable practices, including citizen science as public relations, when used, for example, by companies to deflect attention from negative publicity (Blacker et al. 2021) and predatory citizen science, wherein resulting data remains private for the exclusive (and potentially profitable) use of the project originators (Fontúrbel 2023). Clarity about methodological differences and distinguishing characteristics of MHCS would enable future researchers and citizens to better recognise and articulate whether citizen science or another method may be most appropriate for a project.
Scholars have pointed to the difficulty of providing overarching definitions of citizen science, and stress the need for citizen science to be defined differently depending on context (Haklay et al. 2021a). Citizen science in health contexts has been distinguished from citizen science in other disciplines. A survey by the ECSA’s Citizen Science for Health working group found three main variances: different ethical and consent mechanisms; citizens being more likely to be the population under investigation as well as contributing to the science; and more complex dynamics between parties, for example, patients, doctors, and researchers (Remmers et al. 2024). MHCS participants are also likely to differ from citizen scientists in other areas. They are more likely to have lived/living experience of mental health challenges, and thus may be at greater risk of experiencing distress related to the research focus.
Given these contextual differences, MHCS may require additional attention to ensuring citizens’ safety, wellbeing, and meaningful involvement. To grow towards more co-created projects, attending to ways of sustaining the involvement of mental health citizen scientists across projects and over time is important. Sustained involvement could enable interested citizens to develop their knowledge, research skills, and experience, potentially taking on increased responsibility for designing and initiating projects. There is a current lack of consensus in the literature on ways to ensure a safe, healthy, and sustainable citizen experience within MHCS. As a team of researchers and citizens, we sought clarity on these issues to provide guidance for the design and conduct of future high-quality and methodologically distinctive MHCS research. We also sought consensus on ways of collaborating to create a strong community of practice capable of sustaining longer-term involvement of citizens, to support the development of more co-created projects.
This study therefore aimed to answer two research questions through consultation with MHCS experts to address these context-specific concerns: what is distinctive about a citizen science approach to mental health, compared with other forms of participatory research? (RQ1: Distinctiveness). And how can the quality and sustainability of MHCS be maximised? (RQ2: Quality/sustainability).
Methods
The study was conducted in January–July 2025 as part of the UK Citizen Science To Achieve Co-production at Scale (C-STACS) study (researchintorecovery.com/c-stacs), which aimed to enable mass public participation in mental health research. It was carried out by an interdisciplinary team of researchers and citizens with experience of leading and participating in citizen science studies within health science, psychology, computer science, and geography.
We selected a Delphi consultation as an established method within health science for investigating consensus between experts on issues where there may be uncertainty, contention, or a lack of clarity (Niederberger and Spranger 2020). Delphi consultations have also been used within mental health research to determine collective values and define foundational concepts (Jorm 2015). The method involves asking a panel of experts for their opinions on a relevant issue, summarizing and presenting their collective responses, and repeating this process over at least two iterative rounds (Shang 2023). Its structured communication process is designed to reduce bias and refine expert opinions (Alon et al. 2025). We chose an online Delphi consultation to facilitate participation regardless of location or time zone, to be time- and cost-effective, and to allow ease of data management (Shang 2023). The maximum number of rounds was set at two, owing to study time constraints.
Criteria for selection of expert participants
Two types of experts were identified as potential participants: (i) public contributors and (ii) project leads, all aged 18+. Public contributors were defined as citizens who have contributed to an MHCS project at least once. Project leads were defined as people who have designed or run MHCS projects, through generating research questions, analysing data, and/or working together to disseminate and take further action on project results. Project leads could be professional researchers or citizens. We asked participants with experience of both roles to respond from the perspective of their most frequent role.
Recruitment
We identified experts using seven approaches: (i) the 45 projects identified in Batty et al. (2024); (ii) the nine papers in our systematic review (Todowede et al. 2023), (iii) a Google Scholar search, comprising the first 10 pages of results for “mental health citizen science” and “mental health community science;” (iv) MHCS projects found on 13 citizen science platforms listed on the Association for Participatory Sciences website (https://participatorysciences.org/resources/platforms-for-hosting-participatory-science-projects/); (v) a social media call; (vi) research team contacts including the C-STACS study mailing list and its ten partner organisations; and (vii) snowball sampling, asking potential experts to forward the invitation to colleagues and the public contributors involved in their MHCS projects, where possible. All responding experts who met the definitions were invited to participate.
Measures
Round 1 questionnaire
We developed a pilot questionnaire through analysis of two formative MHCS publications (Batty et al. 2024; Todowede et al. 2023). Areas of MHCS identified in these papers by the research team as being unresolved, conflicting, or requiring further development were framed into statements for participants to consider. We generated further content using the ten ECSA Principles of Citizen Science (Robinson et al. 2018). The pilot was revised by authors CE and CH, the citizen members of the research team.
The final Round 1 (R1) questionnaire contained 59 questions or statements (items) in five sections (Supplemental File 1), presented via Microsoft Forms. Sections A to C addressed distinctiveness (RQ1), and sections D and E addressed quality/sustainability (RQ2). For sections A,B,D, and E, participants were asked to assess the importance of each item for MHCS on a five-point Likert scale from Essential to Not important at all. Free-text questions invited further responses. For section C, participants were asked to categorise seven research example vignettes as either MHCS, participatory research, patient and public involvement (PPI) initiatives, peer research, traditional research, or other, with a free-text response for alternative categorisations.
Round 2 questionnaire
The Round 2 (R2) questionnaire comprised items not reaching consensus in R1 and new items generated from analysis of R1. A personalised Microsoft Form was generated for each participant showing their responses to each R1 question and the R2 questions. The questionnaire was piloted by authors CE and CH and a colleague experienced in Delphi consultations. On the basis of their feedback, we made refinements aimed at facilitating comprehension and minimising response fatigue. The final R2 questionnaire contained 50 questions organised into the same five sections (Supplemental File 2).
Delphi procedure and analysis
In R1, experts were invited to participate via email and social media, and given links to the Participant Information Sheet (Supplemental File 3), consent form, and questionnaire. Responses were collected via Microsoft Forms. Participants were given four weeks to complete R1. Non-respondents were sent a reminder invitation after two weeks.
After R1, we conducted an initial analysis to gauge consensus and inform R2. In line with recommendations to pre-specify consensus levels (Shang 2023), we defined moderate consensus as at least 70% rating an item as either Essential (Likert rating 5) or Very important (4), or at least 70% rating an item Not important at all (1). Strong consensus was pre-defined as at least 80% rating the same ways. Quantitative items were descriptively analysed, with consensus investigated in terms of the group response overall (referred to as “whole-group consensus”) and the responses of five subgroups: (i) project leads, (ii) public contributors, (iii) people with lived experience of mental health challenges, (iv) people with lived experience of supporting family, friends, or colleagues, and (v) people without lived experience in this area. In R2, items reaching consensus across all subgroups were removed to reduce responder burden; items not reaching whole-group consensus, or with disagreement among subgroups, were re-presented. Free-text responses from R1 were summarised. Potential new items based on these responses were generated and refined by the study team.
In R2, items for re-rating and new items were presented. As is the norm in Delphi consultations, participants were shown their previous responses and a table summarising whole-group and subgroup results. Participants were informed they were not required to change their responses, but could do so if they wished. To aid an inclusive approach, a guide to understanding the results was produced (Supplemental File 4). R2 remained open for three weeks, with two reminders sent to minimise potential attrition rates.
Quantitative items were analysed descriptively for levels of consensus. Free-text items were summarised and analysed thematically.
Ethics statement
Ethical approval was prospectively granted for this study by the Faculty of Medicine and Health Sciences Research Ethics Committee, a subcommittee of the University of Nottingham Research Ethics Committee. The reference was FMHS 292–0924. Informed consent was obtained from all participants.
Results
Findings are discussed in terms of five themes relating to the two research questions. Themes 1–3 relate to RQ1 (Distinctiveness) and themes 4–5 to RQ2 (Quality/Sustainability).
Participants
34 participants completed R1. 15 (44%) participants successfully completed R2. Participant demographics can be found in Supplemental File 5. One participant was excluded after R2 as 27 (87%) of their responses to Likert scale items had been changed to Not important at all. This was gauged to be a type of response bias common to questionnaires known as negative extreme responding (Van Vaerenbergh and Thomas 2013), which can distort overall results.
There was a 56% attrition rate of participants between Rounds 1 and 2. Possible reasons for this are discussed below. In the following tables, we present items where consensus was reached, whether in R1 or 2 (Tables 1, 2, 3, 4). Results for theme 3 are presented in Supplemental File 6. Results for all items are provided in Supplemental Files 7 and 8. Throughout, the distribution of responses was often skewed.
Table 1
Consensus on definitive characteristics of mental health citizen science.
| ITEM | ROUND | ALL | PARTICIPANTS BY ROLE | PARTICIPANTS BY LIVED EXPERIENCE | |||
|---|---|---|---|---|---|---|---|
| PROJECT LEADS | PUBLIC CONTRIBUTORS | OWN LIVED EXPERIENCE | SUPPORTED OTHERS | NO LIVED EXPERIENCE | |||
| A1: Project is directly about mental health | 1 | 4.0 (0.9) | 4.0 (0.9) | 3.9 (1.2) | 3.9 (1.0) | 4.3 (0.5) | 4.2 (1.0) |
| 2 | 4.1 (1.0) | 4.0 (1.0) | 5.0 (0.0) | 4.0 (1.1) | 4.5 (0.6) | 4.0 (1.4) | |
| A5: Public contributors are involved from the start in designing projects | 1 | 3.6 (1.1) | 3.4 (1.2) | 4.0 (0.8) | 3.3 (1.2) | 3.8 (1.2) | 4.2 (0.8) |
| 2 | 4.1 (1.0) | 4.0 (1.0) | 4.5 (0.7) | 4.1 (1.3) | 4.0 (0.0) | 4.0 (0.0) | |
| A12: Projects adhere to established scientific methods of doing research | 1 (consensus reached) | 4.1 (1.1) | 4.2 (1.0) | 4.0 (1.5) | 4.3 (1.0) | 3.7 (1.4) | 4.2 (1.0) |
| A14: Public contributors are acknowledged in publications | 2 (New question) | 4.0 (1.0) | 3.9 (1.0) | 4.5 (0.7) | 4.1 (0.9) | 3.5 (1.3) | 4.5 (0.7) |
| A15: Safeguarding procedures are in place for public contributors | 2 (New question) | 4.5 (0.6) | 4.6 (0.7) | 4.0 (0.0) | 4.2 (0.7) | 5.0 (0.0) | 5.0 (0.0) |
| A16: Support for emotional distress (occurring or made worse through participation) is available for public contributors | 2 (New question) | 4.1 (0.8) | 4.2 (0.9) | 4.0 (0.0) | 4.1 (0.8) | 3.8 (1.0) | 5.0 (0.0) |
| A17: Accessibility & inclusion needs of public contributors are addressed proactively | 2 (New question) | 4.4 (0.8) | 4.5 (0.5) | 4.0 (0.0) | 4.3 (0.5) | 4.5 (0.6) | 4.5 (0.7) |
[i] Note: Values presented are mean (standard deviation). 5 = essential, 4 = very important, 3 = somewhat important, 2 = a bit important, 1 = not important. Regular type = no consensus. Italics = moderate consensus (70%–79%). Bold = strong consensus (80% or more).
After R1, whole-group consensus (column “All”) was established on just one item (A12). Qualitative analysis of free-text responses added four further suggested distinctive features of MHCS to R2 (items A14–A17). After R2, whole-group consensus was reached on the two of the re-presented items (A1 and A5), and the four new items (A14–A17).
The biggest change between rounds in this theme was a decrease in ratings for the importance of item A13, regarding paying public contributors, about which there was no consensus. This was rated 18% Essential/Very important in R1 which decreased further to 7% in R2. A notable finding was the strong consensus across all subgroups by R2 that a definitive characteristic of MHCS should be that public contributors are involved from the start in designing projects.
Theme 2: Relevance of the European Citizen Science Association principles for mental health projects
Table 2 shows results where consensus was reached on the relevance of the ECSA’s 10 Principles of Citizen Science for MHCS.
Table 2
Consensus on relevance of the European Citizen Science Association (ECSA) principles for mental health citizen science.
| ITEM | ROUND | ALL | PARTICIPANTS BY ROLE | PARTICIPANTS BY LIVED EXPERIENCE | |||
|---|---|---|---|---|---|---|---|
| PROJECT LEADS | PUBLIC CONTRIBUTORS | OWN LIVED EXPERIENCE | SUPPORTED OTHERS | NO LIVED EXPERIENCE | |||
| B1: Citizen science projects actively involve citizens in scientific endeavour that generates new knowledge or understanding. Citizens may act as contributors, collaborators, or project leaders & have a meaningful role | 1 (consensus reached) | 4.6 (0.7) | 4.7 (0.7) | 4.4 (0.8) | 4.5 (0.8) | 4.8 (0.4) | 5.0 (0.0) |
| B2: Citizen science projects have a genuine science outcome | 1 | 3.9 (1.0) | 4.0 (0.9) | 3.9 (1.2) | 3.8 (1.1) | 4.3 (0.8) | 4.0 (0.6) |
| 2 | 4.1 (0.6) | 4.1 (0.6) | 4.5 (0.7) | 4.1 (0.6) | 4.3 (0.5) | 4.0 (1.4) | |
| B3: Both professional scientists & citizen scientists benefit from taking part | 1 (consensus reached) | 4.5 (0.7) | 4.5 (0.8) | 4.3 (0.8) | 4.5 (0.6) | 4.2 (1.2) | 4.7 (0.8) |
| B4: Citizen scientists may participate in multiple stages of the scientific process | 1 (consensus reached) | 4.1 (0.7) | 4.1 (0.7) | 4.0 (0.6) | 3.9 (0.7) | 4.3 (0.5) | 4.5 (0.5) |
| B5: Citizen scientists receive feedback from the project | 1 (consensus reached) | 4.6 (0.6) | 4.6 (0.6) | 4.6 (0.8) | 4.5 (0.7) | 4.7 (0.5) | 5.0 (0.0) |
| B6: Citizen science is considered a research approach like any other, with limitations & biases that should be considered & controlled for | 1 (consensus reached) | 4.4 (0.8) | 4.3 (0.9) | 4.6 (0.5) | 4.3 (0.8) | 4.7 (0.5) | 4.2 (1.2) |
| B7: Citizen science project data/meta-data are made publicly available & where possible, results are published in an open access format | 1 (consensus reached) | 4.3 (0.8) | 4.3 (0.8) | 4.3 (0.8) | 4.3 (0.9) | 4.5 (0.5) | 4.2 (0.8) |
| B8: Citizen scientists are acknowledged in project results & publications | 1 (consensus reached) | 4.4 (0.8) | 4.4 (0.8) | 4.4 (0.8) | 4.3 (0.9) | 4.5 (0.8) | 5.0 (0.0) |
| B9: Citizen science programmes are evaluated for scientific output, data quality, participant experience & impact | 1 (consensus reached) | 4.4 (0.8) | 4.3 (0.9) | 4.4 (0.5) | 4.3 (0.9) | 4.7 (0.5) | 4.3 (0.8) |
| B10: Project leads take into consideration legal & ethical issues of any activities | 1 (consensus reached) | 4.6 (0.6) | 4.6 (0.6) | 4.6 (0.8) | 4.5 (0.7) | 4.7 (0.5) | 5.0 (0.0) |
[i] Note: Values presented are mean (standard deviation). 5 = essential, 4 = very important, 3 = somewhat important, 2 = a bit important, 1 = not important. Regular type = no consensus. Italics = moderate consensus (70%–79%). Bold = strong consensus (80% or more).
Theme 1: Definitive features of a mental health citizen science approach
Table 1 shows results where consensus was reached on definitive features of an MHCS approach.
After R1, whole-group consensus was established on all ten principles, with strong consensus for nine principles across all subgroups. There was disagreement between subgroups on the relevance of the tenth item (B2). This was re-presented in R2, with a supplemental free-text question to aid our understanding of interpretations of “genuine science outcomes”. After R2, strong whole-group consensus was also reached for item B2.
Qualitative analysis
Analysis of free-text responses showed that participants held a range of epistemological and methodological stances on what might qualify as a science outcome within MHCS. These could be grouped into three categories of outcome: traditional, applied, and broad. First were responses reflecting traditional research outcomes. Here a science outcome was “a statement about the world that is falsifiable” (PX02). It meant “systematising the applied methodology so that the procedure … can be replicated” (PX19). For another respondent, what was meant by “science” was not specified, but was seen as crucial:
for it to be citizen science, it is essential that we have science in the project. This not only means using scientific methods but also generating a scientific outcome. Obviously, it is not the only type of outcome that should be generated, but it is essential (PX13).
For another, the most important thing was that:
any data collected has to be used – this could be in papers, for future projects, for local action etc. It is essential that the data goes somewhere. Projects where data doesn’t go anywhere or get used shouldn’t be called citizen science (PX24).
Second were responses reflecting what one participant called “something more applied” (PX03). Here, science outcomes could be, for example, “social science findings that speak to more information about the community members […] such as inclusion” (PX14), or “discovering things from […] the perspective of the people involved [which] can help to understand a topic better” (PX26).
Third were responses reflecting that citizen science incorporates a wider range of outcomes than might traditionally be labelled scientific:
I think adhering to scientific principles is important, yet I also think that Citizen Science projects may or even ought to break with some of these principles. If they only generate ‘local’ insights that cannot be generalized or rather be made relevant for others/other contexts, then I think the status of science is indeed not warranted. This ‘local’ effect however is an important outcome of Citizen Science projects (PX25).
Participants also gave examples of types of outcomes regarded as genuinely scientific. These represented the generation of either new knowledge, for example, “a scientific project should contribute in some way to expanding knowledge” (PX9), or better knowledge, for example, “better care outcomes and better understanding of medical conditions” (PX10).
Theme 3: Distinguishing mental health citizen science from other forms of mental health research (including other participatory approaches)
The results of the categorising exercise on the methodological approaches of seven vignettes based on real-world research examples can be found in Supplemental File 6. The purpose of this exercise was to gauge whether there was agreement among participants on the methodology described in each vignette. This was order to explore whether vignettes based on projects self-identifying as MHCS were distinguishable from other research studies, either traditional or participatory.
After R1, no consensus had been reached on which methodology was being described in any vignette. The highest agreement reached was for example #7, which 56% of participants identified as MHCS.
All examples were re-presented in R2, with a new category, Crowd science, based on analysis of free-text responses. Participants were asked to explain what had informed their decisions, to enable further identification of features considered distinctive to MHCS. After R2, one item reached consensus: example #5, which described the codesign and dissemination of resources by a university department’s Youth Advisory Group. 80% categorised this as an example of Patient and Public involvement (PPI).
Qualitative analysis
Qualitative analysis of free-text responses highlighted key aspects of participants’ decision-making. The most common rationale was the degree of citizen involvement at different research stages. For example:
citizens collaborate to shape the research, meaning they are very actively involved, which for me is the essence of citizen science (PX9),
and
citizen science could be a good descriptor if participants also have a role in designing the research or determining how it will be used (PX7).
A related rationale was how active or passive the participants’ role was gauged to be overall; for example, an item was not categorised as MHCS because
citizens don’t play an active role. They do things, and researchers study what they do. They could be people or mice (PX8).
Participants also chose not to categorise examples as citizen science if the descriptions made no mention of generating new knowledge or specifying a research question. Also of note was that two participants categorised an example as “participatory research” because it described the researchers as having mental health lived experience.
Two rationales were notable which were not related to the research descriptions. One was being persuaded by other participant responses, from the group as a whole:
As almost nobody answered peer research, and I am not an expert in methodology, I changed my answer to participatory (PX19);
and from the public contributor subgroup in particular:
I amended my responses upwards when public contributors said they were more important, as I thought this perspective was very interesting, and in my view should be taken more into consideration (PX24).
The other rationale was where participants felt there was either not enough detail provided to make an informed choice, or the categories themselves contained overlap. For example:
many forms of participatory research and crowdsourcing are methods for conducting citizen science. So the distinction is sometimes difficult to make (PX22).
Qualitative results for this theme are shown in full in Supplemental File 9.
Theme 4: Improving quality in mental health citizen science
Table 3 shows results where consensus was reached about approaches to improving quality.
Table 3
Consensus on improving the quality of mental health citizen science (MHCS) projects.
| ITEM | ROUND | ALL | PROJECT LEADS | PUBLIC CONTRIBUTORS | OWN LIVED EXPERIENCE | SUPPORTED OTHERS | NO LIVED EXPERIENCE |
|---|---|---|---|---|---|---|---|
| D1: Public contributors & project leads work together to co-produce each stage of the research process | 1 | 3.8 (1.0) | 3.7 (1.0) | 4.3 (0.5) | 3.6 (1.1) | 3.8 (0.4) | 4.3 (0.5) |
| 2 | 4.2 (0.9) | 4.2 (0.9) | 4.5 (0.7) | 4.2 (1.1) | 4.0 (0.0) | 4.5 (0.7) | |
| D3: Digital platforms delivering projects feel safe & accessible for public contributors | 1 (consensus reached) | 4.1 (0.9) | 4.0 (1.0) | 4.4 (0.5) | 4.1 (1.1) | 4.2 (0.4) | 4.2 (0.8) |
| D4: Project leads are transparent, honest & competent in engaging with prospective public contributors | 1 (consensus reached) | 4.6 (0.6) | 4.7 (0.6) | 4.6 (0.5) | 4.6 (0.5) | 4.7 (0.8) | 4.6 (0.8) |
| D5: A clear definition of who can be a public contributor is provided | 1 (consensus reached) | 4.2 (0.9) | 4.1 (1.0) | 4.4 (0.5) | 4.3 (0.9) | 3.7 (1.0) | 4.3 (0.8) |
| D6: An inclusive definition of who can be a public contributor is provided | 1 (consensus reached) | 4.3 (0.8) | 4.2 (0.8) | 4.7 (0.4) | 4.4 (0.7) | 4.0 (1.1) | 4.3 (0.5) |
| D7: All project activities are described clearly | 1 (consensus reached) | 4.4 (0.9) | 4.3 (0.9) | 4.6 (0.8) | 4.3 (1.0) | 4.5 (0.5) | 4.3 (0.8) |
| D8: People using mental health services & people with lived experience NOT using services can be public contributors | 1 | 3.9 (1.1) | 3.9 (1.2) | 4.0 (0.6) | 3.7 (1.2) | 3.8 (0.4) | 4.7 (0.5) |
| 2 | 4.1 (1.1) | 4.1 (1.1) | 4.5 (0.7) | 4.0 (1.3) | 4.0 (0.0) | 5.0 (0.0) | |
| D9: People with significant experience of research in the same area cannot be public contributors | 1 | 1.7 (0.8) | 1.6 (0.6) | 2.3 (1.3) | 1.6 (0.9) | 1.8 (0.8) | 2.0 (0.6) |
| 2 | 1.3 (0.6) | 1.3 (0.6) | 1.5 (0.7) | 1.3 (0.7) | 1.3 (0.5) | 1.0 (0.0) | |
| D10: The aims and content of the planned project are described in an accessible manner to the public | 1 (consensus reached) | 4.8 (0.5) | 4.8 (0.4) | 4.6 (0.8) | 4.7 (0.6) | 4.8 (0.4) | 4.8 (0.4) |
[i] Note: Values presented are mean (standard deviation). 5 = essential, 4 = very important, 3 = somewhat important, 2 = a bit important, 1 = not important. Regular type = no consensus. Italics = moderate consensus (70%–79%). Bold = strong consensus (80% or more).
After R1, consensus had been reached on eight of ten items. For two (D1 and D8), there was notable disagreement among subgroups. These and the two items without consensus were re-presented in R2.
After R2, three of the four re-presented items (D1, D8, and D9) reached strong whole-group consensus. D9 was the only item in either round to reach consensus that it was not important.
Theme 5: Creating sustainable communities of mental health citizen scientists
Table 4 shows results where consensus was reached about approaches to building sustainable communities.
Table 4
Consensus on creating sustainable communities of mental health citizen science (MHCS) public contributors.
| ITEM | ROUND | ALL | PROJECT LEADS | PUBLIC CONTRIBUTORS | OWN LIVED EXPERIENCE | SUPPORTED OTHERS | NO LIVED EXPERIENCE |
|---|---|---|---|---|---|---|---|
| E4: A clear & desirable identity about being a public contributor | 1 | 4.0 (1.1) | 3.9 (1.2) | 4.3 (0.8) | 4.0 (1.0) | 3.7 (1.2) | 4.2 (1.6) |
| 2 | 4.1 (1.0) | 4.0 (1.1) | 4.5 (0.7) | 4.1 (1.1) | 3.8 (1.3) | 4.5 (0.7) | |
| E6: A catalogue of learning resources enabling public contributors to build knowledge & increase involvement | 1 | 3.7 (0.9) | 3.7 (0.9) | 4.0 (1.0) | 3.8 (1.1) | 3.5 (0.5) | 3.7 (0.5) |
| 2 | 3.9 (0.9) | 3.8 (0.9) | 4.5 (0.7) | 3.9 (1.2) | 4.0 (0.0) | 4.0 (0.0) | |
| E7: Involvement opportunities which increase in responsibility as public contributors develop their knowledge | 1 | 3.7 (0.9) | 3.7 (0.9) | 3.7 (1.0) | 3.8 (1.0) | 3.5 (0.5) | 3.8 (0.8) |
| 2 | 3.9 (0.9) | 3.8 (0.9) | 4.5 (0.7) | 4.1 (1.1) | 3.8 (0.5) | 3.5 (0.7) | |
| E8: Co-operation between individual projects to enable continuous engagement in MHCS research if desired | 1 | 3.7 (0.8) | 3.6 (0.7) | 4.3 (0.8) | 3.9 (0.8) | 3.5 (0.8) | 3.5 (0.5) |
| 2 | 4.2 (0.6) | 4.2 (0.6) | 4.5 (0.7) | 4.2 (0.7) | 4.0 (0.0) | 4.5 (0.7) | |
| E10: Opportunities to join social support networks with other public contributors | 1 | 3.6 (1.0) | 3.4 (1.0) | 4.3 (0.5) | 3.7 (1.0) | 3.7 (0.8) | 3.3 (1.0) |
| 2 | 3.9 (0.6) | 3.8 (0.6) | 4.5 (0.7) | 4.0 (0.7) | 3.8 (0.5) | 4.0 (0.0) |
[i] Note: Values presented are mean (standard deviation). 5 = essential, 4 = very important, 3 = somewhat important, 2 = a bit important, 1 = not important. Regular type = no consensus. Italics = moderate consensus (70%–79%). Bold = strong consensus (80% or more).
After R1, no whole-group consensus had been reached on any of ten items presented. However, public contributors had strong consensus that seven items were essential/very important. All ten items were re-presented in R2. After R2, whole-group consensus was established on the importance of five items, all of which had reached strong consensus of importance from public contributors in the previous round. This theme saw the most notable differences between subgroup responses in R1, and the most change in R2, suggesting that the group as a whole had been influenced by public contributor responses.
Final results: Consensus areas of importance
Thirty items were established as important features of MHCS by whole-group consensus. These are presented in Figure 1.
Figure 1
Important features for mental health citizen science (MHCS): an expert consensus.
Discussion
34 experts from 10 countries participated in our study, comprising 27 (79%) project leads and seven (21%) public contributors. They established consensus on the importance of 18 characteristics of MHCS from three areas of consultation: definitive features (n = 1), relevant ECSA principles (n = 9), and approaches for improving the quality of MHCS (n = 8). In R2, consensus was established from a smaller panel (n = 15) on the importance of 12 further items, including a fourth area: developing a sustainable community of public contributors (n = 5). There was no whole-group consensus in either round on the fifth area: distinguishing between methodologies of research examples.
Research question 1: Distinctiveness
In R1, there was consensus on the importance of only one suggested definitive feature of MHCS: adherence to established scientific methods. There was initially disagreement about a related ECSA principle, that citizen science projects should have genuine science outcomes, before consensus was found that this was important in R2. However, our qualitative analysis showed that what constituted a genuine science outcome depended on respondents’ epistemological stances, which varied considerably. These findings suggest a potentially distinctive feature of MHCS compared with other participatory methodologies. For example, while participatory action research and community-based participatory research adhere to scientific methods, they are rooted in interpretive or emancipatory approaches, offering an alternative paradigm focused on social change, collaboration, and systematic reflection, rather than adhering strictly to traditional positivist, linear scientific approaches (Pain et al. 2022; Collins et al. 2018). Our findings show that, in contrast, MHCS is currently used as a method within a range of research paradigms, encompassing traditional, interpretive, and emancipatory epistemologies.
Our R2 results offer further although potentially conflicting insights into the distinctiveness of MHCS and the needs of public contributors in this field. Consensus was reached on six further distinctive features of MHCS, including four items generated through participant suggestions. These items reflected mental health context-specific concerns, including recognition of public contributors and additional measures to ensure participants’ wellbeing. Most notable was that public contributors should be involved from the start in designing projects, which 93% rated as essential/very important, indicating a strong preference for co-creation to be a defining feature of MHCS rather than one of a number of possible categories of involvement. This was reflected in qualitative analysis of decision-making while identifying research example methodologies, where experts used level and degree of active citizen involvement as key features in determining whether an example was MHCS or not.
Another finding supports the view that established citizen science principles may require tailoring to suit a mental health context. There was no consensus on paying public contributors, with just 18% rating this as important in R1. In R2, this had decreased to 7%, possibly because just 14% of public contributors themselves had rated this important. This indicates that paying public contributors is not considered to be a distinctive feature of MHCS. However, within the mental health field more widely, paying participants for their involvement is an established practice (Mah et al. 2025), recommended in best practice guidelines as an important way of acknowledging the expertise and time contributed by “experts by experience” (Hawke et al. 2025). It may therefore be important to retain this practice within MHCS.
These features may not be so relevant in other fields utilising citizen science. They are, however, vital given the particular historical and political context of mental health research, wherein the meaningful involvement of experts by experience in designing and conducting research that positively impacts our/their lives is a central and hard-won principle (Pinfold et al. 2025; Veldmeijer et al. 2023). Although specific participatory approaches within mental health research do differ, an overarching theme is the importance of empowering marginalized voices so that a less oppressive mental health care system can be built. The importance of distinguishing genuinely empowering participation from co-opted forms of involvement has been stressed (Colder Carras et al. 2023). To avoid concern that the adoption of citizen science approaches may represent a more co-opted form of involvement, future MHCS research could incorporate into its project design the items found by this expert consensus to be important features of MHCS. As the co-creation and payment issues suggest, there may also be further work to be done to ensure that important, and hard-won, aspects of mental health research are not lost within this approach. Our findings could ensure a better fit for citizen science approaches to the potential needs of mental health research participants, many of whom have lived or living experience, for whom contributing towards knowledge production runs the risk of having contributed at their own cost (Hawke et al. 2022; Richmond et al. 2023).
Research question 2: Quality/sustainability
In R1, there was consensus on the importance of eight of the ten suggested features for improving quality (rising to nine in R2). Much of the literature on quality in citizen science focuses on the quality and trustworthiness of the data generated, refuting scepticism that non-professionals can produce reliable data (Aceves-Bueno et al. 2017). Our study focused on quality in research design and conduct. We found that an emphasis on citizen empowerment and wellbeing to improve the quality of MHCS is considered important among project leads and public contributors alike. Areas of consensus on markers of quality again included levels of co-production, and the clarity, transparency, inclusiveness, safety, and accessibility of the research process.
In terms of sustainability, we found that the views of public contributors were influential. In R1, there was no overall consensus on the importance of any suggested items for developing a sustainable community of public contributors. However, public contributors rated seven of the ten items as important. After R2, there was a notable change, in that five of these seven features were now deemed important by the whole group (albeit a smaller one).This suggests that the views of public contributors were influential for experts in R2, 87% of whom were project leads. The features included a clear and desirable identity for citizens, access to learning resources, involvement opportunities that increased over time, greater cooperation between MHCS projects to enable continuous involvement, and opportunities for social support networks with other citizens. These findings support those of other studies on the retention of volunteers within citizen science in other areas. For example, a review of volunteer motivations produced best practice guidelines on retention, including creating a sense of community, offering opportunities for training, and enabling feelings of co-ownership (Robinson et al. 2021). Our findings on quality and sustainability form the basis of a checklist that may provide useful guidance for the design of future high-quality and sustainable MHCS research. They sit alongside guidance from other participatory research areas, including methods for exploring sensitive topics (Silverio et al. 2022), positionality considerations (Corrigan and Twiss 2026), reporting guidelines for public involvement (Staniszewska et al. 2017), and the participatory development, and evaluation of digital health interventions (Weirauch et al. 2026).
Strengths and limitations
A strength of this study was the number of participants (n = 34) from 10 countries, in what is still an emerging field. Another is the number of items where consensus was established after one round, covering definitive features, guiding principles, quality, and sustainability, which may be a helpful guide for future research in this area.
There are several limitations to the study. Despite a recruitment strategy that included direct approaches to citizens via citizen science platforms, social media, and research team contacts, only seven public contributors were recruited. Another limitation is the high attrition rate for R2. This can be a major issue in Delphi studies, with attrition rates ranging from 0% to 92% in health-related research (Shang 2023). Although efforts were made to reduce participant fatigue by, for example, simplifying results presentation in R2, there were still 50 items in the questionnaire, several requiring free-text responses. It may have been more effective to make these responses optional, even at the risk of losing valuable insight into participants’ decision-making processes. A further limitation is that the seven vignettes were generated from real-world examples. Another approach would have been a more systematic vignette-generation process to maximise diversity in each candidate citizen science dimension, as used in a six-country survey (Haklay et al. 2021b). Finally, a Delphi consultation, while useful in gauging levels of consensus in a field, can flatten a nuanced discussion of issues such as the question of distinctiveness. Future research could use other methodologies to investigate this question further.
Conclusion
This study has produced expert consensus on 18 features and principles deemed definitive features of MHCS. It provides evidence that a further 12 features deserve consideration, particularly regarding the views of public contributors on what is most likely to sustain their contributions. These 30 features may provide a basis for guidance on adapting citizen science appropriately for mental health contexts. Our findings demonstrate that there is consensus from project leads and public contributors alike that co-creation is a defining feature rather than a possible category of MHCS. An important difference in citizen science compared with other participatory approaches within mental health is that it is currently being used across traditional, interpretive and emancipatory approaches to projects. Future research could provide additional consultation with a larger number of expert public contributors, to ensure that their perspectives remain at the heart of mental health research.
Supplementary Files
The supplementary files for this article are as follows:
Supplemental File 4
Participants’ guide to reading the results. DOI: https://doi.org/10.5334/cstp.971.s4
Data Accessibility Statement
All data that support the findings of this study are included within the article and supplementary files.
Ethics and Consent
Before beginning the consultation, participants were shown a title page that described the purpose of the research being conducted and gave a link to the Participant Information Sheet which outlined what their voluntary participation would entail. This page also indicated that all participants were required to be 18 years old or older. Participants were required to indicate their informed consent before beginning the consultation questionnaire. Personal identifiable information which was collected about participants was stored securely online and only accessible by the research team. Data collected through the iNaturalist API was freely available. Ethical approval was prospectively granted for this study by the University of Nottingham Faculty of Medicine and Health Sciences Research Ethics Committee, reference FMHS 292–0924.
Acknowledgements
We are grateful to all the participants who contributed their expertise to this study, including Amanda Figueras (citizen involved in MHCS), Dr. Sarah Markham (Kings College, London, UK), Mrs. Micaela Santilli, Dr. Rhoda Schuling (Hanze University, Netherlands), and 30 others.
Author Contributions
SRE, OT, DB, SM and MS contributed to the project conception and design of the survey methodologies. JLB acquired and analysed data, and drafted the manuscript. SRE, OT, LD, CG, CE, CH and MS contributed to the editing of the manuscript. All authors critically revised the content. All authors gave final approval of the submitted version and agree to be accountable for aspects of the work they conducted.