Introduction
Artificial intelligence (AI) is an important approach for unbiased analysis of space exploration data (Sanders et al. 2023; Scott et al. 2023). This AI/data mining approach requires volumes of high-quality multimodal data, such as those found in the maximally open access NASA Open Science Data Repository (OSDR) (osdr.nasa.gov) (Scott et al. 2020; Berrios et al. 2020). The creation of NASA OSDR in 2023 was an expansion of NASA GeneLab, and integration-modernization of the phenotypic and physiologically dominant NASA Ames Life Sciences Data Archive (ALSDA). Depositing omics data (genomics, transcriptomics, proteomics, etc.) has been long mandatory with well-developed standards for data archiving (Rutter et al. 2020; Nelson et al. 2021), allowing new large-scale analysis of omics data identifying novel biological pathways affected by microgravity (Afshinnekoo et al. 2020; da Silveira et al. 2020; Nelson et al. 2021). In contrast, outcomes describing phenotypic-physiological changes were published as journal articles and remain inaccessible for data archiving and AI approaches. These data are difficult to access and/or process, generally because of old or obsolete formats, and are referred to as legacy data. Since experiments in space are costly and rare, legacy data acquired over years of space research are important for enriching the data repositories. Additionally, AI approaches have been applied to larger collections of journals to help simplify the steps of systematic reviews and were demonstrated to provide faster results for repetitive assessments (Wagner et al. 2021; Noel-Storr et al. 2021a). However, the AI algorithms tend to struggle in cases of more complex decisions, such as making judgment calls that require more thoughtful consideration during full text screening or data curation (Wagner et al. 2021). For the project examining legacy data, it can be anticipated that the presentation of the data will vary widely due to changes in settings and methodologies, as well as publication norms, which potentially limits the use of AI in systematic reviews of this nature.
There are many diverse physiological adaptations resulting from spaceflight exposure. For example, bone and cardiovascular health have been shown to be critically affected by spaceflight (Afshinnekoo et al. 2020). We have previously used knowledge synthesis approaches to find and analyse legacy data of bone health in human (Stavnichuk et al. 2020), primate, and rodent (Fu et al. 2021; Goldsmith et al. 2022) space travelers. Spaceflight-induced changes in bone were found to be skeletal site-dependent with significant bone loss in the lower body (legs and pelvis) but not in the upper body (Stavnichuk et al. 2020). Microgravity also affects the cardiovascular system, resulting in fluid redistribution, alterations in blood pressure, and changes in cardiac output (Sy et al. 2023), which can locally alter bone environment in microgravity. To examine the contribution of cardiovascular changes to bone loss in microgravity, we conducted a systematic review and meta-analysis of cardiovascular changes induced by actual or simulated spaceflight.
Our previous knowledge synthesis studies of spaceflight-induced bone changes (Stavnichuk et al. 2020; Corlett et al. 2020; Fu et al. 2021; Goldsmith et al. 2022; Moussa et al. 2022) demonstrated that large-scale literature screening is required to identify, organize, and curate publications over the past 60+ years of space research. Starting in 2017, we screened 5,713 publications regarding bone health in human space travelers, followed by complex analysis for overlapping datasets (Corlett et al. 2020) leading to project completion in 2020 (Stavnichuk et al. 2020). The library for bone health in spacefaring animals contained 15,342 items (Fu et al. 2021), which required two years of screening and curation. Though lengthy, these projects were an excellent mechanism for involving undergraduate students with space biology projects and allowing them to contribute meaningfully. In fact, there is current development of online platforms such as Cochrane Crowd (https://crowd.cochrane.org) that facilitate citizen science participation in the literature review process (Nama et al. 2019). A feasibility study of the efficacity of crowdsourcing in screening for systematic reviews showed that the “crowd,” or general population, was able to include relevant articles at rates of 84% and above, with accuracy for exclusion being 98% (Noel-Storr et al. 2021b). While these results are promising, authors suggest further research is needed to explore customised training to improve time-efficiency and accuracy (Noel-Storr et al. 2021b). Drawing from these projects, we aim to describe our preliminary approach toward using citizen science, a methodology of recruiting public help (Dosemagen and Parker 2018; Wehn et al 2021), to advance literature reviews, while also improving legacy data findings and curation. We connected with and invited individuals underrepresented in science, technology, engineering, and medicine (STEM) (first-generation, disadvantaged background, female, minority, disabled, etc.) to bring space science worldwide through our open science initiatives.
The goal of this project was to find all relevant cardiovascular data (from the 1960s to present) from rodents and humans exposed to actual and simulated spaceflight. This paper documents the protocol of designing, creating, organizing, and implementing a citizen science approach (Okop et al. 2021; Skarlatidou et al. 2019) to help complete a large-scale systematic literature search and screening for these data in the published literature.
Protocol
Initial project development
During the conceptualization phase of the project, we presented our plan to perform a systematic review of spaceflight-related cardiovascular outcomes to the ALSDA Analysis Working Group (AWG) in March 2022. The AWG (https://awg.osdr.space/; currently ~500 members) is a community of scientists worldwide around OSDR; it began in 2017 to provide feedback to NASA GeneLab on data standards for omics data. The ALSDA AWG began in 2021 to provide feedback to NASA ALSDA around standards for physiological-phenotypic data and metadata, which led to OSDR being released publicly in 2023. Another purpose of the entire AWG community is to mine and collaborate on projects for new knowledge in the field of space biology and health. These AWG collaborations often result in peer-reviewed publications (https://osdr.nasa.gov/bio/data/publications.html). Most notably, a number of articles with AWG authors were part a package of publications across Cell Press in 2020 (https://www.cell.com/c/the-biology-of-spaceflight), and a separate package across Nature Portfolio in 2024 (https://www.nature.com/immersive/d42859-024-00009-8/index.html). Across the dozens of publications in the latter package of articles, 111 AWG members were authors.
At the 2022 meeting of the ALSDA AWG, twenty-five subject matter experts expressed their interest in collaborating with this citizen science project. Over a four-month period, monthly meetings were held with interested AWG experts to discuss the best possible approach to reviewing such a large set of literature and potential project roles for both experts and participants. The roles requiring varying levels of commitment in two domains of i) literature review and ii) analysis were identified (Figure 1). It was determined that this group of experts would serve primarily as a consultation group, similar to how the larger ALSDA AWG provides feedback on NASA ALSDA data standards. Finally, in preparation for the search, we developed a list of relevant search terms (Table 1) and a strategy for project implementation, and we outlined outcomes for the study.
Figure 1
Overview of potential project roles suggested for each study participant.
Table 1
Focused list of cardiovascular- and microgravity-relevant search terms.
| TIME | OUTCOME | POPULATION | INTERVENTIONS | COMPARISON | STUDY DESIGN |
|---|---|---|---|---|---|
| 1950-present | Cardiac Measurements Cardiac output Left ventricular mass Pressure Blood pressure Mean arterial pressure Vasculature Arterial stiffness Blood volume Heart Rate Heart rate variability Respiratory rate | Humans Astronaut Cosmonaut Rodents Primates | Actual microgravity Space flight Weightlessness Parabolic flight Simulated microgravity Dry immersion Head down tilt Hindlimb suspension | Pre-flight Ground control In flight centrifuge control | Review Experimental |
A multi-center team of four medical librarians, two from Florida State University, United States, and one each from McGill University, Canada, and Dartmouth College, United States, developed and executed the search strategy in Medline, CINAHL, Embase, and selected NASA-supported archives, yielding 18,837 studies, which were imported into the Covidence systematic review management software (www.covidence.org). We have used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) method to report the literature that we reviewed in the process. Figure 2 shows the PRISMA diagram for the search, which visually describes that we initially identified 18,837 papers, of which 62 were duplicates. After screening of titles and abstracts of 18,775 papers, 3,539 were included for the full-text screening. Of these studies, 981 described cardiovascular outcomes in humans after the exposure to actual microgravity. Included NASA articles were chosen by experts who had submitted relevant data to OSDR. It should be noted that prior to importing to the Covidence, the results from the Medline library were subtracted from the search results of CINAHL and Embase, resulting in much smaller libraries.
Figure 2
PRISMA diagram for the literature search.
Recruitment and initial training of citizen scientists
In May through June 2022, while terms were being developed and the search was being executed, we recruited individuals from global space industry–affiliated organizations, aiming to reach young individuals from underrepresented backgrounds in STEM, including those identified as first-generation, female, minority, from disadvantaged backgrounds, etc. (Allf et al. 2022). The recruitment was led by ASN, who reached out to Students for the Exploration and Development of Space (https://seds.org/) and Space Generation Advisory Council (https://spacegeneration.org/) in collaboration with project coordinators from these organizations to advertise and recruit within these groups. Within a week of the project invitation announcement, we received notification of interest from more than 100 individuals from 28 countries with a variety of educational and professional backgrounds expressing their interest to join the project (Figure 3). Backgrounds varied and included undergraduates, graduate students at master’s and PhD or MD levels, postdocs, those employed in industry, and community college students. What should be noted is that volunteers with professional backgrounds were included in the recruitment of these citizen scientists. Primarily, the volunteers with professional background included postdoctoral scientists and full-time employees in industry (the type of industry was not specified in the questionnaire). When these groups apply their time and skillsets outside of their typical workload and during their own personal time, as in our study, these members can be seen as citizen scientists (Haklay et al. 2020). Using this framework, they then receive the benefits of participating as citizen scientists, adhering to the third principle of citizen science from the European Citizen Science Association (ECSA 2015). By including volunteers with professional backgrounds (both in and outside the field of interest), we can reduce the load on project management by providing opportunities for more responsible roles and by devoting more time to training the other citizen scientists. Despite the variety in educational backgrounds, there was no notable difference in performance between backgrounds among the participants who were actively involved in the project. After recruitment, the project leadership completed a series of weekly lectures including introduction to space science, introduction to cardiovascular physiology (in general and in the context of space research), project structure, knowledge synthesis methodologies, and a workshop on the Covidence software (Table 2). For the duration of the project, we maintained office hours, where at least one of the project leaders was available to answer questions and provide additional information.
Table 2
Project lecture division.
| LECTURE | LECTURER | LECTURE TOPICS |
|---|---|---|
| Introduction to Space Science | SVK and ASN | How is space research done, historically and in the present time |
| Introduction to Cardiovascular Physiology | ASN | Components of the cardiovascular system; Cardiovascular research |
| Project Structure | SVK and MN | Study motivation; Project goals; Project roles; Project timeline |
| Knowledge Synthesis Methodologies | MN | Generating research questions; Literature search; Inclusion/Exclusion criteria; Title/Abstract screening |
| Covidence Software | SVK and MN | Screening tools, Screening practice |
Figure 3
Citizen science recruitment demographics. Top left: Geographical location of members. Bottom left: Educational background level of members. Right: Educational background of members who continued (red) or not (blue) with the project, expressed as the percentage of the total for each group.
Screening strategy and success
Each of the 18,837 entries in the libraries required two reviewers, randomly suggested by the Covidence software, as per best practice (Page et al. 2021), and 15–20% of studies needed a third screener to solve the conflicts, bringing the number of individual screening decisions to ~40,000. After the initial training, about 40% of the participants began the title/abstract screening, which advanced at a rate of ~5,000 articles per month. After the McGill team screened ~10% of the studies, potential uncertainties in decision-making were identified. One example of such uncertainty was a distinction between parabolic flights (that were to be included) and airplane flights (that were to be excluded, but often did not provide a clear identifier in the title/abstract) (Table 3). Screeners, unsure about the relevance of the paper to the topic, would read the abstract in depth, which increased screening time and demotivated the screeners, making the screening of thousands of articles extremely daunting. After documenting potential sources of uncertainty, one-on-one instructions to the top 25 screeners were provided. After this, the title/abstract screening rate increased to 60,000 articles per week, and the remaining 75% of the library was screened in only two weeks (Figure 4).
Table 3
Screening uncertainties.
| UNCERTAINTY | IDENTIFICATION | DEFINITION | RESOLUTION |
|---|---|---|---|
| Parabolic flight versus commercial flight | Trends with “Maybe” decisions | Must include “parabolic” in the abstract/methods | Additional screening workshops |
| What is actual spaceflight/actual microgravity? | Questions from citizen scientists directly | Any study that went to space/parabolic flights (regardless of reported cardiovascular outcome) | Modification of inclusion criteria |
| Levels of gravity (partial to hyper) | Project leadership screening | Partial gravity = <1g – include; Hyper gravity = >1g – exclude | Clarification of inclusion/exclusion criteria |
| Austere environment studies | Project leadership screening | Isolation/confinement studies were excluded | Clarification of exclusion criteria |
Figure 4
Left: Screening time before and after individual instruction. Right: Screening by participants. Shaded area denotes contribution by McGill team.
Although the nature of the interactions with the citizen scientists did not allow for the full analysis of the differences between the “top 25 group” and the “dropped group” that did not continue their work with the project, our initial observations are that graduate (doctoral) students were more likely to continue with the project, those with community college degrees and industry members tended to not continue their participation, and undergraduate, graduate (masters) students, and postdoctoral fellows were similarly present in both groups (Figure 3). Further studies are required to fully evaluate the barriers to participation.
Although participants who decided not to continue were not part of further screening stages, they will be contacted for their feedback about their experience in the program and will be provided with an opportunity to contribute to the data-upload stage of the project, in keeping with the ECSA principles, which describe providing people with the opportunity to participate in multiple stages of the project and allowing them to give and receive feedback from the project (ECSA 2015). In the future, we plan to qualitatively analyze the feedback of all citizen scientists to understand the participants experiences and to enhance this citizen science protocol. All data resulting from this project will be uploaded into NASA’s OSDR, which adheres to ECSA principles of citizen science by producing genuine scientific outcomes, by providing citizen scientists the opportunity to see how their work is being used and its implications, and by making the data open access (ECSA 2015).
Based on this experience, we recommend that in future knowledge-synthesis projects performed with the help of citizen scientists, the project leaders first screen 10–15% of the library to identify uncertainties relevant to the particular project and search strategy, after which these nuances can be incorporated into the initial training. In addition, since citizen scientists may have varying experience in reading scientific literature, we recommend reviewing the specifics, nuances, and differences between screening and reading, to improve screening time. Moreover, we suggest setting a conservative title/abstract screening goal to remove clearly irrelevant papers, while retaining potentially relevant ones. During the screening process, 83.3% of the articles were agreed upon by both reviewers, and 3,148 conflicts were resolved by one of the McGill team members (who conceptualized the project) because of their familiarity with the project scope. Once all conflicts were resolved, 18.8% of the original articles were included for full-text screening (3,539 articles). Each included paper was annotated with labels in 5 categories: 1) language (English, Russian, other); 2) population (human, rodent, primate, other); 3) exposure (actual microgravity, simulated microgravity, other); 4) focus (cardiovascular, hematopoietic, other); and 5) publication type (primary, review, other). For the exposure category, if the paper was in the simulated microgravity category, it received an additional label for the method (dry immersion, immersion, head down tilt, bed rest, immobilization, hindlimb suspension, other). In this project, we performed labeling as a separate activity after completing the title/abstract screening; however, after reflection on the study, we suggest that the project completion time can be improved by incorporating labelling during the screening. In this case, a longer training period should be anticipated.
The library was then divided into four sub-projects: human (~25% of the library) and animal (~5%) actual microgravity, human simulated microgravity (~30%), and animal simulated microgravity (~15%), (Figure 5). Although initially we proposed to divide citizen scientists into groups for the four sub-projects, a survey of the participants determined that 20 of 25 participants wanted to focus on humans and actual microgravity. Therefore, it was decided to first analyze the humans and actual microgravity library using all the citizen scientists to quickly complete one section of the project, while documenting the protocol to streamline this process for the other sub-projects.
Figure 5
Full text article division amongst desired subgroups.
Full-text screening
Although most literature review programs can quickly search the internet to find full-text PDFs of articles, this was not the case for this project. Following quick-text upload efforts from McGill team members and librarians, only 10% of the included articles for full-text screening had associated PDFs (using EndNote), as opposed to the expected 50–60%. After further discussion, it was determined that one of the main causes was the timeline of our search, as we required many articles from pre-2000, which may not have been archived electronically properly or at all. Including citizen scientists with their institutional access at this stage allowed us to quickly find and retrieve the associated documents. As a result, 71.2% of the humans in actual microgravity library full-text articles were found within three weeks. Approximately 20% of this library included foreign language articles, with the vast majority being in Russian, of which three members of the McGill team are fluent and will complete the full-text screening. Additional articles in other languages will require the recruitment of other citizen scientist participants through McGill’s peer network center, which in our experience allows immediate access to users of 20 to 30 different languages. Again, using a citizen scientist approach allowed us to drastically reduce the time required to complete simple, yet time-demanding literature review tasks.
Project Limitations
While the citizen science approach continues to show its use in simplifying literature-mining research, several limitations were identified:
Misunderstanding of the project time/commitment requirements by citizen participants. Although we had ~100 initial individuals respond and join the introductory lectures, when screening began, approximately 50 people screened at least 1 article. Of those 50, 25 screened more than 1% of the library, which we determined to be a significant contribution to the project. In other words, only 25% of the individuals who responded to our project announcement stayed on and provided significant contributions.
Some steps had to be completed by the project management team, extending the time between tasks that engaged citizen scientists. For example, article labelling took three months, during which some participants lost interest or had significant change in their circumstances that prevented their continuous participation.
Oversight is needed. No matter the stage of the project, leadership needed to be available to address questions, review progress, and update the project plan accordingly. Therefore, this type of work cannot be completed independently of expert supervision. As citizen scientists are not experts in the project field, this additional supervision is required for quality control of the research.
Locating foreign language articles (particularly numerous articles in Russian, which were prevalent because of the nature of the project). Due to several reasons (disparities between institutions, publication time, lack of full-text translation, etc.), foreign language articles required significantly more time and effort to obtain. All the screeners were fluent in English, and the core team had additional knowledge of French, German, and Russian, which allowed us to use professional international connections to retrieve as many articles as possible.
Although all citizen scientists were able to speak English, there was no explicit language proficiency test to rate their fluency. In our interactions with citizen scientists, we did not experience language proficiency limitations. Nevertheless, it would be important to examine how fluency affects screening skill and speed in future studies.
Future Directions
We plan to continue working with the 25 dedicated citizen scientists in the following steps:
Validate the screening process by i) asking ALSDA AWG investigators to verify that all papers known to them were selected in the final library, and ii) assessing the reference and citation lists of the reviews.
Full-text screening will be performed, focusing on sub-projects led by the core team members and supported by citizen screeners. Inclusion criteria will be a quantitative reporting of any cardiovascular outcome in actual or simulated spaceflight.
Papers selected after the full-text screening will be annotated by cardiovascular outcome type and measurement methodology.
Coding tables will be developed that will combine manuscript and mission/study level covariates with data for assay configurations and data templates for upload to OSDR, in collaboration and consultation with OSDR project coordinators.
The protocol for data extraction, curation, and upload to OSDR will be finalized, with data to be compiled, processed, and curated into OSDR, where it can be openly accessed and reused.
We plan to formalize our project into a course platform where citizen scientists can also receive course credit for their involvement and contributions to these and future projects.
Meta-analysis of extracted outcomes will be performed, and secondary analysis for the effect of mission duration, participant age/sex, etc. will be performed when possible.
Furthermore, integration of the data on cardiovascular health into the AI-ready OSDR repository will enable the use of novel modeling approaches to simulate, predict, and understand spaceflight-induced cardiovascular deconditioning.
Credit allocation
Originally, participants were recruited for their interest in the subject matter. As the program grew, it is now officially recognized by NASA as the SOLSTICE (Space Open Life Sciences Team for International Collaborative Exploration) program. A certificate of participation in SOLSTICE will be provided to all participants involved in our project who complete all elements of the online training provided by our program. The significant contribution (defined by screening a minimum of 1% of the initial library) of 25 citizen scientists who participated in the title/abstract screening was discussed and will be acknowledged in all the published manuscripts. Citizen scientists who contribute to the full-text screening, data extraction, and upload into OSDR will be offered a co-authorship on relevant manuscripts, and local conference presentations about their contribution and experience will be supported. The determination of credit allocation was discussed by the project leadership at the beginning of the project. The credit allocation follows the eighth ECSA principle (ECSA 2015).
Conclusions
We report a successful integration of an open science—citizen scientist approach in a large-scale, complex space biology project. We also provide recommendations for project conceptualization, development, recruitment, training, management, and progress, which improve the experience and effectiveness for all project participants. Recently, AI-automated approaches have been introduced for title/abstract screening and have been demonstrated to save project time (Hamel et al. 2021; de la Torre-López et al. 2023). Our experience suggests that engaging citizen scientists provides similar time saving, while also providing opportunities for active learning for the participants (Torralba and Doo 2020; Haklay et al. 2020) in the subject of literature review as well as in open science approaches and accessibility. Other similar citizen science projects in the field, such as the Cochrane Crowd project, focus on facilitating citizen science engagement by providing opportunities and training videos (https://crowd.cochrane.org), in contrast to the individualized training and communication used in our project. By involving citizen scientists during the title/abstract screening, we provide them with the experience and training necessary to make the more nuanced decisions during the full-text screening stage. Our project also provides citizens scientists with exposure to data curation through NASA’s OSDR portal, promoting open access science. Additionally, more genuine connections developed through this project provide citizen scientists with professional and educational opportunities to connect with mentors and experienced experts in the field of study. Since these gains come at the expense of providing training and oversight for the project, we suggest that projects for which providing the public with necessary scientific education and insight is of critical importance, such as during COVID-19 (Rowbotham et al. 2023), would be strong candidates for employing citizen science approaches. The combination of AI-driven initial screening with full-text screening by trained citizen scientists to complete the more nuanced tasks (Wagner et al. 2021) is also a promising approach. For space research, a citizen science approach will expedite project progress (facilitate data screening, curation, and database upload), address knowledge gaps, and increase public interest and engagement in space science. This process will be useful for other scientific areas that would benefit from legacy data incorporation, such as rare diseases, natural disasters, or rare social events. Indeed, our pilot project aims to build the framework for future investigations to accelerate scientific progress through opening access to space exploration science and data for all.
Data Accessibility Statement
Raw data can be made available upon reasonable request to author Mattias Neset (mattias.neset@mail.mcgill.ca). This citizen science program is listed as SOLSTICE (https://osdr.nasa.gov/bio/awg/solstice.html) through NASA’s open science initiative and there are no limits on the intellectual property. Extracted data from the literature search will be published to NASA OSDR (https://osdr.nasa.gov/bio/).
Acknowledgements
We acknowledge all the citizen scientists associated with SOLSTICE program and members of Ames Life Science Data Base Archive Analysis Working Group.
Funding Information
This work was supported by operating grant from Canadian Space Agency (21HLSRM03) to S.V.K.
Competing Interests
The authors have no competing interests to declare.
Author Contributions
All authors contributed to the conceptualization of the project. M.N. and S.V.K. managed the screening process. A.S.N. lead the recruitment of citizen scientists. All authors taught individual lectures. M.N. prepared the manuscript. All authors provided input, reviewed, and approved the manuscript.