Quality of Available Cardiovascular Disease Knowledge Tools: A Systematic Review

Inclusion and exclusion criteria

Criteria were based on the PICO format, however given the nature of test development, comparators were not considered. We were interested in patients with CVD or individuals in the general population that were not trained in a health profession (Population). We were interested in CVD knowledge test development (Intervention) and their psychometric properties (Outcome). We included primary studies that developed and assessed knowledge of CVD in clinical or general populations. Knowledge tests for health professionals were not considered. CVD knowledge was defined broadly to include basic CVD physiology, prevention, and management (clinical and behavioural risk factors). Eligibility criteria included: the tool is free for public use, the paper provides information on reliability and validity, and the paper is written in English (but the knowledge test did not have to be written in English).

Search strategy

Three core search concepts and their related search terms were reviewed by a cardiologist and CVD patient education expert. Each concept was combined with a proximity operator (e.g. W3 or N3) as assessment tools typically have descriptive names. The three search concepts were i) Heart, ii) Knowledge and iii) Tests. Heart consisted of the terms: “heart”, “cardiovascular”, and “vascular”. Knowledge consisted of the terms: “knowledge”, “comprehension”, “understanding”, “awareness”, “health literacy”, “literacy” and “self-management”. Test consists of the terms: “test”, “exam*”, “asses*”, “evaluat*”, “check” and “tool”. Scopus, Medline, CINAHL, PsycINFO and PsycTESTS were searched between inception and October 2022 (see Appendix 1 for full search strategies). Search results were uploaded to Covidence where screening and full text review were completed. Two similar previous reviews (11, 12) were also identified and whose reference lists also were added to the final list.

Search methods for identification of studies

Abstract screening was performed by two authors (MF, SC), and discrepancies in screening were addressed at full text review. Two authors (MF, SC) completed full text assessment and a third author (CB) resolved discrepancies. Initial full text agreement prior resolving discrepancies was 86% (Cohen’s κ =.74).

Risk of bias

A data extraction form was created for the purposes of this review (see Appendix 2). The psychometric grading framework (14) (PGF) was used to assess the quality of these papers. The PGF is based on commonly used statistical test and values recommended by psychologists and biostatisticians. It provides an evaluation grade between A to D based on the tool’s content, construct and criterion validities, test-retest reliability, and internal consistency. For each of these sections, the PGF provides specific criteria per grade (e.g., for internal consistency, an A grade requires Cronbach’s alpha to be greater than or equal to .90). Inter-rater reliability was not considered as it was not applicable for knowledge scales. Additionally, knowledge scale development may involve testing group differences to establish criterion validity, such as those diagnosed with CVD compared to those who have not been (the assumption being that those with CVD would know more about their condition). The PGF does not differentiate between confirmatory and exploratory factor analysis for the purpose of assessing construct validity (14). When determining the overall grade, papers with insufficient information to evaluate a specific PGF domain were considered a “D” grade whereas those with relevant information which did not fit PGF criteria were considered case-by-case (e.g., when translation studies cited studies that demonstrated criterion validity previously). An overall A grade means the psychometric strength of the instrument is ‘good’ and can be given to an instrument with mostly A’s and some B’s and or C’s. The test development is also described and discussed in the context of educational test development. This included pilot testing, item writer training, predetermined pass levels, qualitative answer rubrics, expert input and type of questions (15).

Data extraction and synthesis

Data extraction was performed by one author (MF) and then reviewed by a second author (CY). Corresponding authors were contacted for missing data. Of the five emails contacted, one was no longer active and four have not returned responses by the submission of this manuscript. Pooled estimates were not considered appropriate for this review. Descriptive statistics were used to summarise characteristics. Knowledge test domain areas were extracted as defined by study authors, thematically organised and then applied to all included knowledge tests to obtain a frequency table of areas assessed. All data and supporting materials have been provided with the published article.

Search results

Figure 1 shows that this search yielded a total of 28 papers: 18 development and 10 translation studies (i.e. a study that validated a knowledge test that was originally written in English then translated to another language or vice versa). The final list can be found in Table 3. We identified 15 additional papers that were not captured in earlier reviews (11, 12).

Figure 1

Knowledge test characteristics

Table 1 describes the key characteristics of the CVD knowledge tests and samples used to assess the psychometric properties of the scale. All studies focused on adults except Cecchetto and Pellanda (16) which was the only child knowledge scale. Table 2 describes the domains of each knowledge test by study. Knowledge of risk factors for CVD was the most common content area measured (10 development studies, nine translated language studies), followed by nutrition and diet (10 development studies, seven translated language studies), medical knowledge/pathophysiology of the heart (nine development studies, six translated language studies) physical activity and exercise (six development studies, five translated language studies), treatment options (eight development studies, one translated language study), symptoms and signs of heart failure/disease (five development studies, two translated language studies), self-care and management (seven development studies, zero translated language studies), psychosocial risk (one development study, four translated language studies) and epidemiology knowledge of CVD (one development study, one translated language study). All tests were multiple choice items (either true/false or select best answer). All scoring for tests was predetermined and typically followed one point per correct answer where higher scores indicated higher knowledge levels.

Test development characteristics

All studies except Butts et al (17) described an acceptable level of expert input as per the PGF towards the development of scale items, however this was because Butts et al (17) was a revalidation study of the Atlanta Heart Failure Knowledge Test Version 2, so expert input was already accounted for. For language translated studies, the translations were completed by bilingual researchers. No study clearly indicated any authors received item writing training (15). All studies except Butts et al (17) and Frayssac et al (18) described pilots. Pilots varied in terms of samples used (general population, clinical, professionals), implementation (single samples, or follow-up samples) and reported outcomes for pilots (clarity, content, understanding). No study identified an a priori level of desired or acceptable knowledge. One study, John et al (19) had parallel forms (two versions of the same form that can be used to minimise practice effects when testing over two time periods). No included study correlated knowledge test score with patient outcomes.

Psychometric properties

Table 3 provides the gradings from the PGF per study ordered by surname within unique and translated language study groups. Two unique studies, Khazaei et al (20) and Thanavaro et al (21), one translated language study, Yang et al (22) obtained an overall A rating. Of the remaining studies, four received an overall B rating, 12 received an overall C rating, and nine received an overall D rating.

Key findings

This review identified a total of 28 papers, 18 of which were development papers and 10 were language translation papers. This review added 15 new papers when compared to previous reviews conducted by Vieira et al (11) and da Rocha et al (12). Collectively, knowledge tests assessed a wide variety of domains within the context of CVD physiology, assessment, and management. The five most frequent domains assessed by included tools were Risk Factors, Nutrition/Diet, Medical Knowledge of Heart/Pathophysiology Physical Activity/Exercise and Treatment Options. Uncommon domains included thoughts and feelings around heart failure and female specific CVD risk factors that may highlight gaps in the literature around heart knowledge.

The psychometric quality of these tools is quite variable, and none were validated against health outcomes. Khazeai et al (23), Thanavaro et al (21) and Yang et al (22) were assessed to have an A rating. As our findings are limited to papers related to the search criteria used, it resulted in a more comprehensive list of knowledge tests than previous reviews. Other tests, however, did not appear in our search, namely tests that are specific to nutrition (24, 25). Readers seeking CVD knowledge tests for specific patient populations or specific risk factors may need to conduct additional searches using relevant terms to find additional tools.

Implications and future directions

CVD knowledge test developers are recommended to have questionnaire item writing training (15), and trained item writers should be involved early in the test development phase (26). Pre-determining an acceptable level of CVD knowledge may support clinical use by providing a minimum level of knowledge that is acceptable for a specific purpose, such as informed consent. This pre-determined level, however, would need to be balanced with the need for variability in measures for statistical development purposes, which may be the reason why very few papers discussed this. Assessing knowledge scales against health outcomes is typically done separately to the initial test development once the traditional psychometric properties of the assessment tool are sound. If validation against health outcomes was conducted as part of the psychometric properties, it would provide a more direct clinical utility to understanding the construct of knowledge and its impact on health outcomes earlier in the process. For example, measuring current self-help behaviours and correlating this variable with self-help domains in knowledge scales.

Given the impact of disease specific knowledge on self-management, clinical uses for CVD knowledge tests can help tailor to patient needs (10). For example, a CVD knowledge test after a cardiac event may be helpful to identify additional education support needs for self-management; whereas a CVD knowledge test in an asymptomatic individual may be an awareness raising tool to prompt a CVD risk assessment with their doctor. Only seven identified papers included self-management questions, which may limit the pool of tests that are useful in cardiovascular settings. Clinicians may benefit from a centralised bank of CVD knowledge questions, with domain specific areas that can be mixed and matched for different inpatient and outpatient settings. Validated questions may be used more qualitatively in practice, to enable clinicians to use these tools in a flexible manner tailored to individual patient needs. Translation services could allow these tools to be adapted to additional languages around the world. Future research could investigate how validated questionnaire items need to be tailored to patient populations with different health conditions, socio-economic position and health literacy status. The understandability and face validity of the questions for patients with lower health literacy should also be considered.

Strengths and limitations

Drawing on educational test methods provides an additional lens to evaluate the quality of disease-specific knowledge tests (15). However, the PGF may not be adequately equipped to address the difference between a confirmatory factor analysis and exploratory factor analysis, which negatively impacted some ratings. Final PGF ratings permit an overall A rating even when instruments receive B or C grades in specific domains (but not D grades), acknowledging that variability or omission in some domains may not necessarily render an instrument inadequate. The search strategy was more comprehensive than earlier reviews but did not specifically search for risk factor knowledge tests or specific cardiovascular populations. Furthermore, publication bias may be considered given this review did not source grey literature, however, given the variability of PGF ratings, the impact of publication bias could be considered minimal.