East Meets West: A Multisite Validity Study of the China Medical Professionalism Inventory

Study design and ethics

We generated validity evidence aligned with Messick’s unified validity framework [31] and the current Standards for Educational and Psychological Testing [32]. We undertook a 2-phase process to develop the China Medical Professionalism Inventory (CMPI): Phase I to create the item pool; Phase II to delineate validity evidence, including content, internal structure, and relationships to other variables. Figure 1 summarizes the methods, participants, and items involved. Appendix 1 details processes, analyses, statistical terms, and standards. China Medical University’s Bioethics Advisory Commission (institutional review board), approved the protocol (ID: 2017075).

Figure 1

Phase I: Development of item pool

We reviewed the literature reporting instruments that measure MP, combining key terms: professionalism AND physicians AND instruments or assessing AND psychometric properties.

We extracted items [13], and organized and eliminated duplicate items, using the framework commended by Lesser et al. [36]. We translated these items into Chinese. To develop evidence for cross-cultural validity, we followed Brislin’s modified model, an accepted process of forward-back translation and expert review [37, 38]. To support this instrument’s representativeness with the Chinese healthcare context, we supplemented the item pool with MP-related Chinese government policies [39, 40, 41]. We converted items expressing attitudes or values into behavioral terms (e.g., “have commitment” was changed to “demonstrate responsibility for”), to create CMPI Version 1.

Phase II: Delineation of validity evidence

We followed a standard practice [34, 42, 43] by inviting experts to review the CMPI Version 1 items. This process produced CMPI Version 2. We then conducted a series of psychometric studies to collect reliability and validity evidence as follows: the first psychometric study of CMPI Version 2 which led to CMPI Version 3; the second psychometric study of CMPI Version 3 which led to CMPI Version 4; an expert panel to shorten the survey and third psychometric analysis to describe the validity evidence of the final version CMPI (hereafter, “CMPI”) (Figure 1). Below we present participants, procedures, and analyses, organized by domain of validity evidence.

Validity evidence for content

In Phase I, we extracted all items from 63 tools, eliminated duplicates, and added Chinese policy items to produce CMPI Version 1 (Figure 1). In Phase II, the first expert survey included 34 (85%) completed questionnaires from examination experts. Experts represented 27 medical schools throughout China; each was ranked “first class” on the Chinese government’s official listing. We eliminated 82 items (71 from our original literature review and 11 from Chinese policies) the experts determined were less related or less important. The resulting 56-item pool included 17 items sourced from policy documents. The experts recommended two additional items: “respects patient autonomy and their informed decisions” and “shares experience, skills, and knowledge with junior colleagues,” resulting in a 58-item CMPI Version 2 (Table 1). In Phase II, the second expert survey included 76 clinicians (response rate, 63%) who ranked the importance of each item. For each factor confirmed by CFA, we retained the top 50% of items to generate the 20-item CMPI (Table 1 and Appendix 2).

Evidence for content was supported by ensuring that all survey participants were clinicians. The first survey’s 34 experts, the second survey’s (different) 76 experts, and survey respondents in the three psychometric studies (Table 2) all had clinical medical training and practice experience, representing an array of disciplines.

Validity evidence for internal structure

In Phase II, 360 physicians (response rate, 92.3%) returned the first psychometric study, and 3653 physicians (response rate, 89.8%) returned the second psychometric study. For the third psychometric study, 955 clinicians (response rate, 95.3%) completed the CMPI. Table 2 presents the socio-demographic characteristics of these physician respondents.

For the first psychometric study, we deleted 14 items that did not meet CITC and CAID standards [52, 53], removed 19 surveys (5%) with missing data, and analyzed 341 responses for EFA. Using the remaining 44 items for EFA, both Bartlett’s Test of Sphericity (χ² = 16,281, P < 0.001) and the KMO test (0.98) indicated that the data were factorable. EFA determined four factors (eigenvalues >1.0). Three items did not meet the predetermined standard (factor loading ≥0.45); thus, CMPI Version 3 retained 41 items for the second psychometric study. We labeled the four factors according to the common theme shared among the items: “Respect, Compassion, and Communication”; “Integrity”; “Excellence”; and “Responsibility.” The four factors account for a total of 66.65% of the scale variance, an acceptable standard [43].

For the second psychometric study, we removed 149 surveys (4%) because of missing data and analyzed 3504 responses for CFA. Additionally, one item was removed: item 26 (“Actively reports any personal medical or research errors”). Following a standard process [33], we deleted this item because it loaded strongly on two factors on EFA: “Integrity” (0.45) and “Excellence” (0.46). The subsequent CFA provided a satisfactory fit to the 4-factor model (Table 3). This process yielded the 40-item CMPI Version 4 (Table 1).

Figure 2

For the third psychometric study, we removed 152 surveys (15.9%) because of missing data and analyzed 803 responses for CFA. After we adjusted for correlated item pairs, the subsequent CFA indicated that all 20 items loaded significantly (P < 0.01) on their respective factors (Figure 2). The CFA indicated a satisfactory fit for the 4-factor model (Table 3).

Additional internal structure evidence

For the first and second psychometric studies, Cronbach’s alpha coefficients for each factor were >0.9. The results of the second study produced CMPI Version 4 (respondents’ mean score = 182.5, SD = 20.46; median = 188; range of scores = 54–200). The third psychometric study provided additional evidence of the final CMPI’s internal structure (Appendix 3). The internal-consistency reliability of the total scale and four factors are all in a satisfactory level (Cronbach’s alpha >0.7). The final CMPI respondents’ mean score was 88.3 with SD = 20.47 (median = 89; range of scores 60–100).

Validity evidence for Relationships to other variables

Our results indicated that the CMPI correlated positively with the Chinese version of the PSPQ (Pearson r = 0.77, P < .01) and correlated negatively with the short version MBI (Pearson r = –.22, P < .01).

Chinese-derived items

After expert surveys and psychometric studies, the CMPI retained items derived from three sources: the initial item pool, Chinese healthcare policies not found in the initial item pool, and additional items provided by Chinese experts (Appendix 4). Items originating from policies and expert supplementation accounted for 25.0% of the total of the CMPI. The two items added by experts, “Demonstrates respect for patient autonomy…” and “Shares experience…with junior colleagues” (Table 1: items #8 and #48), were retained throughout the process.

Validity Evidence

We organize our results and validity evidence following Messick and the Standards frameworks: content, internal structure, and relationship to other variables [31, 32]. We developed evidence based on content by surveying two distinct groups of national experts from across China to assess each item’s fit, importance, and contextual relevance. We administered three psychometric studies to physicians to provide evidence based on internal structure of the 4-factor model. To assess relationships to other variables, we determined convergent evidence. The CMPI showed a strong positive correlation with the PSPQ that assesses MP attitudes and a negative correlation, albeit weak, with the MBI that assesses physician burnout.

The CMPI has 20-items, and the tool’s Cronbach’s alpha suggests room to reduce the number of items further [54]. For example, the CMPI uses self-reports of behaviors. To assess professional behaviors of other physicians (not oneself), future research could remove items that are more cognitive and not clearly observable (i.e., CMPI items #14 and #20) and collect validity evidence from that new CMPI version [15]. The process of reducing the items could also provide information on those new tools’ relative test-retest and interrater reliability.

The CMPI supports the beginning of a process of MP assessment in China, and potentially beyond. Such tools could even support improved care of patients [14]. Subsequent iterations of the CMPI will require new validation evidence in the context of its use, judgment about scoring, generalization, extrapolation, and implication. Future research should include consequential validity, response process [32] and “qualitative and subjective data” from “multiple assessment data points” (p. 561) [34]. Ultimately, tool development should involve “the accumulation of evidence across time, settings and samples to build a scientifically sound validity argument….an ongoing process” (p. 465) [59].

East Meets West

Just as Western tools are used in the East, one could imagine that instruments developed in Eastern contexts, with validation evidence, could inform Western conceptualizations of MP. How much convergence and divergence occurs when the CMPI, derived in one Eastern context, “meets West”?

The CMPI appears to have integrated elements of traditional Chinese culture and contemporary policy requirements with Western conceptualizations of MP. We developed the item pool from Western MP assessment tools, Chinese policies, and Chinese experts. Appendix 4 documents the CMPI’s Chinese-derived items—those retained and those lost in the development process. Appendix 5 broadens our lens and compares multiple Western and Eastern tools.

As one example, we compare the CMPI to a widely applied Western framework, the Physician’s Charter [60]. Two observations stand out: (1) As with prior studies comparing Chinese and Western MP frameworks, many items in the CMPI overlap the Charter while some appear culturally particular [11, 16, 17, 18], and (2) some items appear to reflect differing current emphases in Chinese and Western contexts.

Regarding areas of overlap, 15 of the CMPI’s 20 items (75%) originate from the initial item pool before the Chinese policy items were added. Of the five Chinese policy items, four map to one or more of the core principles of the Charter: autonomy, welfare, and social justice [60]. For example, “Avoids conducting non-scientific or unethical research…” and “Does not provide unnecessary or excessive testing…” might be considered within the Charter’s “Principle of Patient Autonomy” and “Principle of Primacy of Patient Welfare” [60]. One CMPI item “Shares experience…with junior colleagues” is hard to map to the Charter’s principles.

Regarding areas of differing emphasis, we note the earlier CMPI versions retained fewer items about care equity. Another core element of the Charter, care access [60], was omitted entirely in the CMPI. We speculate that surveyed physicians and experts did not retain these items because in China, these behaviors are viewed as culturally foundational—so basic that they would not require MP assessment. In China, “social justice” has deep Confucian roots as a longstanding, expected, and enacted value across society. Since the pre-Qin Dynasty (c.261 BCE) [61], Confucianism has “emphasized human equality overall” and a communal Confucian ethic of “justice” (义, yi) [p. 199, 62]. These principles underpin the requirement of China’s universal health care system to provide access and just distribution of healthcare to all people—rural and urban, wealthy and less resourced, etc. [63, 64, 65]. The system prioritizes access over the Western notion of individual experience of care [66, 67]. The health system, like other sectors of Chinese society, animates Confucianism and supports China’s collectivist culture; we suspect that the transcendent, commonplace nature of these values obviated the impulse for respondents and experts to maintain those items.

As well, the contextually derived CMPI differs from the Charter where the CMPI added items connected to China’s recent 10-year national healthcare reform [68, 69, 70]. This reform emphasizes conflicts of interest, new payment policies, financial integrity, and ending the relationship between physicians’ income and drug dispensing and laboratory examinations [68, 70]. The CMPI also includes a high proportion of items related to communication, shared decision-making, and respecting patient autonomy, domains related to new conceptualizations of the doctor-patient relationship in China [69, 71, 72] and the East more broadly. This approach to MP reflects a change from earlier Chinese medical practice which was hierarchical and paternalistic, whereby patients (and junior colleagues) viewed doctors as powerful authority figures who direct patients’ care [69, 71, 72].

We observe another difference in emphasis between the CMPI and the Charter. In the 58-item CMPI Version 2 (Table 1), item #36 states physicians should “report colleagues’ misconduct…” In China, peer reporting is akin to “whistle-blowing”—a socially unacceptable behavior due to Confucian emphases on social conformity, harmony, and a commitment to trust and loyalty in relationships [73, 74]. In contrast, the Charter’s “Commitment to professional responsibilities” calls explicitly for “remediation and discipline of members who have failed to meet professional standards” and “individual and collective obligations to…accepting external scrutiny of all aspects of their professional performance” [60].

Finally, we believe that the development of the CMPI aligns with the Chinese traditional philosophy of “inclusiveness” and “harmony”—to learn and absorb ideas outside China while preserving core values and cultural foundations. To support and advance MP, this approach encourages a commitment to exploration and openness, grounded in respect for multiculturalism.

Limitations

This study has limitations. Our original and repeat literature searches could have missed tools and items. We conducted psychometric studies in Liaoning Province, and regional factors could influence results. Our sample demographics compared with China’s overall physician demographics [44], but unmeasured factors could influence results. We surveyed urban, tertiary hospital physicians and experts, missing views of rural providers. We note Chinese MP challenges are most reported among urban, tertiary-trained physicians [71, 72]. Our multidisciplinary experts were diverse geographically and diversely experienced, but we did not collect their demographic data. Although all participants were practicing clinicians, we did not undertake cognitive interviewing to produce evidence for response process.

Regarding evidence based on internal structure, our three rounds of surveys relied on self-reports, which has limitations [75] and risks social desirability bias [45]. We attempted to reduce this bias by wording questions for physicians as frequencies (i.e., how often), wording questions for expert reviewers indirectly [46] (“what should a physician demonstrate”), and informing survey participants and experts that surveys were anonymous, and participants were not individually judged or assessed. We did not develop consequential validity evidence.