Identifying Reliable Labeling Comprehension Questions for E-Vapor and Heated Tobacco Products: A Meta-Analytic Approach

INTRODUCTION

Labeling comprehension research is critical to the developmental life cycle of a product because it helps organizations: 1) assess whether prospective users of their products understand how a product functions and whether safety information related to product use is comprehended (e.g., not to use a device if its battery is leaking); and in some cases 2) ensure that products meet regulatory standards – for instance, market authorization of electronic nicotine delivery systems (ENDS) and heated tobacco products (HTPs) by the U.S. Food and Drug Administration (FDA) requires evidence demonstrating that “users (and consumers) are able to understand the labeling and instructions for use and use the product in accordance with those instructions” (2).

Labeling comprehension research typically involves giving prospective users paper or digital resources related to the product of interest (like user guides and/or safety guides) and asking a series of questions to assess comprehension of product use and safety information. The labeling comprehension assessment is often extensive, including many questions (e.g., upwards of 20) to comprehensibly evaluate whether a prospective user understands key safety and usability information related to product use. The methodology of labeling comprehension research can be tedious and resource-constraining because it often requires formative testing – an iterative process of multiple independent evaluations used to ensure that the relevant labeling comprehension interview questions are easily comprehended by participants, and that the questions consistently and accurately assess the intended metric (3). Stated differently, formative testing precedes the labeling comprehension study, and is needed to ensure that the required interview metrics are easily comprehended by the intended users and, critically, reliably assess an intended construct or domain (e.g., does the question ‘According to the materials, who should this product be kept away from’ successfully evaluate comprehension that a product should be kept away from children and pets?).

The iterative nature of these assessments increases the time required to conduct labeling comprehension research and reduces the needed sample pools in the population of interest (because respective samples for a labeling comprehension study and formative testing are drawn from the same population).

Extant literature offers little guidance on how to temper these methodological issues. Although WILLIS (3) offers robust guidance on how to conduct formative/cognitive testing, the resource constraints of metric-reliability assessments (i.e., formative testing) are minimally discussed, highlighting that “in one sense, a questionnaire could be tested forever” (p. 31), while stressing the value of a limited number of testing rounds. Similarly, there are varied examples of literature discussing guidelines and best practices for developing, but not evaluating, product instructions and safety information guides (4,5,6,7).

The goal of this work was to temper the need for formative metric evaluations in labeling comprehension research by identifying product-safety and product-instruction labeling comprehension questions that: 1) reliably demonstrate comprehension of target labeling information (i.e., are reliably answered correctly); and 2) are generalizable across research on ENDS and HTPs. We explored whether, across four labeling comprehension studies for three distinct nicotine products (one ENDS, two HTPs), we could meta-analytically identify labeling comprehension questions that consistently demonstrated high correct response rates (relative to a 100% correct response rate). We describe our meta-analytic approach below.

METHOD

RESULTS

A proportional meta-analysis was performed to assess the proportion of correct responses for each question domain. Table 2 provides pooled proportions (including 95% confidence intervals), tests of heterogeneity (Cochran's Q), and metrics for quantifying heterogeneity (I² and Tau²) for each question domain. See Supplemental Material (Appendix A and Appendix B) for an overview of study demographics and for forest plots of each meta-analysis.

The correct response rates for six of the 13 question domains (i.e., ‘Know to charge device only with proprietary accessories,’ ‘Know to not expose device to high temperatures,’ ‘Know to not use device if it becomes wet or immersed in a liquid,’ ‘Know to not use device if it appears damaged or broken,’ ‘Know not to use the device with other manufacturer's items,’ and ‘Know that the device heats the tobacco’) did not demonstrate statistically significant variability across studies (heterogeneity), indicating consistency in correct response proportions, across studies, for these six domains. Specifically, the pooled correct response proportions for these six question domains were ≥ 96%, with corresponding 95% confidence intervals ranging from 93% to 100%, an I² ≤ 44%, and a Tau² ≤ 0.004 – indicating that the proportions of correct responses for questions within these six domains were generally high (near a 100% correct response rate) and exhibited moderate to no variability (heterogeneity). The prediction intervals ranged between 85% and 100% for these six question domains suggesting that if the questions from each domain were asked again, in a new study, correct response rates would be expected to fall between 85–100%.

The correct response rates for a seventh question (‘Know that the product contains tobacco’) did not demonstrate statistically significant variability across studies (heterogeneity) either, but this finding should be interpreted with caution considering that Tau² is likely highly inflated in these analyses and is consequently widening the prediction interval PI = [0.28, 1.00]. Tau² tends to become inflated in proportional meta-analyses when the number of included studies is small (< 10; N = 3 for ‘Know that the product contains tobacco’ (9)), when study proportions near 100% (10) which is the case for the questions in the ‘Know that the product contains tobacco’ domain), and due to the bounded nature of proportions (cannot exceed 100% (11)). Thus, the wide PI is likely strongly overstating uncertainty, suggesting that the range of correct response proportions for this domain, in future studies, is likely much narrower. By contrast, the correct response rates for the remaining six question domains (i.e., ‘Know to not use device if it is operating in a strange manner,’ ‘Know to keep device away from children,’ ‘Know that the device contains nicotine,’ ‘Know how many times you can use the tobacco consumable,’ ‘Know how long the tobacco consumable will last,’ ‘Know not to use the device if the battery behaves unexpectedly’) demonstrated statistically significant variability (heterogeneity) across studies – indicating variation in correct response proportions across studies for these six domains. The pooled correct response proportions for these six domains were ≥ 86%, with corresponding 95% confidence intervals CI ranging from 46% to 100%, I² ≥ 82%, and Tau² ≥ 0.021 – indicating that the proportions of correct responses for questions within these six domains were generally high (near a 100% correct response rate) and exhibited high variability (heterogeneity). Prediction intervals ranged between 0% and 100%, suggesting wide expected variability in correct response proportions if the questions within these domains were asked within a new study.

DISCUSSION

Labeling comprehension research is necessary in the developmental lifecycle of a product, but it is also tedious and resource constraining because it often requires extensive formative testing. This project examined whether the need for formative metric evaluations could be tempered by identifying product-safety and product-instruction labeling comprehension questions that: 1) reliably demonstrate comprehension of target labeling information (i.e., are reliably answered correctly); and 2) are generalizable across research on ENDS and HTPs.

The data provide robust evidence that most of the assessed questions are easily comprehended and are consistently answered correctly across the included labeling comprehension research on HTPs and ENDS. Evidence from six of these domains (‘Know to charge device only with proprietary accessories,’ ‘Know to not expose device to high temperatures,’ ‘Know to not use device if it becomes wet or immersed in a liquid,’ ‘Know to not use device if it appears damaged or broken,’ ‘Know not to use the device with other manufacturer's items,’ and ‘Know that the device heats the tobacco’) indicates that if the domain-specific question were asked again in a future study, a high proportion of participants would be expected to answer the question correctly. These six domains can serve as bench-marks for crafting questions that engender high correct response rates for the evaluated question domains. Five question domains (‘Know to not use device if it is operating in a strange manner,’ ‘Know that the device contains nicotine,’ ‘Know how long the tobacco consumable will last,’ ‘Know not to use the device if the battery behaves unexpectedly,’ and ‘Know that the product contains tobacco’) generally displayed high pooled proportions and narrow CIs (but wide PIs), indicating utility in crafting similar question for new studies, albeit context specific adjustments would help bolster the likelihood of high correct response proportions for these domains. Data for the two remaining domains (‘Know to keep device away from children,’ and ‘Know how many times you can use the tobacco consumable’) indicated wide variability in correct response proportions, thus questions from these domains would likely require significant context-based adaptation in future studies to engender high correct response proportions. Generally, items from domains that exhibited heterogeneity are still useful from an item-benchmarking standpoint, but caution should be exercised when using items from these domains by including modifications that ground the item in a device-specific context. Relatedly, it is worth noting that high correct response rates are not unambiguously positive, and these may at times reflect item simplicity, such that an item/question may be sufficiently intuitive to elicit a correct response but may be less sensitive at detecting nuanced misunderstandings. Indeed, the current research reduces the burdens of labeling comprehension research, but it should not be used as a platform to discourage robustness in the item-crafting step of future labeling comprehension research.

Together, the current findings identified varied labeling comprehension questions that can serve as benchmarks in future labeling comprehension evaluations of ENDS and HTPs to reduce the number of questions that need to be tested, thus shortening study-session time, reducing the cost of the study, and reducing or eliminating the potential need for iterative testing if the question domains of interest are those that were tested here. In a similar vein, it is important to highlight the utility of the user guide content that is tied to each tested question domain in the current study (Table 2). Answering the tested questions correctly is a function of both the structure and content of the questions and the user guide information that describes the answers to the questions. Thus, the content and structure of the user guide information that includes the questions' answers can also be used in future labeling comprehension evaluations of ENDS and HTPs to promote the clarity of information in user guides.

There are broader implications for these data that extend beyond labeling comprehension research on HTPs and ENDS, insofar as many of the questions tested here are applicable to other devices not categorized as ENDS or HTPs. Aside from five question domains (‘Know that the device contains nicotine,’ ‘Know that the product contains tobacco,’ ‘Know that the device heats the tobacco,’ ‘Know how many times you can use the tobacco consumable,’ ‘Know how long the tobacco consumable will last’), all domains included here are applicable to a variety of electronic devices, particularly those that are hand-held and contain a battery, like cell-phones, video gaming consoles, smartwatches, digital cameras, and so on. Accordingly, labeling comprehension research for these product categories may also benefit from including the questions, and user guide content, from the current work in their methodology (as applicable to device features), in terms of reducing the burden of formative testing. Specifically, the research burdens of labeling research, which often accompany other resource intensive research like usability testing, can lead to a “check mark” approach to this type of work – mainly organizations conducting narrowly focused research that is not exhaustive in evaluating comprehension of all product information. By providing researchers with benchmarks for item and user guide content creation, the current work creates capacity to more exhaustively evaluate comprehension of information related to product-specific features across the spectrum of handheld-battery powered devices. The applicability of this work to devices different from HTPs and ENDS also provides an avenue for researchers across product domains to publish their labeling comprehension work, thus increasing the data pools for, and generalizability of, future meta-analytic work.

A different broader implication involves how the current work facilitates future regulatory research. As noted at the outset, some labeling comprehension research is necessary if market authorization for the product of interest is regulated by a third-party agency, like the FDA. When considering HTPs and ENDS – potentially reduced harm products that may support the transition of adults who smoke to smoke-free products – the current research provides tools for facilitating regulatory authorization. At minimum, this work provides a resource-saving tool-kit for labeling comprehension regulatory research that allows researchers more time to explore comprehension of novel labeling information (like feature-specific labeling information), an outcome that is consistent with FDA's PMTA framework that encourages applicants to rely on existing scientific evidence (while) demonstrating consumers' comprehension of labeling (see FDA, p. 55424–55425, 2021 (2)). At most, the current work may help to streamline regulatory submission by allowing researchers to focus on labeling information related to key product features rather than labeling content that has already been evaluated in the literature.

It is worth noting that the list of question domains that was considered in the current research is not exhaustive, in part because they were developed to assess products from specific categories (ENDS and HTPs) from a single company (ALCS); however, it is critical for organizations to consider conducting and publishing similar work on their products to facilitate analyses that could further expand the current list of labeling comprehension questions. The goal of this approach would be to allow for the continued improvement of labeling comprehension methodology through a cross-company and cross-product validation process that would engender an increasing list of metrics that have question reliability and are generalizable across product categories. This process would 1) reduce resource burdens or constraints for organizations conducting product research, 2) improve communication (to consumers) and evaluation of product-safety and product-instruction information, and 3) promote a culture of reciprocal item benchmarking within the product research literature. Labeling comprehension product research in isolation has a limited scope, but cross-company and cross-product benchmarking of items increases item reliability and generalizability and facilitates further meta-analytic work. Labeling comprehension research is rarely published because it is product specific, but this work provides evidence that labeling research can be analytically lever-aged to ease the burdens of new research – In this case, facilitating labeling comprehension research by meta-analytically demonstrating correct response consistency across varied question domains, and providing design resources for researchers who are crafting labeling questions and user guides/safety guides.