Identifying Reliable Labeling Comprehension Questions for E-Vapor and Heated Tobacco Products: A Meta-Analytic Approach
Abstract
Background
Labeling comprehension research can be tedious and resource-constraining because it often involves iterative reliability assessments of relevant metrics (e.g., formative testing of any safety or product-use related questions), leading to increased time requirements and shrinking sample pools in the population of interest. The goal of this work was to temper the need for formative metric evaluations in labeling comprehension research by identifying labeling comprehension questions that: 1) can be reliably answered correctly and 2) are generalizable across research on electronic nicotine delivery system (ENDS) products and heated tobacco products (HTPs). We explored whether, across four labeling comprehension studies for three distinct nicotine products (one ENDS, two HTPs), we could meta-analytically identify labeling comprehension questions that consistently demonstrated high correct response rates (relative to a 100% correct response rate).
Methods
Forty-five relevant questions were identified across studies and were categorized into 13 groups/domains for analysis purposes. Correct response rates for questions within a group/domain were analyzed using a proportional meta-analysis. A proportional meta-analysis is a data-synthesis method that allows for the calculation of a pooled overall proportion from multiple individual proportions (1).
Results
Evidence from six of the tested domains (‘Know to charge device only with proprietary accessories,’ ‘Know to not expose device to high temperatures,’ ‘Know to not use device if it becomes wet or immersed in a liquid,’ ‘Know to not use device if it appears damaged or broken,’ and ‘Know not to use the device with other manufacturer's items’) indicates that if the domain-specific question were asked again in a future study, a high proportion of participants would be expected to answer the question correctly. These six domains can serve as benchmarks for crafting questions that engender high correct response rates for the evaluated question domains. Five question domains (‘Know to not use device if it is operating in a strange manner,’ ‘Know that the device contains nicotine,’ ‘Know how long the tobacco consumable will last,’ ‘Know not to use the device if the battery behaves unexpectedly,’ and ‘Know that the product contains tobacco’) generally displayed high pooled proportions and narrow confidence intervals (but wide prediction intervals), indicating robust utility in crafting similar question for new studies, albeit context specific adjustments would help bolster the likelihood of high correct response proportions. Data for the two remaining domains (‘Know to keep device away from children,’ and ‘Know how many times you can use the tobacco consumable’) indicated wide variability in correct response proportions, thus questions from these domains would likely require significant context-based adaptation in future studies to engender high correct response proportions.
Conclusions
This work provides evidence that labeling research can be analytically leveraged to ease the burdens of new research; in this case, facilitating labeling comprehension research by meta-analytically demonstrating correct response consistency across varied question domains, thus providing design resources for researchers who are crafting labeling questions and user guides/safety guides.
© 2026 Jonathan M. Gallegos, Jennifer N. Lewis, Elizabeth Becker, published by Beiträge zur Tabakforschung GmbH
This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License.