Assessing the Consistency of Reporting for E-Cigarette Brand, Device Type, and Flavor in the National Youth Tobacco Survey: Implications for Regulatory Decision-Making

In November 2023, the U.S. Food and Drug Administration (FDA) and the Centers for Disease Control and Prevention (CDC) reported annual estimates of youth tobacco use (1). The data source for these estimates was the 2023 National Youth Tobacco Survey (NYTS), which is a cross-sectional, school-based, self-administered online survey of U.S. students in middle school (grades 6–8) and high school (grades 9–12) (2). Estimates from the 2023 NYTS suggested that 10.0% (2.80 million) of middle and high school students reported current use of one or more tobacco products, with current use defined as any past-30-day use, even one time. The most common product use categories reported by youth were e-cigarettes (7.7%; 2.13 million), followed by cigarettes (1.6%; 0.43 million), cigars (1.6%; 0.42 million), nicotine pouches (1.5%; 0.40 million), smokeless tobacco (1.2%; 0.33 million), hookahs (1.1%; 0.29 million), heated tobacco products (1.0%; 0.26 million) ⁽¹⁾ , and pipe tobacco (0.5%; 0.13 million).

The FDA and CDC further reported that 4.6% of middle school students (0.55 million) and 10.0% of high school students (1.56 million) reported any past-30-day e-cigarette use. The device types reported to be used most often by past-30-day e-cigarette users were disposable devices (60.7%; 1.24 million), followed by e-cigarettes that use prefilled or refillable pods or cartridges (16.1%; 0.33 million) and tanks or mod systems (5.9%; 0.12 million). The most commonly reported usual brands of e-cigarettes used by past-30-day e-cigarette users were Elf Bar (31.1%; 0.63 million), Vuse (8.7%; 0.17 million), Esco Bars (6.0%; 0.12 million), and JUUL (3.4%; 0.07 million). Importantly, 13.2% (0.27 million) of middle and high school students reported “Some other brand(s) not listed here” and 19.8% (0.40 million) reported “Not sure/I don't know the brand”. E-cigarettes with flavors other than ‘tobacco’ or ‘unflavored’ were used by 89.4% of past-30-day e-cigarette users (1.90 million), and the most commonly reported flavors used were ‘fruit’ (63.4%; 1.28 million), ‘candy, desserts, or other sweets’ (35.0%; 0.70 million), ‘mint’ (27.8%; 0.56 million), and ‘menthol’ (20.1%; 0.40 million) (1).

The Family Smoking Prevention and Tobacco Control Act (TCA) gave authority to regulate tobacco products to the FDA. In February 2020, the FDA exercised its enforcement discretion against flavored, cartridge-based e-cigarette products (other than ‘tobacco’ and ‘menthol’) (3). This action created a loophole whereby products with flavors characterized as ‘fruit’, ‘candy, desserts, or other sweets’, and ‘mint’ in cartridge-based e-cigarettes were to be removed from the market, but these flavors remained available in disposable e-cigarettes and refillable tank systems.

The regulatory authority given to the FDA over tobacco products requires that manufacturers who wish to market a new tobacco product must apply for and obtain marketing authorization. Furthermore, the FDA requires that manufacturers applying for marketing authorization of new tobacco products, such as e-cigarettes, provide product-specific data on the use of those products, including data specific to each flavor. Manufacturers must also provide evidence of an increased benefit (e.g., reduced cigarette consumption among adult smokers) for flavored e-cigarettes that exceeds the benefit shown for tobacco-flavored e-cigarettes, and these benefits must more than offset any potential harms that may be associated with the potential for youth use of the product.

The FDA's assessment of the potential risks associated with e-cigarette use among youth has been based on data from the NYTS, and the FDA specifically cites the NYTS data when it has publicly announced its decisions denying marketing authorization of e-cigarette products (4,5,6); the FDA has also cited these data as supportive of low levels of youth use when authorizing a menthol-flavored e-cigarette (7). Thus, from a regulatory perspective, estimates from the NYTS on e-cigarette use among youth are the measure of population risk that products must offset in order to receive marketing authorization. Therefore, accurate estimates of youth use of e-cigarettes by brand should be used to guide regulatory decision-making.

This study sought to build upon findings from a previous study (8) that found inconstancies in reported e-cigarette brands and device types among youth using a combined analysis of the 2021 and 2022 NYTS. That study found that one-third of youth who reported using a pod-based e-cigarette brand and approximately 20.0% of youth who reported using a disposable e-cigarette brand reported device types that were not consistent with their reported brand used. Reporting inconsistencies regarding current e-cigarette use do not appear to be limited to youth, as similar analyses of data from the adult survey of the Population Assessment of Tobacco and Health (PATH) study also showed discrepancies between reported brand used and device type used (9). Given the significant regulatory implications of youth e-cigarette use, the current analyses extend the findings from a previous study (8) by examining the consistency of responses between reported brand used and device type used, in addition to reported flavors used, based on data from both the 2023 and 2022 NYTS. Specifically, the analyses examined past-30-day e-cigarette users' reported only brand or usual brand used and their reported device type used most often and all flavors used during the past 30 days. The brands examined in the current analyses include Vuse, JUUL, Elf Bar (2023), and Puff Bar (2022), which represent two pod-based devices (Vuse and JUUL) and two disposable devices (Elf Bar and Puff Bar) that are reported by sufficient proportions of youth relative to the other response options available in the NYTS.

Data from the nationally representative NYTS were collected in March through June of 2023 (2) and in January through May of 2022 (10). The 2023 NYTS was completed by 22,069 students from 179 participating schools (1), and the 2022 NYTS was completed by 28,291 students from 341 participating schools (11).

Questions on e-cigarette use, including those used to identify only or usual brand used, device type used most often, and flavors used during the past 30 days, differed slightly between the two years of the NYTS. Relevant to the current analyses, the e-cigarette brands and flavors that were presented to the respondents varied slightly (e.g., Elf Bar was only shown in the 2023 NYTS and Puff Bar was only shown in the 2022 NYTS).

For the current analyses, data from the 2023 and 2022 NYTS were analyzed separately using survey year specific weighting (2, 10), and all analyses were completed using SAS software version 9.4 (12).

For each year of the NYTS, the sample for the analyses included respondents who were past-30-day only brand or usual brand users of Vuse, JUUL, Elf Bar (2023), or Puff Bar (2022) e-cigarettes. These respondents reported ever using an e-cigarette, using an e-cigarette at least one day in the past 30 days, and selected one of these four brands as the only brand used or usual brand used.

The brand of e-cigarette used was based on the question “During the past 30 days, what e-cigarette brands did you use? (Select one or more).” Response options included “Vuse”, “JUUL”, “Elf Bar” (2023 NYTS only), and “Puff Bar” (2022 NYTS only), in addition to other brands. ⁽²⁾ If only one brand was selected, that brand was considered a respondent's only brand used. Those who selected more than one brand were then asked “During the past 30 days, what brand of e-cigarettes did you usually use? (Choose only one answer)”. If Vuse, JUUL, Elf Bar, or Puff Bar was selected as the only brand used in the first question or the brand usually used in the subsequent question, that brand was considered the respondent's usual brand. ⁽³⁾ This designation of usual brand is the same as that reported by CHEN-SANKEY et al. (13) and GENTZKE et al. (14). Write-in responses were allowed and were included in these analyses if respondents wrote in responses that indicated use of Vuse, JUUL, Elf Bar, or Puff Bar. ⁽⁴⁾

Findings from analyses of the 2023 NYTS data examining the consistency between past-30-day e-cigarette users' reported only brand or usual brand used - specifically, Vuse, JUUL, or Elf Bar - and the device type used most often and flavors used during the past 30 days are presented in Table 1. Among all middle and high school students (N = 21,718, excluding 351 with missing data), 333 respondents (1.5% of all youth [all proportions weighted]) reported any past-30-day use of Vuse e-cigarettes, 237 respondents (1.2% of all youth) reported any past-30-day use of JUUL e-cigarettes, and 839 respondents (4.2% of all youth) reported any past-30-day use of Elf Bar e-cigarettes. Focusing on middle and high school students' only brand or usual brand used in the past-30-days (Table 1), 134 respondents (0.6% of all youth) reported using Vuse as their only or usual brand, 42 respondents (0.3% of all youth) reported using JUUL as their only or usual brand, and 478 respondents (2.3% of all youth) reported using Elf Bar as their only or usual brand.

Assessing the consistency of responses between brand and device type among past-30-day Vuse e-cigarette users (only brand or usual brand users, n = 134) shows that 96 respondents (58.4% of Vuse brand users; 0.4% of all youth) reported a device type consistent with that brand (i.e., an e-cigarette that uses pre-filled or refillable pods or cartridges) (Table 1). Among those reporting past-30-day use of JUUL e-cigarettes (only brand or usual brand users, n = 42), 21 respondents (57.1% of JUUL users; 0.1% of all youth) reported a device type consistent with that brand (i.e., an e-cigarette that uses pre-filled or refillable pods or cartridges). Among those reporting past-30-day use of Elf Bar (only brand or usual brand users, n = 478), 392 respondents (85.4% of Elf Bar users; 2.0% of all youth) reported a device type consistent with that brand (i.e., a disposable e-cigarette).

Consistency of responses between past-30-day e-cigarette users' reported only brand or usual brand used and all flavors used during that period was assessed among respondents who provided consistent responses for their device type. Table 1 shows that 84 respondents who reported past-30-day use of Vuse e-cigarettes (as their only brand or usual brand) and provided a consistent response for the device type they used most often also reported flavor(s) used in the past 30 days that were consistent with the use of Vuse e-cigarettes (i.e., ‘tobacco-flavor’ and/or ‘menthol’). Thus, 45.6% of only brand and usual brand past-30-day Vuse e-cigarette users provided consistent responses for both device type and flavors used, representing 0.3% of all youth.

For respondents who reported past-30-day use of JUUL e-cigarettes (as their only brand or usual brand) and a consistent response for device type, 3 respondents (1.4% of only one brand and ususal brand JUUL users; 0.1% of all youth) also reported flavor(s) used in the past 30 days that were consistent with the use of JUUL e-cigarettes (i.e., ‘tobacco-flavor’ and/or ‘menthol’) (Table 1). Flavor analyses were not conducted for past-30-day users of Elf Bar e-cigarettes due to the considerable number of flavors available for this brand and the lack of certainty regarding which flavors were available during the survey periods.

The analytic approach used to assess consistency of responses among past-30-day Vuse e-cigarette users (only brand or usual brand users) regarding device type used most often and flavors used during the past 30 days, as well as an accounting of all respondents is shown in Figures 1a and 1b. The same approach was used for past-30-day JUUL e-cigarette users (only brand or usual brand users), and a modified approach was used for past-30-day Elf Bar e-cigarette users. See Supplemental Material for JUUL [Figures S1a and S1b] and Elf Bar [Figures S2a and S2b] e-cigarette users. All middle and high school students who participated in the 2023 NYTS (N = 22,069) were accounted for in these analyses.

Figure 1a.

Analytic approach for assessing consistency of reporting for past-30-day Vuse e-cigarette users: 2023 NYTS.

Figure 1b.

Analytic approach for assessing consistency of reporting for past-30-day Vuse e-cigarette users: 2023 NYTS.

In the 2022 NYTS, 627 respondents reported any past-30-day use of Vuse e-cigarettes (2.2% of all youth), with 342 respondents (1.1% of all youth) reporting Vuse e-cigarettes as their only or usual brand. For these 342 only or usual brand users, 177 respondents (0.5% of all youth) provided responses for device type used most often and flavors used during the past 30 days that were consistent with the use of Vuse e-cigarettes. A total of 542 respondents (2.0% of all youth) reported any past-30-day use of JUUL e-cigarettes, with 130 respondents (0.5% of all youth) reporting JUUL e-cigarettes as their only or usual brand. For these 130 only or usual brand users, 23 respondents (0.2% of all youth) provided responses for device type used most often and flavors used during the past 30 days that were consistent with the use of JUUL e-cigarettes. Lastly, 874 respondents (2.8% of all youth) reported any past-30-day use of Puff Bar e-cigarettes, with 444 respondents (1.3% of all youth) reporting Puff Bar e-cigarettes as their only or usual brand. For the 444 only or usual brand users, 354 respondents (1.1% of all youth) provided responses for device type used most often that were consistent with the use of Puff Bar e-cigarettes (flavors could not be validly assessed). See Supplemental Material (Table S1, Figures S3a, S3b, S4a, S4b, S5a, and S5b) for results based on the 2022 NYTS data.

Evidence from the 2023 NYTS clearly shows that merely presenting youth self-reported data on past-30-day e-cigarette use by only brand and usual brand without further examining whether those data are consistent with other reported data, such as device type used most often and flavors used during the past 30 days, substantially overestimates youth e-cigarette brand prevalence. Data from the 2023 NYTS suggest that 1.5% of all youth (426,682 youth) reported any past-30-day use of Vuse e-cigarettes and that 0.6% of all youth (178,544 youth) reported using Vuse e-cigarettes as their only brand or usual brand; these weighted estimates are consistent with the weighted (and rounded) estimates of 420,000 and 170,000 youth, respectively, reported by the FDA and CDC (1). Applying a straightforward data quality assessment - in this case, assessing the consistency of responses for device type used most often and flavors used during the past 30 days with the reported only brand or usual brand - reduces the estimate of Vuse e-cigarette users to 0.3% of all youth (81,400 youth). For analyses that assessed consistency based only on flavors used during the past 30 days, the estimate of Vuse e-cigarette users is reduced to 0.5% of all youth (142,669 youth). These estimates are far lower than those reported by the FDA and CDC. It is also noteworthy that 41.6% of only brand or usual brand users of Vuse e-cigarettes did not provide responses for the device type they used most often that were consistent with the use of a Vuse e-cigarette, despite Vuse being listed as an example of a pre-filled or refillable pod or cartridge e-cigarette in the 2023 NYTS questionnaire.

The reporting of JUUL e-cigarette use among youth showed an even larger discrepancy, with 1.2% of all youth (339,495 youth; similar to the 330,000 reported by the FDA and CDC (1)) reporting any past-30-day use of JUUL e-cigarettes and 0.3% of all youth (70,127 youth) reporting JUUL e-cigarettes as their only brand or usual brand. Again, applying a straightforward data quality assessment shows that only < 0.1% of all youth (1,007 youth) who reported JUUL e-cigarettes as their only brand or usual brand provided responses for device type used most often and flavors used during the past 30 days that were consistent with the use of JUUL e-cigarettes. Nearly 43.0% of only brand or usual brand users of JUUL e-cigarettes did not provide consistent responses for the device type they used most often, despite JUUL being listed as an example of a pre-filled or refillable pod or cartridge e-cigarette in the NYTS questionnaire. Similarly, 14.6% of only brand or usual brand Elf Bar e-cigarette users did not provide consistent responses for device type used most often, despite Elf Bar being listed as an example of a disposable e-cigarette in the NYTS questionnaire.

Parallel analyses of data from the 2022 NYTS show that the low levels of consistency in reporting by youth was not limited to the 2023 NYTS. Findings from analyses of the 2022 NYTS data show that over half (55.7%) of those who reported Vuse e-cigarettes as their only brand or usual brand and more than two-thirds (67.2%) of those reporting JUUL e-cigarettes as their only brand or usual brand provided responses for device type used most often and flavors used during the past 30 days that were not consistent with the use of those respective brands. Thus, the evidence suggests that inconsistent reporting of tobacco use behaviors by youth - at a minimum, those youth participating in the NYTS - is common.

These results expand upon those from a recent study that reported inconsistencies between self-reported brand and device type among youth e-cigarette users in the 2022 and 2021 NYTS (8). A similar analysis among adults using data from the PATH study that examined the consistency between self-reported e-cigarette device type and actual device type also found inconsistencies (9), suggesting that inconsistent reporting of e-cigarette brand and device type used is not unique to youth or to the NYTS.

A potential limitation of the current analyses is that they cannot assess consistency of responses among youth who reported any past-30-day use of Vuse, JUUL, Elf Bar, or Puff Bar e-cigarettes but did not report one of those brands as their only brand or usual brand. In addition, the relative standard error for some of the estimates provided in the current analyses exceeded 30%, which is the threshold used to evaluate their reliability. Nonetheless, the purpose of the analyses was to evaluate the consistency of reporting current e-cigarette use among youth, and this limitation further highlights the need for careful consideration of these data.

Beyond the practical limitations of assessing consistency of responses for these any past-30-day users (due to limitations associated with the questions asked in the NYTS), a more consequential consideration is technical accuracy in data reporting. Respondents in the NYTS who reported any past-30-day use of e-cigarettes were asked to report all brands of e-cigarettes used during that period. For those who reported using only one brand, that brand was considered to be the respondents' usual (only) brand. Those who selected more than one brand were additionally asked to report the one brand that they used most often during the past 30 days. This means that every respondent who reported using e-cigarettes in the past 30 days was required to report a usual brand of e-cigarette (only brand or usual brand used), which allows each respondent to be counted once in the population-based estimates.

Given that an important outcome measure from these analyses was weighted estimates (proportions) for all youth, analyses that include respondents who report any past-30-day use of a product but do not report only or usual use of that product artificially increases the total number of youth reporting past-30-day use of e-cigarettes. For example, estimates provided by the FDA and CDC (1) suggested that an estimated 2.13 million youth reported using e-cigarettes during the past 30 days in 2023, but the estimated number of users based on use by brand was more than twice that number, or 4.87 million youth, due to a considerable number of youth who reported using more than one e-cigarette brand. Thus, for technical accuracy, each respondent should only be counted once when reporting estimates of e-cigarette brand prevalence (including by the FDA) and when using such estimates during regulatory decision-making.

Over-estimating the number of youth who report using a particular e-cigarette brand has significant regulatory implications, including for the FDA's pre-market review of e-cigarette applications. The TCA requires that the FDA assess new tobacco products, including e-cigarettes, based on risks and benefits to the population as a whole. Currently, the FDA requires that product-specific data be included in an application requesting marketing authorization for an e-cigarette, including data for each flavor of an e-cigarette, to demonstrate a benefit (i.e., reduced cigarette consumption among adult smokers) that more than offsets any potential harms that may occur with youth e-cigarette use. The FDA also requires evidence of an increased population benefit for non-tobacco flavored e-cigarettes above the benefit shown for tobacco-flavored e-cigarettes.

To date, the FDA's assessment of the risks likely to be associated with youth use of e-cigarettes has been based on brand-specific and, in some instances, category-specific estimates of flavored e-cigarette use from the NYTS. From a regulatory perspective, these estimates are used by the FDA as the threshold of risk that must be more than offset by a product(s) in order to receive marketing authorization. Thus, this regulatory threshold must be based on complete and accurate analyses of data on youth e-cigarette brand use, which have not been conducted in recent published reports (1, 11, 15).

The FDA is required by the TCA to assess the risks and benefits of e-cigarette use at the population level and, for that reason, has requested that those seeking marketing authorization provide population-level data to assist with its review. It is noteworthy that for FDA's most recent denials of marketing authorization for e-cigarettes, the agency has specifically mentioned the NYTS data in its press releases (4,5,6), citing to rates of any past 30-day use of the products among youth e-cigarette users (not the population rates among all youth). It is unclear why the FDA does not conduct analyses similar to those presented here, to adjust its estimates based on consistency of reporting.

Even without adjusting prevalence estimates for consistency in reporting, self-reported past-30-day use by youth is low for Vuse (1.5% of all youth, or 0.43 million) and JUUL (1.2% of all youth, or 0.34 million) e-cigarettes. The analyses presented herein clearly show that, after adjusting these prevalence estimates based on respondents who provided responses for device type used most often and flavors used during the past 30 days that were consistent with the use of those e-cigarettes, the population-level use estimates are even lower for Vuse (0.3% of all youth, or 0.08 million) and JUUL (< 0.1% of all youth, or 0.001 million) e-cigarettes.

These estimates of e-cigarette brand prevalence at the population level, as a proportion of all youth, raises an important question regarding the level of youth e-cigarette use that would be sufficiently low during the FDA's risk assessment to permit the marketing authorization of an e-cigarette product. Perhaps the answer to that question is self-evident, as the FDA has denied marketing authorization for the vast majority of e-cigarette product applications it has reviewed, granting marketing authorization for just 34 e-cigarette products and devices (16) and only one non-tobacco flavored product.

Given the clear weight that the FDA places on brand-specific prevalence estimates in its assessment of whether to grant marketing authorization for an e-cigarette product, and the high level of inconsistent reporting of youth e-cigarette use in the NYTS data, the structure of the NYTS should be modified so that each identified brand is assessed separately for device type and flavors associated with that specific brand. Additionally, as suggested by other researchers (8, 9), the inclusion of images of e-cigarette products in the NYTS would serve to improve reporting accuracy. In the absence of changes to the NYTS questionnaire, the FDA's sole reliance on data from that survey for prevalence estimates of youth e-cigarette use during its regulatory decision-making cannot be justified.

These analyses of the 2022 and 2023 NYTS data demonstrate that a sizable proportion of youth who reported an e-cigarette brand as their only or usual brand used provided information on the device type or flavors used that were not consistent with the use of that brand of e-cigarette. Estimating the population prevalence of use based on only those youth reporting consistent data demonstrates that very small proportions provided responses that suggest they used the brand reported: Vuse (2023: 0.3%; 2022: 0.5%), JUUL (2023: < 0.1%; 2022: 0.2%), Elf Bar (2023: 2.0%), and Puff Bar (2022: 1.1%). The implications for this lack of consistency reach into the regulatory decision-making process in the U.S., as the FDA employs these data as part of its assessment of the population-level risk of e-cigarettes. A more rigorous assessment of these survey data is warranted for regulatory decisions.