Best Practices and Guidelines with Respect to Psychometric Consumer Reported Outcome Measures for Use in Research on Tobacco- and Nicotine-Containing Products – A Consensus-Based Approach

What is the concept to be measured, and how is it defined? What are the components/aspects of the construct that should be represented in the CROM? How broad should the scope of the CROM conceptually be (particularly relevant for multi-faceted constructs, such as quality of life)?

Is the construct likely stable (a trait) or unstable (a state) over time?

Especially for constructs that vary over time, is a specific timeframe required (e.g., urge to smoke “over the past 7 days” vs. “right now”) as part of the construct’s definition?

Is the researcher looking for a single total score, or separate scores to reflect the different aspects of the construct?

What sort of guidelines for score interpretation are desired?

What target population do the participants in the study represent (e.g., adults who smoke cigarettes)? An ideal CROM would have psychometric evidence supporting its use with participants representing the target population. Will the CROM need to be appropriate for any subpopulation(s) of interest (e.g., those with limited health literacy, youth)?

What is the study type and study design (e.g., cross-sectional vs. longitudinal; if longitudinal, over what duration)?

Will the CROM be applied to different products (candidate and comparator products)? Is the intent to make quantitative comparisons between products?

Does the study require electronic administration? If so, on devices with different screen sizes?

Does the CROM require electronic administration/scoring (e.g., computer adaptive testing, display of digital images, skip logic)?

Does administration of the CROM involve conditions that might affect participant responses (e.g., responding to CROM in front of study staff)?

Does the CROM need to be self-administered or interviewer-administered?

If administered repeatedly, how frequently will the CROM be completed? Is assessing change over time part of the objective? Does high frequency of administrations impact on data quality (lack of engagement and attention resulting in straightlining, increased frequency of missing responses, etc.)?

Are there study restrictions regarding the length (time required for administration) of the CROM?

In which countries / languages will the CROM be administered?

Licensing fees, permission to use, copyright clearance

Changing instructions and/or item content to reference a different product category (e.g., “electronic nicotine delivery systems (ENDS)” instead of “cigarettes”, updating language/terminology)

Adding item(s)

Removing item(s)/only administering a subset of items

Adding images to items to improve clarity/comprehension

Changing the recall period (e.g., “in the past 30 days” to “in the past 7 days”)

Removing or introducing a response option of “I don’t know”

Adding response labels so that a scale is fully labeled

Changing the number of response categories

Changing response category labels

Administering a CROM developed for paper-and-pencil electronically

Changing the method of administration from self-report to interviewer-administered

Modifying a CROM to fit a small screen device (smartphone) by administering one item per screen instead of the items together as a grid

Changing a rating task (asking the participant to respond to each item by selecting a value on a numerical rating scale) to a drag-and-drop task

Changing the order of item administration (fixed order vs. randomized)

A measure of smoking susceptibility developed to assess susceptibility among youth is used to assess smoking susceptibility in adults

A measure of cigarette dependence developed for use with adults who smoke cigarettes is administered to people who use ENDS to assess dependence on ENDS

Translating a CROM into a different language and administering it to a new population (i.e., individuals whose primary language differs from languages in which the CROM has been validated)

Administering a CROM to individuals from another culture (i.e., individuals whose cultural background differs in a relevant way from the background of individuals for whom the CROM was originally validated)

Making the text bold and underlining the recall period in the instructions (“In the past 7 days”) for visibility and emphasis

Changing font size or font style

Adding additional clarifying language to an item or instruction without changing the substance

Adding an image of the product being referenced

Administering a paper-and-pencil CROM electronically, without changing the presentation of the CROM

Administering items forming a single dimension from a multi-dimensional CROM ^b

Administering a subset of items from a unidimensional CROM (developing a “short form”)

Changing the type of response task (e.g., a numerical rating task is changed to a drag-and-drop task)

Changing the type of response scale (e.g., from a 5-category fully labeled scale to a visual analog scale, from 5-point descriptive response scale to 11-point numerical rating scale)

Changing the content of the response scale (e.g., replacing a frequency scale with an intensity scale)

Adding items to a CROM

Administering the CROM to a population for which it was not developed (e.g., a measure of cigarette dependence developed for use with adults who smoke cigarettes is administered to adolescents)

Applying the CROM to TNPs for which it was not developed (e.g., a measure of cigarette dependence is administered to individuals who use ENDS to assess dependence on ENDS)

Translating a CROM into a new language and administering it to individuals who speak this new language

Generally, no evidence is needed

In certain circumstances, qualitative evidence may be helpful (e.g., to ensure that new clarifying language added to instructions is clear)

Usability testing may be helpful when modifying a paper-and-pencil CROM for electronic administration

Qualitative and/or quantitative evidence is always recommended

Quantitative evidence is recommended to support development of a short form ^c

If CROM content is substantially changed (e.g., changing response task or response scale, adding new items), either (or both) qualitative and quantitative evidence could be used to support the modification ^d

If scores from two versions of a CROM are being directly compared in a study (e.g., ENDS dependence vs. cigarette dependence), quantitative evidence is recommended

Quantitative and/or qualitative evidence is needed when administering a CROM to a new population (e.g., youth vs. adults) or product

Qualitative and in some cases quantitative evidence is recommended when translating a CROM into a new language