Table 1
Description of topics predictive of pre-clerkship performance at the Feinberg School of Medicine.
| TOPIC NAME | DESCRIPTION | EXAMPLE WORDS AND TERMS | HAVING MORE OF THESE TERMS INCREASES THE LIKELIHOOD OF A STUDENT BEING: |
|---|---|---|---|
| Change | Phrases that indicate the student may need to change a behavior | “I would encourage,” “make sure,” “I suggest,” “be more” | Not Yet Ready |
| Common Negative Terms | Frequently appearing words | although, but, little, more, not | Not Yet Ready |
| Frequency Words | Words that indicate the frequency with which something occurred | few, generally, instance, many times, mostly, multiple times, occasionally, occasions, once, rarely, seldom, several times, sometimes, tend, tendency, usually | Not Yet Ready |
| Hedging | Words that soften a comment or phrase, or add uncertainty to a comment | at least, fairly, less, little bit, maybe, might, much, nearly, perhaps, possibly, rather, seem, slightly, sometimes seemed, somewhat, sort of, while | Not Yet Ready |
| Late/Absent | Words and phrases suggesting a potential issue with student attendance or timeliness | absence, absences, absent, arrive on time, attendance, be on time, late | Not Yet Ready |
| Negative Adjectives | Adjectives associated with negative or problematic behaviors | argumentative, arrogant, cautious, detached, detrimental, disengaged, disinterested, disorganized, flustered, irrelevant, rambling, reticent, superficial, underprepared, vague, withdrawn | Not Yet Ready |
| Negative Standardized Patient Feelings | Negative feeling words appearing in feedback from standardized patients | confused, disconnected, fearful, frustrated, nervous, overwhelmed, uncomfortable, uneasy, unsure, upset | Not Yet Ready |
| Negative Terms | Words and phrases associated with negative or problematic behaviors | allow others, ask more questions, error, excuses, fail, forget, impression, interrupt, looking down, more consistent, more detail, nervous energy, not paying attention, omit, phone, reminder, repetition, skip, struggle, suffer, surprise, trouble, unable, unfortunately, wrong | Not Yet Ready |
| Speak-up | Phrases that indicate the student needs to participate more in group conversations or speak up | speak up, participate more, hear more, contribute more, more vocal | Not Yet Ready |
| Common Positive Terms | Frequently appearing, positive words | active, clear, clearly, effective, excellent, great, helpful, leader, leadership, professional, respectful, team member, thoughtful, well organized | Ready |
| Positive Adjectives | Positive general adjectives (not specific skills or attributes) | amazing, awesome, brightest, exceptional, exemplary, extraordinary, fantastic, highly, impressive, incredible, indispensable, instrumental, integral, outstanding, remarkable, star, superb, superior, superlative, talented, terrific, tremendous, unparalleled, wonderful | Ready |
| Positive Adverbs | Positive adverbs | absolutely, actively, consistently, effectively, effortlessly, excellently, extremely, extremely well, incredibly, professionally, wonderfully | Ready |
| Positive Attributes | Positive personal attributes | asset, calm, caring, cheerful, clearly prepared, compassionate, consistently prepared, eloquent, empathic, engaging, excellent communicator, friendly, great attitude, hardworking, incredibly helpful, insightful, kind, proactive, reassuring, valuable, very effective, warm, welcoming | Ready |
| Positive Presentation | Words and phrases suggesting good presentation skills | concise presentations, effective presentations, engaging presentation, excellent presentation, interactive, organized presentations, strong presentation, useful information, well presented | Ready |
| Positive Skills | Words and phrases indicating desired behaviors and skills | bedside, calm demeanor, clarity, clear understanding, consistently demonstrated, create, decision making, encouraged others, enhance, excellent communication, excellent eye contact, excellent rapport, facilitate, good explanation, good eye contact, good questions, good technique, great teacher, implement, insightful question, often volunteered, open ended questions, probing questions, same page, skilled, took initiative, utilize, well articulated, well written | Ready |
| Positive Teamwork | Words and phrases suggesting good teamwork skills | active member, awesome group, enthusiastic member, excellent group, excellent team, great teammate, group focused, integral member, strong member, team dynamic, teamwork, teamwork skills | Ready |
Table 2
Use cases of example students comparing model predictions of the test dataset to actual review outcomes.
| STUDENT “A” | STUDENT “B” | STUDENT “C” | |
|---|---|---|---|
| Predicted outcome | Overall score of 0.001, indicating this student is highly likely to be found “Ready.” | Overall score of 0.27, indicating this student may be at risk of being found “Not Yet Ready.” | Overall score of 0.88, indicating this student is highly likely to be found “Not Yet Ready.” |
| Characteristics of narrative feedback impacting the prediction | Relatively higher number of words and terms in the “Positive Attributes,” “Positive Adjectives,” and “Positive Skills” topics. Relatively lower number of words and terms in the “Change,” “Hedging,” and “Late/Absent” topics. | Relatively higher number of words and terms in the “Speak Up,” “Hedging,” and “Negative Adjectives” topics. Relatively higher than average number of words and terms in the “Positive attributes,” and “Positive Presentation” topics. Relatively lower number of words and terms in “Positive Teamwork” and “Positive Skills” topics. | Relatively higher number of words and terms in the “Late/Absent,” “Change,” and “Frequency Words” topics. Relatively lower number of words and terms in the “Positive Teamwork,” “Positive Attributes,” and “Positive Adverbs” topics. |
| Actual competency committee review outcome | Student is considered “Ready” across all competency domains. Specifically exceeds expectations in Patient Care domain. | Student is considered “Not Ready” in the Teamwork domain. Student is directed to meet with faculty mentor to improve ability to contribute to group discussions. | Student is considered “Not Ready” in the Professionalism and Teamwork domains. Student is directed to work with an educational support team to grow effective leadership skills, and to work with a faculty mentor to improve accountability. |