Harnessing AI for Compassionate Care: Identifying and Supporting Patients with Social Determinants and Medical Complexity

Background: Social determinants of health (SDoH) play an important role in hospital-to-home transitions and outcomes aligned with the Quintuple Aim, yet current point-of-care surveys suffer from low response rates, leaving capture of SDoH incomplete and challenging to use for evaluation and spread. To address these issues, we developed and validated a large language model (LLM) that captures SDoH from electronic health records (EHR) at University Health Network in Toronto, Canada.

 

Approach: Following a literature review and discussion with partners, including patients and caregivers, a specialized keyword list relevant to hospital-to-home and patient outcomes was developed. In the first phase, relevant SDoH included the following: language barrier, financial strain, social isolation, housing insecurity, depression, addiction, food insecurity, transportation barriers, and health literacy.

 

This list was used to search the EHR and to build a labeled keyword set. The LLM (DeepSeek-32B) was applied to inpatient notes from patients discharged between August 2021 and March 2024 who were eligible for an integrated care pathway. For each run, the LLM produced a classification (yes/no) indicating whether that social need is present, and when present, a supporting span of text from the notes with a brief reasoning explaining the decision was produced.

 

Human reviewers subsequently reviewed the model’s output, creating confusion matrices for each SDoH reporting precision (positive predictive value), recall (sensitivity), F1 (a score that balances both precision and recall), and prevalence for every category. In Phase 2, the model is being applied to a larger cohort with an expected higher prevalence of patients with social complexity. Our team is performing a targeted chart review and updating prompts based on the results to improve accuracy and reduce false negatives.

 

Results: In Phase 1, we processed 67,000 notes from 2890 inpatients, flagging 1,139 patients (30%) with at least one SDoH. The LLM doubled the capture of social needs (prevalence) from 4% to 8%, with 98% accuracy. Precision, recall, and F1-scores averaged 76%, 97% and 84% respectively, the latter of which is close to human agreement on the same task (80%).

 

Issues identified included:

 1) the need to refine the LLM to improve accuracy for select SDoH like food insecurity,

 2) adding further distinctions between active and historical social needs like depression and addiction,

 3) consideration for human reinforcement so that information irrelevant to current care planning can be incorporated directly feedback loops.

 

 Phase 2 (currently underway) extends the evaluation and refinement to 232,253 notes among 7,000 patients, demonstrating a current increase in capture to14% with social needs. Ongoing error analyses inform documentation and prompt engineering and design.

 

Implications: LLMs can rapidly identify, extract, and summarize SDoH data from a large volume of unstructured interdisciplinary notes in EHRs, offering a practical way to capture SDoH for hospital-to-home integrated care programs. We are validating the model with our Calgary partners to demonstrate feasibility across different hospital settings and documentation practices. This demonstrates scalability and transferability to international health systems addressing similar challenges in SDoH capture and care coordination.