“Can AI write my care plans?” - how frontline care managers perceive safety, accuracy and usefulness of LLM-s

Background: Large language models (LLM-s or artificial intelligence or AI) are expected to support chronic care coordination, yet its perceived safety, clinical adequacy, and added value of this technology remains uncertain from the frontline perspective. This study explored how expert care managers evaluated and perceived AI-generated care plans (AI-CPs) for their patients.

 

Methods: Expert care managers chose 11 patient from their practice for whom LLM (ChatGPT-5, Plus subscription level, standardised prompts and iterative creation process) generated care plans using the same inputs as care managers - i.e discharge summaries, national e-health records, prescription data, and the holistic InterRAI assessment results. These same care managers then evaluated these AI-CPs. They rated four dimensions on a 5-point Likert scale: diagnostic relevance, added value, missing key information, and safety. Free-text comments were analyzed qualitatively to identify recurrent themes of trust, usefulness, and risk perception.

 

Results: Most reviewers considered the AI-CPs plans clinically acceptable: 8 of 11 (73 %) judged the listed diagnoses as relevant or mostly relevant, while 3 noted outdated or misplaced entries (“some diagnoses not currently active” or “belongs to patient’s spouse”).

Also the majority (9 of 11, 82 %) described the AI plans as clinically safe or minimally risky, though several highlighted medication inaccuracies or duplications ocurred (“metoprolol dosing unclear”, “duplicate antihypertensive therapy”).

Perceived added value was mixed:

• 4 reviewers (36 %) found that the AI contributed “meaningful or clarifying additions” to the care-plan such as newly documented sleep apnea, allergy information, or more structured follow-up guidance; 5 (45 %) felt the AI added little beyond what was already known; 2 (18 %) reported that some details were factually incorrect or confusing.

Qualitative analysis revealed four dominant themes:

 

1.Structure and clarity – most care-managers found the AI-CP structure as “clear, concise, and easy to read,” particularly for summarizing complex multimorbidity.

2.Contextual limitations – almost all reviewers noted that the AI plan “lacked temporal accuracy” or failed to reflect recent updates such as discontinued therapies.

 

3.Medication safety concerns – recurrent observations included missing active drugs or incorrect combinations; in two cases, this was judged to pose potential clinical risk if left uncorrected.

 

4.Human expertise remains essential – multiple reviewers emphasized that while AI can support synthesis and overview, it “cannot yet replace clinical reasoning or direct patient dialogue.”

Some respondents appreciated that the AI text “captured psychosocial elements better than expected,” while others criticized it as “too general” or “detached from the patient’s story.” Despite occasional factual errors, most evaluators described the process as useful for reflection and saw potential for blended human–AI documentation models.

 

Conclusion: s Care managers viewed AI-generated care plans as generally safe, structured, and occasionally insightful, but not yet clinically reliable without expert verification. Still in our opinion this small pilot illustrates how frontline professionals today perceive AI - not a substitute but a high potential assistive co-author (i.e clinical assistant). Therefore at the current maturity level of off-the-shelf LLM-s, it is critical that human oversight and accountability are maintained.