Dear Editor,
ChatGPT is recognized as a promising tool in healthcare by providing continuous support, quick accessibility, and personalized guidance for weight management [1]. In the field of bariatrics, the potential of ChatGPT in post-surgery weight management and nutrition is particularly noteworthy. In a study, the accuracy of the model’s responses to bariatric questions was confirmed; however, the reliability of its responses to nutrition-specific issues has not yet been examined [2].
In this study, the aim is to evaluate the nutritional recommendations provided by the ChatGPT language model for patients following bariatric surgery through expert dietitians.
This study was designed as a cross-sectional study. The responses provided by the free version of the ChatGPT 3.5 were evaluated by three experts in bariatric surgery. Initially, frequently asked postoperative nutrition questions were identified based on input from three experts who consult bariatric patients. This process led to the creation of 27 questions, which were then posed to ChatGPT in Turkish, with responses recorded immediately. For evaluation, responses were categorized as: '1: consistent with the guidelines', '2: consistent but incomplete', '3: consistent but contradictory', and '4: inconsistent'. The responses were assessed by two experts, with a third expert reviewing any disagreements.
This study is non-human research and does not require ethics committee approval.
The results of the study revealed that 44% (12/27) of the responses generated by ChatGPT to patients' questions following bariatric surgery were consistent with established guidelines. 37% (10/27) of the generated responses were found to be consistent with guidelines but contained incomplete information. According to experts, 11.1% (4/27) of ChatGPT’s responses were found to be consistent with guidelines but contained contradictory statements. Only one response generated by the model was found to be entirely inconsistent with the guidelines. All questions are analysis in Table 1, while examples of evaluations for select questions are provided in Table 2.
Analysis of Questions: Compatible, Incomplete, Contradictory, and Incorrect
| Categories | Number of Compatible Questions | Number of Incomplete Questions | Number of Conflict Questions | Number of Completely incorrect Questions | Sum |
|---|---|---|---|---|---|
| Nutrition and weight management | 4 | 7 | 2 | 1 | 14 |
| Protein | 2 | 0 | 2 | 0 | 4 |
| GIS problem | 3 | 2 | 0 | 0 | 5 |
| Fluid intake | 3 | 1 | 0 | 0 | 4 |
| Total Percentage | 44,4 | 37 | 14,8 | 3,7 | 100 |
Examples of Evaluations of Responses to the Questions
| EXAMPLE OF QUESTIONS | |||
|---|---|---|---|
| 1 | 2 | 3 | 4 |
| What can I consume during the different stages?' | Will I regain weight after bariatric surgery? If so, what should I do?' | If I am vegan, how can I get enough protein after surgery? | 'If I feel hungry after eating, should I eat again after bariatric surgery? |
| Consistent | Incomplete information: Patient was not referred to a bariatric dietitian, nor was an evaluation for physiological causes of weight regain emphasized. | Contradictory: The model classified green leafy vegetables, such as spinach, broccoli, and Swiss chard, as rich protein sources; however, they are low in protein content. |
|
| How long should I take multivitamins after the surgery?' | What should my daily energy intake (kcal) be after bariatric surgery? | How many meals should I consume daily after the surgery?' | |
| Consistent | Incomplete information: The calorie amount was suitable for the first year but lacked guidance on increasing intake after one year. | Contradictory: In this case, the number of meals suggested for the first year was unclear, and the phrase 'large, frequent portions' was found to be confusing. | |
In the present study, nearly half of the responses generated by the ChatGPT model to frequently asked nutrition-related questions by bariatric surgery patients were found to be consistent with the guidelines, while the remaining responses showed deficiencies and contradictory statements.
In contrast to our study, Samaan et al. found that 86.8% of ChatGPT's responses to bariatric surgery questions were comprehensive, supporting its use as a reliable information source for patients [2]. Another study emphasized ChatGPT's role in improving the clarity of educational resources related to bariatric surgery, highlighting the potential of artificial intelligence in patient education [3]. Similarly, a study evaluating AI-generated responses to bariatric textbook questions reported an 83.0% accuracy rate for ChatGPT, indicating its practical and educational utility in bariatric care [4]. Moazzam et al. demonstrated ChatGPT's accuracy in responding to bariatric inquiries, while emphasizing the importance of tailoring responses to meet individual patient needs [5].
The primary reason for the lower accuracy rate of the model’s responses in our study compared to other studies is thought to be the model's limitations in the field of nutrition, where individualized treatment is highly emphasized. While the model can accurately convey textbook knowledge, it may fail to deliver the desired outcomes when required to generate personalized data.
The results of the study indicate that the ChatGPT model may provide incomplete and contradictory information to patients regarding nutrition after bariatric surgery. It is suggested that in specialized areas like nutrition, the model should be used as an additional resource alongside professional guidance. While the results of the study may vary with the advancement of the model driven by technological progress, evaluating the model as a complementary assistant in light of the current data is considered a more appropriate approach.