Much hyped AI products like ChatGPT can provide medics with ‘harmful’ advice, study says

The much-hyped AI products like ChatGPT may provide medical doctors and healthcare professionals with information that can aggravate patients’ conditions and lead to serious health consequences, a study suggests.

Researchers considered three nutrition-related diseases in their study, with two experienced dietitians examining a total of 63 ChatGPT outputs of disease management against medical guidelines.

Their findings sound a warning to medics and health personnel who only lean on AI and related products in passing health and medical advice to their patients.

Farah Naja, the University of Sharjah’s Professor of Nutritional Epidemiology, highlighted that the research warns against relying solely on AI interfaces for diabetes management, citing potentially incorrect or harmful advice from ChatGPT, which could compromise patient health and safety.

Prof. Naja is the lead author of the study published in the European Journal of Clinical Nutrition in which scientists from the United Arab Emirates, Lebanon, Bahrain, and Turkey investigate the correctness and accuracy of ChatGPT answers to various prompts of how to manage and prevent diabetes and metabolic abnormalities.

The study investigates dietary management, nutrition care processes, and menu planning for a 1,500-calorie diet, using 9 prompts per condition for each domain.

“A total of 63 prompts were fed into the GPT3.5-turbo0301 model through the ChatGPT interface provided by OpenAI, during October 2023. Two experienced dietitians evaluated the chatbot output’s concordance with the Academy of Nutrition and Dietetics’ guidelines,” the scientists write.

Scientists aimed to raise awareness about artificial intelligence’s role in dietetic care for diseases like diabetes and metabolic abnormalities, assessing ChatGPT’s accuracy in providing nutritional management.

Diabetes and metabolic abnormalities, including high blood glucose and high blood lipids, are prevalent worldwide and pose significant challenges for individuals, societies, and governments. The management of these diseases necessitates not only medication but also subtle and complex lifestyle changes, including dietary intake.

Researchers evaluated ChatGPT’s limitations by assessing 7 diet-related metabolic diseases, including Type 2 diabetes, metabolic syndrome, central obesity, hyperglycemia, hypertension, low high-density lipoprotein, and hypertriglyceridemia.

Prof. Naja pointed to weight loss, which she said was “critical in the management of diabetes and metabolic abnormalities … Yet the outputs of the ChatGPT missed the weight loss recommendations along with guidance on achieving an energy deficit.”

When asked to provide sample menus for the health conditions considered in the study, ChatGPT outputs did not meet the requirements in terms of energy, carbohydrates, and fat, in addition to calcium and vitamin D, Prof. Naja maintained.

Statistics show that diabetes accounts for 2 million out of 17 million people who die annually from a non-communicable disease before age 71, according to the World Health Organization (WHO).

WHO attributes 19% of global deaths to metabolic risk factors, like overweight and high blood pressure, which are part of the study’s three main nutrition management-related domains.

The authors acknowledge ChatGPT’s role in health nutrition-related health issues, underscoring its “dynamic conversational capabilities” and its potential to offer “personalized and engaging education.”

They also note that the prevalence of an “increasing number of real-world applications of ChatGPT has been launched in the field of nutrition and health, including nutritional counseling platforms, health and fitness apps, public health campaigns and chatbots’ incorporations within school health and nutrition curricula.”

However, they stress that research is scant on whether “the transformative potential of using ChatGPT in nutrition education” and its huge potential powers can be of any “limitations in providing clinical nutritional advice.”

Prof. Naja said the study found that ChatGPT missed “appropriate physical activity and weight loss recommendations along with guidance on achieving an energy deficit” despite their being “critical in the management of diabetes and metabolic abnormalities.

“ChatGPT outputs were incomplete in terms of guidance on specific nutrients … and did not address the need to increase fiber intake or to consume whole grain products for all the considered conditions.”

The study concludes “ChatGPT, and potentially other future AI chatbots, react to the user’s prompts in a human-like way, but cannot replace the dietitians’ expertise and critical judgment. Healthcare practitioners should exercise caution when using AI chatbots in clinical practice and raise collective awareness about associated risks.