In a recent study conducted by researchers from AnsibleHealth and Massachusetts General Hospital (MGH), ChatGPT – an artificial intelligence (AI) chatbot that’s been making headlines since its release in November 2022 – was put up against questions from the United States Medical Licensing Exam (USMLE). After the makeshift exam, according to a report by Dr. Tiffany Kung, ChatGPT scored “at or near the passing threshold of 60% accuracy.”
Ultimately, that study shows the potential of ChatGPT in a healthcare context. But further studies aimed to determine more, including whether ChatGPT could accurately diagnose conditions.
Can ChatGPT Diagnose Diseases?
While functionally passing the USMLE was a significant feat, Dr. Joshua Tamayo-Sarver decided to assess whether ChatGPT could assist with the diagnosis of various medical conditions. Using anonymized data for approximately 35 to 40 patients, he provided ChatGPT with relevant details about patient symptoms and requested differential diagnoses based on how the patient presented in the emergency room (ER).
In some cases, ChatGPT performed admirably. For about half of the patients, ChatGPT provided roughly six potential diagnoses, and the one that aligned with Dr. Tamayo-Sarver’s diagnosis was present within the suggestions.
However, ChatGPT did demonstrate clear shortcomings. Dr. Tamayo-Sarver outlined the symptoms of a 21-year-old female patient, which included pain in the lower-right abdominal quadrant. ChatGPT presented options like appendicitis and an ovarian cyst but didn’t list ectopic pregnancy among the options offered. The actual diagnosis was an ectopic pregnancy, a condition that can be fatal if not quickly treated. Fortunately, she was correctly diagnosed in the ER.
The Risks of Using ChatGPT for Medical Diagnoses
Dr. Tamayo-Sarver ultimately determined that ChatGPT has potential when it comes to assisting with diagnosing medical conditions, but the results are highly dependent on being provided with thorough information. Additionally, ChatGPT can’t necessarily determine what follow-up questions may need to be asked or may struggle if a patient presents in a way that isn’t textbook.
As a result, relying on ChatGPT when diagnosing diseases comes with risks. ChatGPT may miss severe medical conditions if it isn’t provided with thorough information. Plus, it doesn’t have the ability to gauge an interaction with a patient and determine which follow-up questions are potentially pertinent based on how the actual conversation with a patient unfolds.
The risks associated with everyday people using ChatGPT for medical diagnoses are even greater. The average person has limited medical knowledge, so an accurate description of what they’re experiencing could prove challenging. Plus, patients may be unaware of critical factors.
For example, the 21-year-old woman described above didn’t know she was pregnant at the time of her diagnosis of an ectopic pregnancy. While Dr. Tamayo-Sarver knew the questions to ask to assess the possibility of a pregnancy, ChatGPT would likely fall short in that regard. If that patient had relied on AI instead of heading to the ER, the outcome could have been far more dire.
From the perspective of physicians, ChatGPT may cause a doctor to focus incidentally on incorrect diagnoses. Based again on the example of the female patient, had Dr. Tamayo-Sarver turned to ChatGPT first, it could incidentally create biased thinking, increasing the odds that he would have missed the critical diagnosis.
Does ChatGPT Have Potential as a Diagnostic Tool?
While there are undeniable risks associated with using technologies like ChatGPT for diagnosing diseases, that doesn’t mean AI doesn’t have potential as a diagnostic tool. AI can process a wealth of information incredibly quickly, potentially leading to insights or pattern identification that would be challenging for people.
However, more accurate outcomes rely on AI getting access to a treasure trove of patient data. At this time, ChatGPT isn’t trained on enough data to achieve a reliable degree of accuracy, so it’s critical to meter expectations until an AI does reach that level of competence.
Improving Patient Care with Language Services
ChatGPT and other AI technologies are poised to revolutionize the healthcare industry, as well as many others. However, these platforms are still in their infancy, and using them comes with an inherent degree of risk. For translations, ChatGPT and technologies like it can’t guarantee accuracy. Since accurate communication is crucial for patient care, working with a top-quality language services provider is the best choice.
If you need professional, reliable translators, Acutrans provides certified document translations in 24 hours. Along with generation translation and post-machine translation services, Acutrans has a program designed specifically for the healthcare industry, ensuring the provided translators have the necessary amount of medical knowledge to produce accurate translations.
Additionally, the Acutrans team offers over-the-phone, on-site, and video remote interpretation services in more than 200 languages. As with translations, Acutrans has an industry-specific interpretation program for the medical sector. Contact us for a free quote today.