Courtesy of Pexels.
GPT3 is a large language model built upon a neural network with 175 billion parameters, and it’s transforming the way we learn. With simple, accurate explanations of complex topics and step-by-step answers on tasks ranging from coding a website to cooking a steak, its responses can promote creativity and our acquisition of knowledge. These capabilities have spurred discussions on how ChatGPT, our interface with the GPT3 model, can be used as an educational tool in a variety of fields, including medicine.
Recently, researchers at Yale University and the National University of Ireland, Dublin tested how well ChatGPT could perform on the United States Medical Licensing Examination (USMLE) Steps 1 and 2. USMLE Steps 1 and 2 are considered to be the most important exams that medical school students take on their path to matching with a residency program.
“We saw ChatGPT [when it came out] and started asking [it] clinical questions, and we were like, ‘Wow, it does better than we expected.’ So we started wondering what would be a more systematic evaluation of this,” said Vimig Socrates, a fourth-year PhD student at Yale. Socrates and the team tested ChatGPT’s performance on the USMLE Step 1 and 2 exams by using the National Board of Medical Examiners (NBME) practice exams, known to be the most reputable, and AMBOSS practice exams, which offer explanations for each question.
The researchers found that ChatGPT scored 64.4 percent, 57.8 percent, 44 percent, and 42 percent on the NBME-Free-Step1, NBME-Free-Step2, AMBOSS-Step1, and AMBOSS-Step2 exams, respectively. Above 60 percent is considered a passing rate for the USMLE Step 1 exam.
These findings highlight the potential for ChatGPT to be leveraged in medical applications. Faced with complex and overwhelming patient charts, doctors must often navigate cluttered information to identify the necessary details required to treat their patients. “You have to know exactly what you’re looking for. You can look up if a patient was on warfarin [an anticoagulant medication], but it’s really hard to find all blood thinners or surgical risk factors a patient may have,” said Aidan Gilson, a fifth-year Yale medical student currently doing a year of research. However, by integrating ChatGPT or similar artificial intelligence models into patient charts, doctors could efficiently identify crucial information pertaining to their patients’ medical histories.
Similarly, the potential for ChatGPT can be applied on the patient side. Patients often receive a large amount of information from their healthcare provider, which can be challenging to interpret. By using a digital assistant like ChatGPT to navigate healthcare data, patients may be better able to comprehend their medical records and communicate with their healthcare providers. By leveraging the capabilities of large language models, healthcare providers and patients alike can collaborate more effectively to achieve better outcomes.