The system was tested on 70 complex clinical cases, ranging from rare disease diagnoses to multifactorial complications, scenarios that often require years of experience and intensive collaboration between specialists. o1-preview correctly diagnosed 78.3% of all cases examined , accurately diagnosing 88.6% of the most complex ones , significantly outperforming its predecessor GPT-4, which managed 72.9%. o1-preview proved to not only compete, but outperform professionals in several areas, identifying challenging pathologies with impressive accuracy. When it came to medical reasoning, o1-preview’s performance was even more impressive.
a standard measure for job function email database assessing the quality of medical reasoning, the AI system achieved perfect scores in 78 out of 80 cases. To put things into perspective, experienced doctors achieved perfect scores in only 28 cases, while trainee doctors managed to do so in only 16 cases.
This performance has been attributed to the system's ability to analyze huge amounts of clinical data, combine information from different sources, and use deep learning approaches to suggest plausible diagnoses. However, experts emphasize that the real strength of AI is not just in numbers, but in the ability to act as a diagnostic assistant, supporting doctors in making complex decisions.
Despite its success, the study did not fail to highlight some critical issues. One of the main limitations concerns the system's ability to assess the relative probability between different diagnoses. Often, o1-preview proved too precise in detailing less relevant diagnoses, providing long-winded answers that could confuse a doctor or require further interpretation. Furthermore, while AI excels at models based on structured data, it struggles to interpret subjective clinical aspects, such as symptoms described verbally by patients. This human element, often crucial in medicine, represents a significant challenge for a technology that aims to integrate with daily clinical practice.