AI System Possibly Capable of Surpassing Human Experts in Breast Cancer Prediction

In a report published in Nature, an artificial intelligence (AI) system was suggested to be capable of surpassing human experts in the prediction of breast cancer.

To understand the full extent to which this sort of technology could benefit patient care, the researchers indicated that prospective clinical studies will be required.

“These analyses highlight the potential of this technology to deliver screening results in a sustainable manner despite workforce shortages in countries such as the UK,” the authors wrote. 

In order to assess the program’s performance in the clinical setting, researchers curated a large representative dataset from the UK and a large enriched dataset from the US. From this, the researchers demonstrated an absolute reduction of 5.7% and 1.2% (US and UK, respectively) in false positives, and 9.4% and 2.7% in false negatives. Additionally, they were able to provide evidence that the system could generalize from the UK to the USA.

In an independent study of 6 radiologists, the area under the receiver operating characteristic curve (AUC-ROC) for the AI system was great than the AUC-ROC for the average radiologist by an absolute margin of 11.5%, indicating that the AI system outperformed all of the human readers. Additionally, the researchers ran a simulation in which the AI system participated in the double-reading process used in the UK, finding that the AI system maintained non-inferior performance and reduced the workload of the second reader by 88%. 

“We present early evidence of the ability of the AI system to generalize across populations and screening protocols,” the authors wrote. “This suggests that in future clinical deployments, the system might offer strong baseline performance, but could benefit from fine-tuning with local data.”

The optimal use of the AI system within typical clinical workflows has yet to be determined, according to the researchers. The specificity advantage exhibited by the system suggests that it could serve to decrease recall rates and unnecessary biopsies. The gain in sensitivity observed in the US data indicates that the AI system might be capable of identifying cancers earlier than the standard of care. Moreover, a study of the localization performance of the AI system suggests it holds early promise for flagging questionable regions for scrutiny by experts.

Remarkably, the additional cancers identified by the AI system tended to be invasive rather than in situ disease.

“As a shortage of mammography professionals threatens the availability and adequacy of breast-screening services around the world, the scalability of AI could improve access to high-quality care for all,” the authors wrote.

Of note, all of the radiologists in the study were eligible to interpret screening mammograms in the US, however they did not uniformly receive fellowship training on breast imaging, indicating that a higher benchmark for performance may have been achieved from more specialized readers. The vast majority of images used in this study were also acquired on devices made by Hologic, and future research should assess the performance of the system across a variety of manufacturers. 

Reference:

McKinney SM, Sieniek M, Godbole V, et al. International evaluation of an AI system for breast cancer screening. Nature. doi:10.1038/s41586-019-1799-6.