Various artificial intelligence (AI) algorithms submitted as part of a challenge demonstrated the ability to identify different breast cancers during screening mammography, according to the results of a study published in Radiology. Ensemble models of the top submitted algorithms indicated that the algorithms enabled increased sensitivity while maintaining low recall rates.
“We were overwhelmed by the volume of contestants and the number of AI algorithms that were submitted as part of the Challenge,” stated lead study author Yan Chen, PhD, Professor of Digital Screening, Faculty of Medicine & Health Sciences, University of Nottingham, United Kingdom. “It’s one of the most participated-in RSNA AI Challenges. We were also impressed by the performance of the algorithms given the relatively short window allowed for algorithm development and the requirement to source training data from open-sourced locations.”
Background and Study Methods
Many different AI algorithms were submitted to the Radiological Society of North America (RSNA) as part of the crowdsourced RSNA Screening Mammography Breast Cancer Detection AI Challenge from 2023. More than 1,500 international teams participated in the challenge with the goal of creating an AI model that would improve the automation of cancer detection from mammography images.
The teams were given a training data set of about 11,000 breast screening images provided by Emory University in Atlanta, Georgia, and BreastScreen Victoria in Australia, and the teams could also source publicly available data to train their models.
Then, Dr. Chen and her team analyzed 1,537 final algorithms by testing them on 10,830 single-breast exams that were confirmed for presence or absence of cancer with pathology results; these patients were followed for at least a year to confirm a lack of cancer in those with negative pathology results. The analysis results were collected in the report published in Radiology.
The study authors then tried to combine the best-performing algorithms into ensemble models to try to further improve diagnostic performance.
Key Findings of Analysis
For all of the final AI algorithms, the median recall rate was 1.7%, the median sensitivity rate was 27.6%, the median specificity rate was 98.7%, and the median positive predictive value was 36.9%. Individual algorithms varied depending on factors of cancer type, imaging equipment manufacturer, and clinical site of screening.
For the top AI algorithm, the recall rate was 1.5%, the sensitivity rate was 48.6%, the specificity rate was 99.5%, and the positive predictive value was 64.6%.
Ensemble models of the top three algorithms showed a sensitivity rate of 60.7%, a specificity rate of 98.8%, and a recall rate of 2.4%; the rates increased to 67.8%, 97.8%, and 3.5%, respectively, when testing the top ten algorithms.
“When ensembling the top performing entries, we were surprised that different AI algorithms were so complementary, identifying different cancers,” Dr. Chen stated. “The algorithms had thresholds that were optimized for positive predictive value and high specificity, so different cancer features on different images were triggering high scores differently for different algorithms.”
The US dataset showed a lower sensitivity than the Australian dataset of 52% vs 68.1% based on the top three ensemble model (odds ratio [OR] = 0.51; P = .02). A greater sensitivity was seen for detection of invasive cancers compared with noninvasive cancers using the top three ensemble model (68% vs 43.8%; OR = 2.73; P = .001).
The researchers noted that the performance of the ensemble model of the top ten AI algorithms was similar to that of the average screening radiologist in Europe or Australia.
Many of the algorithms are open source, which could lead to further improvement of these and other tools. By releasing the algorithms and a comprehensive imaging dataset to the public, participants provide valuable resources that can drive further research and enable the benchmarking that is required for the effective and safe integration of AI into clinical practice,” Dr. Chen said. Further studies of these algorithms are planned.
Disclosure: For full disclosures of the study authors, visit pubs.rsna.org.
