We evaluate commercial AI products using the TB REACH-CXR Archive, a de-identified evaluation database of 30,957 postero-anterior chest radiographs collected through 4 projects funded by the Stop TB Partnership’s TB REACH initiative between 2015-2020. These are privately held for the purposes of independent and external validation of commercial CAD products. These projects and analyses were funded by Global Affairs Canada.
Contact us to be evaluated using the Stop TB chest X-ray archive: firstname.lastname@example.org
Comparing different versions of computer-aided detection products when reading chest X-rays for tuberculosis
We comprehensively compared the performance of the newest versions of two CAD (CAD4TB and qXR) to their WHO-evaluated predecessors. We used a case control sample of 12,890 chest X-rays to compare performance and model the programmatic effect of upgrading to the newer versions. We found that both newer versions significantly improved upon their predecessor’s ability to detect TB, performing better than the human readers. We also showed that the AI underlying new software versions can differ remarkably from the old and resemble an entirely new product altogether. We further demonstrate that, unlike laboratory diagnostic tools, CAD software updates could significantly impact the selection of appropriate threshold scores, the number of people with TB detected and cost-effectiveness. Our results underscore the need for rapid evidence generation to evaluate newer CAD versions in the fast-growing medical AI industry
Tuberculosis detection from chest x-rays for triaging in a high tuberculosis-burden setting: an evaluation of five artificial intelligence algorithms
We evaluated five commercially available AI algorithms for TB triage using 23,954 chest X-rays collected at TB screening centers in Bangladesh and Xpert MTB/RIF as the reference standard. All CAD outperformed experienced Bangladeshi radiologists and could halve the number of Xpert tests required while sensitivity remained greater than 90%. All algorithms performed worse in older age groups and those with a history of TB. Finally, we proposed a new evaluation framework for CAD to better inform selection of threshold scores.
This evaluation showed that CAD can be a highly accurate triage tool for TB.
A new resource on artificial intelligence powered computer automated detection software products for tuberculosis programmes and implementers
We conducted a landscape analysis to collect information from developers known to have, or soon to have, a CAD product for TB. We identified 27 CAD developers and 11 completed our survey with details about the certification, deployment, operational characteristics, input requirements, output format, pricing, and data privacy of their latest product version. For each response, a summary product profile was created based on the information provided and these were published on an open-access website: ai4hlth.org.
CAD products are constantly being improved and the site will continuously be updated to account for updates and new products. This unique online resource aims to inform the TB community about available CAD tools and enable TB programmes to identify the most suitable product to incorporate in interventions.
Using artificial intelligence to read chest radiographs for tuberculosis detection: A multi-site evaluation of the diagnostic accuracy of three deep learning systems
We conducted a retrospective evaluation of three CAD systems (CAD4TB, Lunit INSIGHT, and qXR) for detecting TB-associated abnormalities in chest radiographs from outpatients in Nepal and Cameroon. All 1196 individuals received a Xpert MTB/RIF assay and a CXR read by two groups of radiologists and CAD. Xpert was used as the reference standard. When matching the sensitivity of the radiologists, the specificities of the DL systems were significantly higher except for one.
Using CAD systems to read CXRs could reduce the number of Xpert MTB/RIF tests needed by 66% while maintaining sensitivity at 95% or better. CAD should be considered by TB programs where human resources are constrained, and automated technology is available.