Artificial intelligence (AI) technologies offer unprecedented opportunities within a healthcare context. The use of AI technologies to aid in the diagnosis of diseases such as cancer and TB has attracted much attention. AI, specifically the application of deep-learning neural networks, is increasingly being applied in the field of medical imaging for the computer-aided detection (CAD) of disease. Artificial neural networks mimic human neural networks and are able to learn supervised or unsupervised from training datasets. Multiple computer-aided reading software that utilize deep neural networks for the recognition of TB-related abnormalities from CXRs are now commercially available. These technologies hold promise as a TB screening and triage tool, and the potential to expand diagnostic capabilities, particularly in rural and low-resource contexts, is tremendous.

The Stop TB Partnership is working through a vast network of partners to collate information on commercially available AI products, to evaluate the products as an independent body and to support implementers to conduct pilot studies and project.

Evidence produced by the Stop TB Partnership contributed to the World Health Organization’s TB screening guideline update in April 2021; when, for the first time, AI was recommended as a triage tool for TB in adults (read the full guideline here). Stop TB Partnership continues to work closely with WHO on global policy and guidance for CAD implementers.

Read our new practical guide on the screening and triage for TB using computer-aided detection (CAD) technology and ultra-portable x-ray systems

This comprehensive implementation guide provides advice on how to convert WHO policy guidance into a practical implementation plan, building on initial field experience gained by early implementers. This guide offers various technical explanations of the newly added products available in the GDF catalogue and relevant implementation resources, including a high-level checklist of vital implementation steps and budgetary considerations, technical specifications for use during procurement, and checklists for site assessment. Early implementer experience is presented in several case studies that identify lessons learned and challenges that arose.

Access Screening and Triage for TB using Computer-Aided Detection (CAD) Technology and Ultra-portable X-Ray Systems: A Practical Guide here

CAD Software Product Landscape Analysis

With many commercial CAD products available on the market, it is hard for country program, implementing partners, and civil society and communities to learn to be updated with the market of CAD products and be informed of new development.

Stop TB invites you to access the newly released www.ai4hlth.org, co-developed with the FIND, the global alliance for diagnostics, which provides the first extensive landscape overview of available CAD products that can interpret CXR images for TB and with detailed technical specifications, including:

  • Certification
  • Price
  • Development Stage
  • Deployment Method: cloud vs offline
  • Hardware and Software requirements
  • Input requirements
  • Output format
  • Target Setting
  • Current market
  • Input and Output requirements
  • Product development and training

As of Sept 1, 2020, there are eight CAD products specific to TB which are available on the market, CAD4TB from Delft imaging (Netherlands), DxTB from DeepTek (India), InferRead DR Chest from Infervision (China), JF CXR-1 from JF Healthcare (China), JVIEWER-X from JLK, Lunit INSIGHT CXR from Lunit (South Korea), qXR from Qure.ai (India), XrayAME from Epcon (Belgium). Solutions with CE certification are CAD4TB, InferRead DR Chest, JVIEWER-X, Lunit INSIGHT CXR, and qXR. A further three products are currently in a validation stage: AXIR from Radisen (South Korea), Dr CADx from Dr CADx (Zimbabwe), and T-Xnet from Artelus (India).

Read more

Conduct Independent Evaluation of Commercial CAD Software Products

We evaluate commercial AI product using the TB REACH-CXR Archive, a privately held and de-identified evaluation database of chest radiographs for independent and external validation of commercial CAD products. The TB REACH-CXR Archive is large dataset of 30,957 posterior-anterior chest x-ray images for 30,957 patients collected through 4 projects funded by the Stop TB Partnership’s TB REACH Initiative between 2015-2020. All images have been de-identified to protect patient privacy. Randomly generated identifiers are used to group distinct reports and patients. Each project study was conducted in a different setting and geographic regions.

Our evaluation report includes:

  1. Area under the Receiver Operating Characteristic and Precision Recall Curves
  2. Comparison with human radiologist
  3. AI accuracy histogram
  4. Threshold Score selection (including measures of accuracy and cost-effectiveness)

We are developing an independent online platform for the automated evaluation of AI software products using our large and diverse data sets.

Publications from the TB REACH-CXR Archive can be found below.

Contact us to be evaluated using the Stop TB chest x-ray archive: dhthub@stoptb.org.

Virtual Innovations Spotlights on AI-Powered CAD Software

In order to support country programmes, healthcare providers, the communities and people affected by TB and our partners during COVID-19, the TB REACH team presented current work on CAD at the Stop TB’s virtual innovation spotlights (VIS). The presentation and recording can be accessed:

Artificial Intelligence (AI)/Computer Aided Detection (CAD) - Breaking with Tradition: The Value and Use of AI/CAD for TB Detection powered by Stop TB Partnership - TB REACH

Webinar recording

Presentation

More Virtual Innovations AI Spotlights can be found here.

TB REACH’s AI / CAD Projects

TB REACH has funded and supported the implementation of CAD products across different countries (Peru, Pakistan, Zambia, India, Myanmar, Cambodia, Vietnam, Cameroon, Kenya, Moldova and Bangladesh), targeting different populations. The AI/CAD tools include CAD4TB, qXR, and Lunit.

Read more in our article about artifical intelligence and TB diagnostics here.

Recent Publications

We conducted a landscape analysis to collect information from developers known to have, or soon to have, a CAD product for TB. We identified 27 CAD developers and 11 completed our survey with details about the certification, deployment, operational characteristics, input requirements, output format, pricing, and data privacy of their latest product version. For each response, a summary product profile was created based on the information provided and these were published on an open-access website: ai4hlth.org. Online deployment was most common, but offline versions were found to be available for settings without stable network connection. Almost all CAD products are agnostic to brand and model of the digital x-ray platform and almost all integrate with legacy systems. Input is commonly in DICOM and/or JPEG, PNG, TIFF format. Output is often a heatmap and numeric abnormality score for TB. This study provided, for the first time, an extensive overview of the available CAD products that can interpret chest x-ray images for TB. By making information more accessible and searchable, we hope to better brief TB implementers on the variety of products available and facilitate informed decision making when using this technology to ultimately serve more people with TB.

Read more

We conducted a retrospective evaluation of three DL systems (CAD4TB, Lunit INSIGHT, and qXR) for detecting TB-associated abnormalities in chest radiographs from outpatients in Nepal and Cameroon. All 1196 individuals received a Xpert MTB/RIF assay and a CXR read by two groups of radiologists and the DL systems. Xpert was used as the reference standard. The area under the curve of the three systems was similar: Lunit (0.94, 95% CI: 0.93–0.96), qXR (0.94, 95% CI: 0.92–0.97) and CAD4TB (0.92, 95% CI: 0.90–0.95). When matching the sensitivity of the radiologists, the specificities of the DL systems were significantly higher except for one. Using DL systems to read CXRs could reduce the number of Xpert MTB/RIF tests needed by 66% while maintaining sensitivity at 95% or better. Using a universal cutoff score resulted different performance in each site, highlighting the need to select scores based on the population screened. These DL systems should be considered by TB programs where human resources are constrained, and automated technology is available.

Download PDF

Background 

Artificial intelligence (AI) algorithms can be trained to recognise tuberculosis-related abnormalities on chest radiographs. Various AI algorithms are available commercially, yet there is little impartial evidence on how their performance compares with each other and with radiologists. We aimed to evaluate five commercial AI algorithms for triaging tuberculosis using a large dataset that had not previously been used to train any AI algorithms. 

Methods 

Individuals aged 15 years or older presenting or referred to three tuberculosis screening centres in Dhaka, Bangladesh, between May 15, 2014, and Oct 4, 2016, were recruited consecutively. Every participant was verbally screened for symptoms and received a digital posterior-anterior chest x-ray and an Xpert MTB/RIF (Xpert) test. All chest x-rays were read independently by a group of three registered radiologists and five commercial AI algorithms: CAD4TB (version 7), InferRead DR (version 2), Lunit INSIGHT CXR (version 4.9.0), JF CXR-1 (version 2), and qXR (version 3). We compared the performance of the AI algorithms with each other, with the radiologists, and with the WHO's Target Product Profile (TPP) of triage tests (≥90% sensitivity and ≥70% specificity). We used a new evaluation framework that simultaneously evaluates sensitivity, proportion of Xpert tests avoided, and number needed to test to inform implementers’ choice of software and selection of threshold abnormality scores. 

Findings

Chest x-rays from 23 954 individuals were included in the analysis. All five AI algorithms significantly outperformed the radiologists. The areas under the receiver operating characteristic curve were 90·81% (95% CI 90·33–91·29) for qXR, 90·34% (89·81–90·87) for CAD4TB, 88·61% (88·03–89·20) for Lunit INSIGHT CXR, 84·90% (84·27–85·54) for InferRead DR, and 84·89% (84·26–85·53) for JF CXR-1. Only qXR (74·3% specificity [95% CI 73·3–74·9]) and CAD4TB (72·9% specificity [72·3–73·5]) met the TPP at 90% sensitivity. All five AI algorithms reduced the number of Xpert tests required by 50% while maintaining a sensitivity above 90%. All AI algorithms performed worse among older age groups (>60 years) and people with a history of tuberculosis. 

AI algorithms can be highly accurate and useful triage tools for tuberculosis detection in high-burden regions, and outperform human readers. 

Read more

Funding source:

Pilot projects and analyses primarily funded by the Government of Canada through the TB REACH Initiative.