Deep Learning Detection of Active Pulmonary Tuberculosis at Chest Radiography Matched the Clinical Performance of Radiologists

Data Sets

This retrospective study leveraged data from multiple sources. Data were collected with the participant’s consent and/or de-identified in accordance with local regulatory requirements and/or reviewed by the institution’s ethics committee or institutional review board before our receipt of the data set. The study adhered to the tenets of the Declaration of Helsinki. We leveraged de-identified chest radiograph data sets spanning nine countries for training and five countries for testing the DLS (a four-country combined test set and an additional test set of miners from South Africa), for a total of 10 countries (Tables 1, 2). Data were acquired between 1996 and 2020. DLS training used 160 187 images from Europe (13) (Azerbaijan, Belarus, Georgia, Moldova, Romania), India, and South Africa, and tuning used 3258 images from China (14–17), India, and Zambia (18). In addition, we used 550 297 images (19,20) for pretraining purposes (10 310 of which overlapped with the training and validation sets and none of which overlapped with the test sets) and 138 images with labeled lung segmentation masks from the United States data set for training and validating the lung cropping model. The trained DLS was tested on 1236 images from China (14,15), India, the United States (14,15), and Zambia and 1073 images from South Africa, using one image per patient for all (2309 test images). Additional details including inclusion and exclusion criteria, enrichment, and reference standard are presented in Figures 1 and E1 (online) and Tables 1 and 2. This study was funded by Google.

Table 1: Baseline Characteristics of Training and Validation Data Sets

Table 2: Baseline Characteristics of Test Data Sets

Figure 1:

Download as PowerPoint

Reference Standard for TB Status

For all validation and testing data sets, positive TB status was confirmed by means of microbiological testing (sputum culture or sputum smear) or NAAT (eg, GeneXpert MTB/RIF, Cepheid) or mixed clinical in the case of the United States (Montgomery, Md) and China (Shenzhen), two well known TB data sets (Tables 1, 2, and E1 [online]). In the training data sets, the reference standard varied due to site-specific practice differences and data availability, including microbiological testing, radiologist interpretation, clinical diagnoses (based on medical history and imaging), and NAAT. Additional information is provided in Appendix E1 (online).

DLS Description

Our DLS was developed to detect active pulmonary TB on chest radiographs and consists of three modules: a lung cropping model for identifying a bounding box spanning the lungs, a detection model for identifying regions containing possible imaging features of active TB (nodules, airspace opacities with cavitation, airspace opacities without cavitation, pleural effusion, granulomas, and fibroproductive lung opacities), and a classification model that takes the output from both the lung cropping model and the detection model to predict the likelihood of the chest radiograph being TB positive as well as whether there were any non-TB abnormalities (Fig 2). Further details on the model and training process are provided in Appendix E1 (online).

Figure 2:

Download as PowerPoint

Comparator Radiologist Reviews

To contextualize the DLS performance across data sets of varying levels of difficulty, all test set images were reviewed by radiologists. To examine the extent to which radiologists accustomed to practice in endemic versus nonendemic settings vary, these radiologists included two groups (10 India-based consultant radiologists and five U.S.-based board-certified radiologists). All India-based radiologists had experience in reading TB images; one U.S.-based radiologist had fellowship training in body and musculoskeletal and one had fellowship training in pediatrics. Data from one radiologist from India were excluded due to low sensitivity; this exclusion is described in more detail in the Results section. The India-based radiologists had an average of 6 years of experience (range, 3–9 years), and the U.S.-based radiologists had an average of 11 years of experience (range, 3–22 years). The radiologists were provided with both the image and additional clinical information about the patient when available (age, sex, symptoms, and HIV status); clinical information was not incorporated into the DLS. The radiologists labeled each of the 2309 images in the test data sets for the presence or absence of TB, their confidence (on a scale from 0 to 100) when they indicated the presence of TB, and whether there were any minor technical issues visible on the image. The validation data sets were labeled similarly.

Statistical Analysis

Our prespecified primary analyses compared the performance of the DLS with that of the India-based radiologists on the combined four-country test data set. For comparison with radiologists, we thresholded the DLS continuous score using an operating point of 0.45, chosen using the validation data sets and before evaluating the DLS on any of the test sets. We tested for noninferiority of sensitivity and specificity, both with a 10% margin prespecified based on sample size simulations on the validation set. To account for correlations within patient and within radiologist, we used the Obuchowski-Rockette-Hillis procedure (21,22) configured for binary data (23) and adapted to compare readers with the standalone algorithm (24) in a noninferiority setting (25). P < .0125 was considered indicative of a significant difference for the primary analyses (a conservative one-sided α of .025 was halved for a Bonferroni correction for two tests). Subsequent superiority testing was prespecified if noninferiority was met, which does not require multiple testing correction (26).

Prespecified secondary analyses included per–data set subgroup analysis for receiver operating characteristic (ROC) curves; sensitivity and specificity at the prespecified operating point; operating points corresponding to the WHO target sensitivity and specificity; matched sensitivity and specificity analysis on a per–data set and per-radiologist level; and comparisons of the India-based and U.S.-based radiologists and comparison of the DLS to the U.S.-based radiologists. Additional secondary analyses were performed in subgroups based on HIV status, images flagged by the reviewing radiologists to have minor technical issues, demographic information, and symptoms. Exploratory sputum smear (27) subgroup analysis was also conducted. Unless otherwise specified, 95% CIs were calculated using the bootstrap method with 1000 samples. We evaluated the performance of the DLS and radiologists on 247 randomly chosen TB-positive chest radiographs (the entirety of positive images from the four-country test set and a sample from South Africa) stratified by the presence of TB-related radiologic findings on chest radiographs. TB-related findings were labeled by a U.S.-based radiologist (C.L., with 17 years of post-residency experience). Due to small sample sizes, P values were not calculated in this smaller subgroup analysis. Finally, we conducted a simulated cost analysis (Appendix E1 [online]). All analyses were conducted using Python version 3.7.10 and the libraries numpy v1.19.5, scipy v1.2.1, sklearn v0.24.1, and pandas v1.1.5.

Data Set Characteristics

We had two test sets. Our four-country (China, India, the United States, and Zambia) test set included 1236 patients with a mean age of 38 years ± 20 (SD) (483 female subjects [39%]), and our one-country (South Africa) test data set included 1073 patients with a mean age of 43 years ± 8 (16 female subjects [1%]). DLS performance was first evaluated on a combined four-country test data set incorporating diverse subjects representing multiple races and ethnicities (Tables 1, 2). Of 1236 images (from 1236 subjects), 212 subjects were positive for TB based on culture or NAAT. Patient sources for these four data sets included TB referral centers, outpatient clinics, and active case findings. The India test data set was from a site independent of those used for training and validation of the model. An independent test data set from South Africa comprising a sample of mostly male subjects from a gold mining population served as an additional test set.

DLS Performance

In our combined four-country test data set, the DLS attained an area under the ROC curve (AUC) of 0.89 (95% CI: 0.87, 0.91) (Fig 3A). To contextualize the performance of the model, two groups of radiologists (from India [TB endemic] and the United States [TB not endemic]) reviewed the same images. One India-based radiologist was found to have a rate of flagging positives (and consequently sensitivity) substantially below the others (Fig E2 [online]) and so was excluded from subsequent analyses to avoid under-representing radiologist performance. The ROC curve of the DLS was above the sensitivity and specificity points of all nine remaining India-based radiologists (range of sensitivity increase at matching specificity: 1.9%–14.2%, P < .05 met for six of nine radiologists) (Fig 3A).

Figure 3:

Download as PowerPoint

Our prespecified primary analyses involved comparing the DLS at a prespecified operating point with India-based radiologists (Table 3). The sensitivity of the DLS (88%) was higher than that of the India-based radiologists (mean sensitivity, 75%; P < .001). At the same operating point, the specificity of the DLS (79%) was similarly noninferior to that of the India-based radiologists (mean specificity, 84%; P = .004). The distribution of the India-based radiologists’ sensitivity and specificity is shown in Figure E2 (online).

Table 3: Comparison of the Sensitivity and Specificity of the DLS and Radiologists

Comparison of India-based and U.S.-based Radiologists

Although both India-based and U.S.-based radiologists had sensitivities and specificities that tracked closely and slightly below the ROC curve of our model (for U.S.-based radiologists, the range of sensitivity increase at matching specificity was 0.9%–15.6%, P < .05 met for four of five radiologists), the conservativeness with which the two groups of radiologists called images as positive for TB appeared to differ (Figs 3A–C, E2 [online]). India-based radiologists were more specific than U.S.-based radiologists (mean specificity, 84.0% vs 74.4%, respectively; P = .019, exact one-tailed permutation test) and trended toward less sensitivity (mean sensitivity, 74.6% vs 77.9%; P = .299). The sensitivity and specificity of the DLS remained comparable to that of the U.S.-based radiologists (P value for noninferiority <.05 for both sensitivity and specificity) (Table 3).

Per–Data Set Analysis

Results of detailed per-country subgroup analysis for the combined test data set are presented in Tables 3 and E7 (online) and Figure 3B and described in Appendix E1 (online). For the additional independent test data set from a mining population in South Africa (Fig 3D, Table E3 [online]), the ROC curve of the model was above the curves for all but one radiologist. At the same prespecified operating point as the other data sets, the DLS was noninferior both in terms of sensitivity and specificity to both India-based and U.S.-based radiologists (P < .05 for all). At a higher specificity operating point selected based on the South Africa validation data sets, the DLS was again noninferior in both sensitivity and specificity compared with the India-based radiologists but had higher specificity (P = .01) than the U.S.-based radiologists at the cost of not being noninferior in sensitivity (P = .59) (Table E3 [online]). When matching DLS sensitivity to mean radiologist sensitivity per data set, the DLS was non-inferior to both India-based and U.S.-based radiologists (Table E7 [online]).

Inter–Data Set Comparisons

To better understand inter–data set differences, histograms of DLS prediction scores were plotted separately for images from subjects who were TB positive or negative (Fig 4). The distribution of DLS scores for both TB-positive and TB-negative images remained similar across the China, India, and U.S. data sets (Table E6 [online]). However, there was a higher proportion of TB-negative images with high DLS scores in the Zambia data set. This appears to have been a consequence of first-round computer-aided detection screening of the Zambia data set, which censored many normal-appearing chest radiographs, resulting in a more challenging data set with a relative paucity of normal chest radiographs.

Figure 4:

Download as PowerPoint

Visualizing Challenging Images

We next inspected chest radiographs to better visualize the types of images that are challenging for the DLS. In Figure 5, we show images where the DLS provided the correct interpretation. In images where the DLS had confident predictions (where the DLS value was close to 1 or 0), there is high correlation with radiologists’ interpretations. In challenging predictions (where the DLS value was close to the operating point, 0.45), radiologist performance also decreased. In two challenging TB-positive images that the DLS predicted incorrectly (Fig 6), radiologist annotations indicated lymphadenopathy and miliary infection. In two challenging TB-negative images that the DLS predicted incorrectly, radiologist annotation indicated findings consistent with bronchiolitis and left lower lung infection.

Figure 5:

Download as PowerPoint

Figure 6:

Download as PowerPoint

Matched Performance to Radiologists and WHO Targets

To facilitate comparisons despite the wide range in radiologist sensitivities and specificities, both across data sets and readers, we next conducted a matched analysis by shifting the DLS operating point on a per–data set level to (a) compare sensitivities at mean radiologist specificity and (b) compare specificities at mean radiologist sensitivity, as detailed in Tables 4 and E7 (online) as well as in Appendix E1 (online).

Table 4: Comparing Performance of the DLS to Radiologists after Matching to Mean Radiologist Specificity per Data Set

The WHO “target product profile” for a TB screening test recommends a sensitivity greater than 90% and a specificity greater than 70%. In matched performance analysis at 90% sensitivity, the DLS had a specificity of 77% on the combined data set; at 70% specificity, the DLS had a sensitivity of 93%. Both of these met the recommendations. This finding remained true in the China, India, and U.S. data sets but not in the enriched Zambia data set (Table 5).

Table 5: Model Performance at the WHO’s Target Sensitivity and Specificity Thresholds

Subgroups according to HIV Status

We next considered subgroups based on HIV status where available, which corresponded to most subjects in the Zambia data set. In the HIV-negative confirmed subgroup, the DLS had an AUC of 0.92 (95% CI: 0.85, 0.97). In the HIV-positive confirmed subgroup, the DLS had an AUC of 0.80 (95% CI: 0.79, 0.89). A similar lower sensitivity and specificity were observed for radiologist assessments (Fig 3C). The DLS remained comparable to the radiologists for both subgroups, notably even though HIV status information was not integrated within the DLS. Additional subgroup analyses are presented in Figures E3–E9 (online).

Detecting Non-TB Pulmonary Findings

We further evaluated the “abnormality” detector in the DLS on the India and Zambia test sets, using labels provided by three India-based radiologists as the reference standard (Appendix E1 [online]). A positive image was defined as either being TB-positive or having another abnormal chest radiograph finding. Depending on how many radiologists indicated the presence of the abnormality, the AUC of the DLS ranged from 0.93 to 0.97 (Fig E6 [online]).

Stratification according to TB-associated Radiologic Findings

We found that our DLS had high sensitivity (range, 89%–100%) for all TB-associated radiologic abnormalities (Table 6). All DLS sensitivities were higher than the mean India-based and U.S.-based radiologist sensitivities. For images without any of our 10 TB-specific findings, sensitivity was lower for all: 33% for the DLS, a mean of 25% ± 8 for the India-based radiologists, and a mean of 45% ± 15 for the U.S.-based radiologists.

Table 6: Efficacy for Various Signs for the Diagnosis of Mycobacterium tuberculosis by the DLS in a Subgroup of 247 Subjects with a Positive Nucleic Acid Amplification Test

Cost Analysis

Finally, in our analysis of potential cost savings, we simulated a workflow where subjects only proceed to NAAT if flagged as positive by the DLS, as detailed in Appendix E1 (online). On the basis of the performance in the India data set, as prevalence decreases from 10% to 1%, the cost per TB-positive patient detected increased substantially and the cost savings compared with use of NAAT alone increased from 73% to 82% (Fig 7). The corresponding cost savings at low prevalence is not as profound when simulating the WHO target (range, 47%–53%) and a lower-specificity device (range, 42%–48%).

Figure 7:

Download as PowerPoint