Deep learning in the radiologic diagnosis of osteoporosis: a literature review

We developed this review based on the PRISMA guideline.¹³ A literature search was conducted using the PubMed and Web of Science databases with the following search terms: (“artificial intelligence” OR “deep learning” OR “neural network”) AND osteoporosis AND (“CT” OR “X-ray” OR “MRI” OR “panoramic radiography”). In the process of searching, we excluded review articles. This study included literature published from 2018 to 2023. Specifically, studies after 2018 were selected because researchers have made significant advancements in the application of DL since then whereas substantial contributions of ML and AI can be found in previous research. Furthermore, we focused on routine radiologic methods, such as X-ray, CT, and magnetic resonance imaging (MRI), used to opportunistically screen for osteoporosis. Based on the categorization criteria proposed by the WHO, a T-score lower than or equal to −2.5 is indicative of osteoporosis whereas a T-score between −1 and −2.5 denotes osteopenia. This standard was widely accepted in the included studies.

After the initial search, a two-step screening scheme was implemented. Titles and abstracts were screened for inclusion, and the following exclusion criteria were applied: (1) duplicated studies, and (2) studies that were irrelevant to ML and osteoporosis. Additionally, we restricted studies to only those reports written in the English language. The full texts of each study were then collated for further screening to ensure study quality. In cases where there was disagreement between the two reviewers (He and Zhu) during the initial screening, the corresponding author (Xu) facilitated discussion and made the final decision. Owing to the diversity of methods and objectives, quantitative analysis was limited.

Data extraction from the included studies was performed by two independent reviewers (He and Zhu) using a standardized data extraction form. Any disagreements were resolved through consensus or with the corresponding author (Xu) making the final judgment. The extracted information included the country, year, author, model selection, data amount, study objective, and performance metrics of the model such as area under the curve (AUC), accuracy (ACC), sensitivity, and specificity, among others.

This study was a literature review regarding deep learning in osteoporosis rather than an original study. Thus, there was no observational or interventional process in the study in terms of epidemiological design. For this reason, the requirement for ethics approval and informed consent was waived.

In the initial search of PubMed and Web of Science, 330 records were identified. After the exclusion of duplicates, defined as citations found in both databases, the titles and abstracts of 202 records were screened, and 122 of these studies were excluded after screening. Next, we conducted a full-text review of the remaining 80 articles. A total of 40 articles were finally included in the review. The entire screening process is illustrated in Figure 1.

The characteristics of the included studies are presented in Table 1 and Figure 2. All included studies were published between 2018 and 2023, with most being published after 2020. X-ray (17/40)^14–25 and CT (16/40)^26–41 images were the most frequently used modalities; digital projection radiography (DPR)^42–44 was used in three studies and MRI^45,46 in two studies. Two studies^47,48 using vertebral fracture assessment (VFA) images were considered in this review. In total, 75% of studies were conducted in Asia, with China (13/40) and South Korea (9/40) contributing most to these studies. The included studies mainly relied on unstructured data (i.e., imaging) as the original input. Five studies^{18,19,24,43,49} combined the original image input with clinical covariates, which were structured data. Some studies used images as raw input; a series of structured feature data was obtained for DL.

Overall, more complex DL algorithms have been proposed to achieve better performance through the use of different hyperparameter tuning methods. In recent years, data pre-processing and data augmentation techniques have been applied to enhance the reliability of the results and reduce the risk of overfitting. We observed a trend in research moving from single-center studies to multi-center studies, which indicates stronger generalization capabilities and potential clinical applications.

Twenty studies^{18,19,24,26–29,34,42,44,50–52} focused on osteoporosis screening and classification. The convolutional neural network (CNN) was widely used in all of these studies. Osteoporosis was identified based on opportunistic imaging from CT, X-ray, DPR, and MRI, although MRI is not recommended for opportunistic osteoporosis screening. The incorporation of clinical covariates into the model construction process has resulted in improved model construction, with an improvement in the AUC by 2%–4%. Furthermore, the use of pre-trained models, pre-data augmentation, and image standardization has enabled studies with relatively limited data to achieve good performance. Visualization of models was performed using gradient-weighted class activation mapping (Grad-CAM). One study showed a perfect match with bone fracture,⁵¹ but another study did not.¹⁸

Among the studies on osteoporosis, it is worth noting that several studies^14,35,38 used U-Net for image segmentation. U-Net is a well-known CNN semantic segmentation algorithm. Liu et al.¹⁴ proposed the use of U-Net for diagnosing osteoporosis. They compared the performance of U-Net with a back propagation network and support vector machine (SVM), and U-Net demonstrated the best performance. The results indicated that the U-Net algorithm could better distinguish between normal bone mass and osteoporosis, achieving an AUC of 0.872. However, U-Net had poorer recognition of lower bone mass and osteoporosis. The proposed U-Net model effectively dealt with image interference, thereby improving accuracy.

U-Net is commonly included in automatic systems for medical image segmentation. However, in some cases, the segmentation performance falls below expectations. This can be attributed to factors such as the complex structures of vertebral bodies, label noise, semantic segmentation, and variations in CT scanners. Pickhardt et al.³⁸ developed TernausNet, a variant of U-Net with a VGG-11 encoder. Among the CNN-based studies, the development of an automatic system for fully automated detection of osteoporosis has potential clinical application and commercial value. Zhang et al.³⁵ proposed an end-to-end multitask joint learning framework that integrated positioning, segmentation, and classification. In this framework, U-Net has a prerequisite role. The framework achieved the best performance for diagnosing osteoporosis (ACC: 0.957, sensitivity: 0.962, specificity: 0.922, F1-score: 0.975) with a learning rate of 10e-3, surpassing state-of-the-art DL models.

Transfer learning has gained popularity in the field of DL owing to its ability to leverage knowledge from previous tasks, improving generalization and achieving more accurate outcomes with limited data input. Wani et al.¹⁶ used transfer learning to develop four CNN networks: AlexNet, ResNet, VggNet-16, and VggNet-19. To mitigate the risk of overfitting, data augmentation techniques were applied, considering the limited amount of data available. The results showed that the pre-trained CNN networks consistently outperformed the normal CNN networks in all cases. Notably, the pre-trained AlexNet achieved the best classification performance with an accuracy of 91%, surpassing the performance of the normal AlexNet by 15.3%. The classification accuracy for ResNet, VggNet-16, and VggNet-19 was 74.3%, 78.9%, and 73.68% for the normal CNN and 86.4%, 86.3%, and 84.2% for the pre-trained CNN, respectively.

In another study by Lee et al.,⁴² transfer learning and fine-tuning were used to build transfer learning models based on VGG-16 (VGG-16_TF), and further fine-tuning was performed with the transfer learning model (VGG-16_TF_FT). To prevent overfitting, a five-fold cross-validation approach was followed. The VGG-16_TF and VGG-16_TF_FT models outperformed the VGG-16 and CNN-3 models, achieving an AUC of 0.858 and 0.782, respectively. These results highlight the important contribution of transfer learning and fine-tuning in improving the performance of DL models. By combining a pre-trained CNN network with fine-tuning, the screening performance using small image datasets is comparable to that of previous studies.

Dzierzak et al.²⁸ used six pre-trained deep convolutional neural network (DCNN) architectures for model construction. Among these models, the VGG-16 model demonstrated superior performance, achieving an AUC of 0.985 and an ACC of 95%. Additionally, three models, (VGG-16, VGG-19, and InceptionResNetV2) achieved an ACC of over 90%. The remarkable improvement in performance can be attributed to the application of pre-training and fine-tuning techniques, which have proven to be highly effective despite the limited amount of available data. Notably, with its relatively shallow architecture consisting of 19 layers, VGG-16 emerged as the most effective model for handling small datasets.

In a study conducted by Lee et al.,⁵³ a combination of VGG-16 and random forest classifier (RFC) yielded the best overall performance, with an AUC of 0.74, ACC of 0.71, sensitivity of 0.81, and specificity of 0.6. These findings further support the notion that VGG-16 is a popular and influential model in the field. With its 13 convolutional layers and three fully connected layers, VGG-16 offers a compact architecture. The use of small 3 × 3 convolutional layer filters allows for a reduction in parameters, resulting in decreased computational effort and shorter model-construction time.

In 2020, Yamamoto et al.¹⁸ developed an ensemble model by incorporating clinical covariates with hip radiographic images. This combined model outperformed those using only radiographic images or clinical covariates. The ensemble EfficientNet-b3 network was the most successful among the five ensemble models, surpassing the EfficientNet-b3 network that relied solely on images. Those authors used guided Grad-CAM for visualization of model classification. However, the heatmap was not accurately distributed to the proximal femoral trabeculae but rather to the exterior femoral bone, indicating that osteoporosis location was not adequately addressed. However, in the study by Jang et al.,⁵¹ the visualization results were closely aligned with the proximal femur structure. In the following year, Yamamoto et al.¹⁹ pursued a related study, which combined image features with patient variables and statistically analyzed the differentiation caused by the inclusion of these variables. Whereas the study showed performance improvements, some CNN models did not show similar improvements. The effect size, a measure of the true effect size of an experiment or the strength of the association of variables, was evaluated in the model, yielding a value of 0.871, indicating a large effect size. Osteoporosis screening was also conducted using DPR and six ensemble CNN models, including EfficientNet-b0, b3, and b7 and ResNet-18, 50, and 152. The EfficientNet-b7 and ResNet-152 models demonstrated comparable performance. However, the visualization produced by Grad-CAM varied, with ResNet focusing on the cortical bone of the lower edge of the mandible, and EfficientNet extending its focus to the area above the cortical bone.

Hsieh and colleagues⁵³ reported that image-only models of VGG-16 and ResNet-34 performed well, and the addition of sex and age did not enhance model performance. The extent to which clinical characteristics can aid in performance improvement remains a topic for future investigation. Some studies have also explored the impact of CT channel on ML outcomes, in addition to considering clinical covariates.

Multi-center studies conducted by both Mao et al.²⁴ and Zhang et al.¹⁵ aimed to explore performance improvements associated with the incorporation of CT channel and clinical covariates. The combination of anteroposterior and lateral channels was superior to each single channel. Mao’s study achieved higher AUC and sensitivity in diagnosing osteoporosis compared with Zhang’s study, potentially owing to data imbalance and the absence of cortical bone in X-ray images. Both studies highlighted potential biases in BMD measurements using DXA and the CNN models owing to factors such as aortic sclerosis, bowel gas, and osteophytic spurs. Tariq et al.³⁴ developed a fusion model, which combined coronal and sagittal images from enhanced CT scans with basic patient information to screen for low bone density. This model achieved a high area under the receiver operating characteristic curve of 0.86, indicating its potential in clinical application. The model performed well in a prospective group with and without the use of contrast material and demonstrated limited disparities when trained and validated on four sites with different CT scanners.

In their research, Chen et al.²⁷ developed a two-level classifier that combines ML techniques with radiomics texture analysis. The proposed classifier used SVM binary classifiers, which were trained using nested cross-validation. This approach involved five outer and inner iterations to optimize the hyperparameters. The vertebral body segmentation model achieved a Sorenson–Dice coefficient of 0.87, which is consistent with the findings of Pan et al.³⁰ Furthermore, the two-level classifier demonstrated excellent performance, with an AUC of 0.98, ACC of 0.94, sensitivity of 0.95, and specificity of 0.93.

Jang et al.⁵¹ addressed the limitations of CNNs, which primarily focus on local features rather than global features, by developing a nonlocal neural network (NLNN) based on the VGG-16 architecture. The NLNN model demonstrated promising results, achieving an overall ACC of 81.2%, sensitivity of 91.1%, and specificity of 68.9%. Notably, the Grad-CAM visualization technique revealed that the NLNN model accurately captured the bone structure, further validating its effectiveness in analyzing global features.

A total of 13 studies^{17,21,23,26,30–33,36,39,53,54} aimed to measure BMD. Among these, nine studies focused on predicting vertebral BMD,^{17,26,30–33,39,40,53} three studies assessed hip BMD,^21,23,36 and only one study examined both vertebral and hip BMD.⁵⁴ It is worth noting that whereas most studies used quantitative assessment methods, one study³³ conducted qualitative assessment.

Pan et al.³⁰ used U-Net for fully automated segmentation and labeling of vertebral bodies, which is a crucial step in BMD prediction. The developed system demonstrated a strong correlation with BMD measurements, with a coefficient of determination (r²) ranging from 0.964 to 0.968. The system also exhibited high diagnostic performance for osteoporosis, with an AUC of 0.927, sensitivity of 85.71%, and specificity of 99.68%.

In another study by Fang et al.,²⁹ U-Net was applied for automated segmentation of vertebral bodies, and DenseNet-121 was used to calculate BMD. The results from three independent testing cohorts showed that the average Dice similarity coefficients for L1–L4 vertebral bodies were consistently near or above 0.8. Moreover, the BMD values calculated using the CNN exhibited a strong correlation with BMD derived from DXA, with r > 0.98.

Breit et al.²⁶ used a deep image-to-image network (DI2IN) in their study. The proposed DI2IN is a multi-layer CNN that incorporates wavelet features and geometric constraints (AdaBoost) to accurately segment the vertebral bodies of the thoracic spine. Cylindrical volumes of interest (VOIs) were placed at the center of the segmented vertebral bodies. Mean attenuation, measured in Hounsfield units (HU), was calculated to assess bone density. The study showed that the mean HU values for T1–12 were significantly higher in individuals with normal bone density compared with those who had osteopenia or osteoporosis. Furthermore, a statistically significant correlation was observed between the mean HU values of T1–12 and absolute bone density. The mean HU values demonstrated a higher ACC (0.75), AUC (0.8), Youden index (0.54), sensitivity (0.93), and specificity (0.61) compared with clinical reports. However, Kang et al.⁴⁰ found that not only the vertebral body but also other bone areas contributed to BMD estimation. Therefore, putting VOIs in the center of the vertebral bodies may not enhance performance, although the center of the segmented vertebral bodies received greater attention in the Gard-CAM of that study. Similarly, Yasaka et al.³⁹ used supervised learning to develop a CNN model using axial CT images as input. The CNN model estimated BMD with AUCs of 0.965 and 0.970 for the internal and external validation datasets, respectively, outperforming BMD measurements derived from CT values of the lumbar vertebrae. However, a limitation of that study was the use of unenhanced CT. Sollmann et al.³² established an automated CNN model for segmentation, labeling, and extraction of vertebral bone mineral density (vBMD) from QCT scans. The study showed that vBMD derived from QCT (vBMD_QCT) exhibited a stronger correlation with opportunistic CNN-based vBMD derived from routine CT data (vBMD_OPP) than with noncalibrated HU values. Discrimination between patients with and without osteoporotic vertebral fractures was slightly better when using vBMD_OPP compared with vBMD_QCT or noncalibrated HU values. Considering the scanner-specific nature and susceptibility to contrast media, noncalibrated HU values were not recommended in this study. To address the bias introduced by contrast material, Rühling et al.³¹ used three artificial neural network (ANN) models. Among these models, the 2D anatomy-guided DenseNet model aimed to minimize contrast medium-induced errors and achieved an ACC of 0.983, sensitivity of 0.983, and specificity of 0.991. Additionally, the 2D anatomy-guided DenseNet model, which used combined axial images from T8–L2, demonstrated excellent generalizability, performing well on the public dataset.

Tang et al.³³ developed a qualitative assessment method for diagnosing osteoporosis. The process involved extracting 2D CT image slices of the pedicle level in the lumbar vertebrae (L1) from 3D CT images. The lumbar vertebrae were then identified and segmented using MS-Net. Subsequently, qualitative classification of the lumbar vertebrae was performed using BMDC-net. The study results demonstrated high precision of 80.57%, 66.21%, and 82.55% for predicting normal bone mass, low bone mass, and osteoporosis, respectively. However, it is worth noting that the recognition accuracy for low bone mass was significantly lower compared with that for normal bone mass and osteoporosis, indicating a challenge that still needs to be addressed. To visually represent the distribution of probabilities for the three categories, a pie chart was created, providing an overview of the bone condition. However, in cases where the probabilities of two categories are similar, providing a definitive clinical diagnosis becomes challenging. This highlights the need for further research and improvement in the diagnostic accuracy of the qualitative assessment method.

In the multi-center study by Sato et al.,¹⁷ the proposed method uses chest X-ray, age, and sex to predict BMD and T-score. The predictive performances for true BMD are as follows: femoral BMD (r: 0.75, r²: 0.54) and lumbar spine BMD (r: 0.63, r²: 0.40). The predicted femoral BMD demonstrates an excellent correlation with BMD measured by DXA. This indicates that the model can accurately estimate the BMD values based on the provided inputs. Furthermore, the prediction for T-score, which is used to diagnose osteoporosis (T-score below −2.5), exhibits strong performance with an AUC of 0.8. This suggests that the model can effectively identify individuals with osteoporosis based on their T-scores. In addition to the imaging data, clinical features such as age, height, and weight are normalized and included as input data for training an ensemble ANN. The model has achieved excellent performance in assessing BMD, with a high r value of 0.8075 (p < 0.0001) when compared with DXA measurements. This indicates strong agreement between the predicted BMD values and BMD values obtained using DXA.

Two included articles concerned the risk prediction of osteoporotic fractures. Du et al.²² first segmented the femur region of interest and the hard bone edge. They then conducted a quantitative analysis of the femoral trabecular score and femoral neck strength to assess the quality of the femoral neck. A risk prediction ML-based model of femoral neck osteoporosis in older adults was established, which makes sense in auxiliary diagnosis. In the test dataset, the neural network model showed the highest ACC of 95.83% and a recall of 100%.

Kong et al.⁴⁹ used DL techniques to demonstrate superior performance compared with traditional fracture risk assessment tools such as the FRAX and Cox proportional hazard (CoxPH) models. The DL model, specifically the DeepSurv model, achieved a higher C-index value of 0.612 compared with FRAX (0.547) and CoxPH (0.594). Notably, the authors found that using multiple slices in the spine region with DL yielded even better results than using a single slice. This highlights the importance of considering a broader range of information when using DL algorithms for fracture risk assessment. The findings of the above study suggest that DL models have the potential to achieve performance that is comparable to that of FRAX, even when using only a single X-ray image as input. This indicates the clinical potential of DL in improving fracture risk assessment and potentially enhancing patient care.

Another five studies were aimed at detecting osteoporotic fractures. Osteoporotic vertebral fracture (OVF) is the most common type of osteoporotic fracture. Tomita⁴¹ and colleagues used CNN for the extraction of features and achieved an ACC of 89.2% in detecting OVF. Xiao et al.²⁵ designed automated vertebral fracture detection software based on lateral chest radiographs of older women. In a study by Yabu,⁴⁶ data augmentation techniques and ensemble learning were applied to distinguish fresh OVFs from MRI images. However, that study did not use any images from normal individuals or patients with pathological fractures, a notable limitation.

VFA via DXA can be used to detect a vertebral fracture. VFA can be conducted using both dual-energy (DE) and single-energy (SE) mode scanners. Monchka et al.⁴⁷ and Derkatch et al.⁴⁸ used DL to detect osteoporotic fracture from VFA images. Derkatch et al. trained and tested their model on DE only. Monchka et al. constructed their model on the basis of a DE training set, an SE training set, and a composite dataset comprising both DE and SE VFAs. The authors found that CNN models are highly sensitive to the scan mode. The model trained with DE images performed poorly in the SE test set. Training on the composite dataset allows CNN to generalize to both scan modes and enhance the performance in DE VFAs. The performance of this CNN is on par with that of Derkatch et al., with a similar AUC, slightly lower sensitivity, and higher specificity. Thus, a composite training dataset including not only DE VFA images but also SE images is highly recommended. Notably, the study by Monchka has some limitations in that they did not try to identify external validation sets. However, there are few studies toward this direction. Further research is needed to reveal the best solution.