Artificial Intelligence in Musculoskeletal Radiology: Past, Present, and Future
How to cite this article: Pankhania M. Artificial intelligence in musculoskeletal radiology: Past, present, and future. Indian J Musculoskelet Radiol 2020;2(2):89-96.
Musculoskeletal radiology is an important tool for the diagnosis of muscle damage, bone fractures, bone tumors, musculoskeletal infection, and other diseases. However, all currently used radiological techniques, including radiography, ultrasonography, computed tomography, and magnetic resonance imaging are associated with their own challenges. With its ability to address these challenges, artificial intelligence (AI) holds the promise to transform a musculoskeletal radiologist’s job in several areas. In the past, AI-based approaches in musculoskeletal radiology were primarily used for measuring bone mineral density or identifying bone tumors. However, recent studies have expanded the application of AI in several other areas, such as image segmentation, resolution enhancement, and fracture identification as well automatic diagnosis of other forms of musculoskeletal damage. This review article discusses numerous older as well as more recent studies to highlight how the development and application of AI-based approaches have evolved in the field of musculoskeletal radiology and how the applicability of these approaches may be improved in the future.
Musculoskeletal diseases and damage conditions present a plethora of complex possibilities for radiologists and orthopedics alike. In the clinic, radiology is a crucial tool used for the diagnosis (that is, identification) of musculoskeletal damage, fractures, bone tumors, musculoskeletal infection, and other diseases.[1-3] Several techniques are available at the clinician’s disposal through the radiology department including X-ray, computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound. While these techniques have their own strengths, they are not without their weaknesses. For example, MRI sections or slices with high thickness (2.5 mm–4 mm) are associated with partial volume effects which can lead to missed subtle lesions. Moreover, thick slices in MRI imaging can lead to difficulties in assessing the images of tissues that do not strictly lie in a planar orientation including cartilage and ligament tissues. At the same time, radiologists also face challenges when X-ray images have an overlap of anatomical structures. Further, X-rays taken across different radiology visits by a patient may be inconsistent. Similarly, it is difficult to achieve segmentation (or identification of sets of pixels that correspond to curves, lines, or other feature boundaries) of different muscles in CT images. Meanwhile, diagnostic variability from one clinician or radiologist to another is one of the major issues with using ultrasound for diagnosing musculoskeletal problems. These challenges make the diagnoses of musculoskeletal damage, such as bone fractures, difficult. In recent times, numerous studies have focused on addressing the above-mentioned challenges with the help of artificial intelligence (AI) and machine learning (ML).
AI represents an evolving class of computer algorithms and methods that emulate various aspects of human intelligence. Within AI, ML refers to a method that trains computer algorithms to perform a variety of tasks and improves the efficacy and accuracy of these tasks through statistical approaches [Figure 1]. One of the most commonly used ML approaches is deep learning (DL), which uses deep neural networks (DNNs) to process large sets of data. Indeed, the fast-paced advancements in AI and ML have accelerated the evolution of health care not only in orthopedics but also in many other fields.[8,9] DL as well as other ML approaches have been found to be useful in musculoskeletal radiology. This review summarizes the state of the art of AI-based approaches in musculoskeletal radiology. Particular focus has been placed on how the use of AI in musculoskeletal radiology has evolved overtime. Further, the need and application of AI in various musculoskeletal radiological approaches, including X-ray, CT, MRI, and ultrasonography, have been discussed. The limitations of these techniques and the manner in which AIand ML-based are being used to overcome those limitations have also been discussed in detail.
Over the past decade, AI has been developed to improve the interpretation of a wide variety of musculoskeletal radiological tests. However, before that, AI-based models had primarily been developed for applications such as the identification of bone tumors, assessment of bone mineral density, and segregation of femoral and tibial features, among other similar tasks.[10-14]
As early as 1963, Lodwick et al. developed a program which had a greater than 80% accuracy for predicting five types of bone tumors through X-ray images, namely, chondroblastoma (87.5%), Ewing’s sarcoma (81.8%), fibrosarcoma (81.8%), giant cell tumor (100%), and parosteal sarcoma (100.3%). Later, Mundinger et al. used AI to analyze the texture of bone CT images for the diagnosis of osteoporosis. Similarly, Geraets et al. used AI to predict bone mineral density from X-ray images.
Artificial neural networks applied to musculoskeletal radiology
Today, the importance of adapting AI and ML into day-to-day use in musculoskeletal radiology is being noted by more and more scholars. Recent studies have tried to use AI and ML to transform the field of musculoskeletal radiology by enhancing the quality of assessment that can be derived from radiological techniques. DNNs, radial basis function neural networks, and generative adversarial ML are among the most commonly used AI-related approaches in musculoskeletal radiology. DNNs, such as convolutional neural networks (CNNs), constitute a specific form of DL AI algorithms that allow the compilation of data from multiple characteristics of the same input dataset provided to them. A subset of the available data is used to train the AI algorithms, which can then generalize the information and use it to analyze and interpret the information contained in other similar data sets. CNNs are best suited for working with imaging data as they are designed to preserve spatial relationships. A grid-based input image is provided to a CNN, within which a filter is applied to all locations, giving rise to a convolutional layer of “artificial neurons” [Figures 2 and 3]. Features identified through the filter are pooled and further inputted into a new convolutional layer and so on. Each new layer focuses on a subset of the area in the previous layer. In the end, a fully connected layer is formed, which connects all neurons of the most recent layer to all neurons of the previous layer. Finally, the predicted output(s) are presented based on the prior training of the algorithm, along with the probability of each predicted output.
Meanwhile, another class of neural networks called the radial basis function neural networks has been found to be better in performance and faster in training compared to many other neural networks. Another commonly used AI approach is generative adversarial learning. Generative adversarial networks (GANs) represent a class of DL methods that generate new data instead of processing existing information. GANs combine a generative network and a discriminative network to enhance the performance of a DL model. Neural networks such as DNNs may be used as discriminative networks to train GANs. In the field of musculoskeletal imaging, the use of GANs is still very limited. However, some studies have utilized GANs to analyze musculoskeletal MRI scans and radiographs. Using the above-mentioned AI and ML techniques, numerous studies have improved on the pre-processing, acquisition and analysis of radiographs, CT scans, MRI scans, and ultrasonographs [Figure 4 and Table 1]. Some of these studies have been discussed below.
|S. No.||Imaging modality||Musculoskeletal tissue||AI-based approach used||Size of imaging data set used||Performance||Reference|
|1.||Ultrasonography||Muscle, tendon||CNN||4 images from 1 cadaver||75% sensitivity, 81% specificity||Jabbar et al.|
|2.||Ultrasonography||Supraspinatus (rotator cuff) tendon||Semiautomatic CAD program||99 images from 93 adult patients (43 men, 50 women)||Accuracy of diagnosing rotator cuff inflammation – 88.4%, tears – 92.3%, calcific tendonitis – 83.3%||Chang et al.|
|3.||X-ray||Bone||LBF-SVM||36,770 images (MURA)||35% sensitivity, 62% accuracy||Mall et al.|
|4.||X-ray||Bone||Stacked Random Forest Classifier||145 images, with 1900 healthy patches and 100 fractured patches per image||81.2% accuracy, 24.7% precision||Cao et al.|
|5.||CT||Hip and thigh muscles||CNN and Bayesian U-Net||20 CT volumes in the training dataset, 18 CT volumes in the testing dataset||X||Hiasa et al.|
|7.||Ultrasonography||Supraspinatus tendon||Curvelet-based automatic segmentation||116 images in the testing dataset||For full thickness tears:95.6% sensitivity, 95.0% specificity; for partial thickness tears: 94.0% sensitivity, 93.6% specificity||Gupta et al.|
|8.||X-ray||Arm skeletal muscles||Dilated DenseNet||40,895 radiographs from 12,251 patients||Accuracy for fingers: 41.7%, forearm: 80.2%, hand: 92.7%, wrist: 93.1%, shoulder: 86.4%||Li et al.|
|9.||X-ray||Arm skeletal muscles||CNN (Keras Deep Learning Framework)||36,808 samples for training, 3197 samples for validation||73% accuracy||Wang|
|10.||MRI||Knee||3D CNN (DeepResolve)||Training dataset of 124 patients, testing dataset of 17 patients||Achieved “diagnostic quality” super-resolution images||Chaudhari
Many recent studies have used the musculoskeletal radiographs dataset (MURA) to train and test different AI models to improve on image pre-processing. MURA is an open-source X-ray image dataset that makes both normal and abnormal musculoskeletal images available to researchers and radiologists for studies. Mall et al. used ML algorithms to classify bone radiographs from MURA as fractured or intact. The four ML algorithms used in this study performed image pre-processing tasks such as color transformation and contrast enhancement before extracting as many as 12 relevant gray level cooccurrence matrix features, including autocorrelation, contrast, dissimilarity, and energy from each image. These textural features were subsequently used to classify the images into “fracture” and “non-fracture” groups. About 67% of the data was used to train the algorithm while the rest was used for testing. The radial basis function support vector machine (LBF SVM) algorithm achieved the highest accuracy (62%) among the four ML algorithms tested.
Galbusera et al. conducted a study on applying GANs to preprocess spine MR images. They evaluated the efficacy of GANs in detecting defects such as herniation, degenerative disc diseases, and end-plate changes from MRI data. Moreover, radiographs of the lumbar spine were used to identify abnormal numbering of the vertebra and vertebral body compression fractures. Ultimately, the GANs used in the study were able to make low-resolution images sharper and more informative, highlighting the importance of AI-based approaches in aiding musculoskeletal radiologists in their evaluation of spine imaging data.
MRI is preferred when the application necessitates superior soft-tissue contrast. Superresolution musculoskeletal MRI can help generate thin, high-resolution image slices from thicker slices of complex body regions such as the knee. This can also help accelerate the process of MRI data acquisition as data, that is, inferior in amount or quality can later be converted to high-resolution images using AI methods. In one study, Chaudhari et al. used CNNs to develop a superresolution technique to improve the usability of thick slice MRI data. Chaudhari et al. developed a 3D CNN called DeepResolve to create high-resolution thin slice resolved from a low-resolution thick slice. DeepResolve includes 0.7 mm thin slices from knee images obtained using Double Echo in Steady State. The training dataset for this model came from 124 patients from the osteoarthritis initiative. The model was subsequently tested on data from 17 patients. Based on qualitative assessment of the images by two radiologists along with quantitative assessment, it was found that DeepResolve could generate superresolution knee MR images of the required diagnostic quality or better. Other studies that have focused on improving MRI image acquisition have used AI to help decide if intravenous contrast is needed in musculoskeletal MRI. Similarly, there are studies that have shown the use of AI to reduce the amount of radiation used while acquiring CT data from patients.
Ultrasonographic image acquisition has also been targeted by some AI-based studies, particularly because commonly used ultrasonographic techniques are limited by a narrow field of view. Jabbar et al. suggested a way to overcome this issue. The idea was to obtain panoramic ultrasound images by stitching together individual images, thereby helping to visualize wider tissues through a single panoramic image. Quantifying geometric parameters using panoramic ultrasound images require image processing, including image enhancement, edge detection, and segmentation. The previous studies have noted that the edges of muscle tissue are not preserved upon image segmentation. Jabbar et al. addressed this issue by designing a CNN to identify the pixels in ultrasound images that constitute the edge of the musculoskeletal tissue [Table 1].
Many AI-based studies have focused on improving image analysis to help radiologists with the interpretation of imaging data. To identify the instances of fracture in a set of X-ray images, Cao et al. developed an ML algorithm using the stacked random forest multilayer classifier. The algorithm collates and fuses different features identified from the training dataset of fractured and healthy bone X-ray images. After training, the algorithm can generate confidence score maps for areas in the X-ray images to indicate which regions have a greater likelihood of having a fracture. This study used 80% of the imaging dataset for training the algorithm and the remaining 20% for testing. In another study, Wang achieved a fracture identification accuracy of 73% using a Keras DL framework, which is a CNN based approach. In another study, Li were able to detect skeletal muscle damage using an adversarial learning assisted deep CNN (DCNN) on radiographs. They used MURA images to conduct their study on various arm musculoskeletal areas such as fingers, shoulder, elbow, and forearm. This model achieved an overall accuracy of 82.1% in the identification of damaged muscular regions. Yet, another study found that DL resulted in human radiologist-like fracture and body part identification in X-ray images.
Some other X-ray-based studies have explored the use of AI technologies such as artificial neural networks, SVMs, and regression-based methods in determining bone age.[2,25,26] These include studies conducted to assess differences in skeletal maturity based on gender, race, and age.[25,26] Other studies have used Bayesian models to calculate the probability of different diagnoses based on radiographical characteristics.[3,27] For example, Do et al. conducted a study that tested radiological bone tumor data against 29 potential diagnoses, predicting the correct diagnosis in only 44% of the cases. This study used a minimum of five image samples per diagnosis to train their model (as a comparison, ML-based methods always require larger imaging datasets to train the AI models). In another study, Hiasa et al. used Bayesian DL and CNNs to accelerate the segmentation of muscles in CT imaging and improve its accuracy. By demonstrating the segmentation of 19 muscles in the hip and thigh region, this study showed that the use of CNNs can significantly improve the accuracy of muscular segmentation. In another MRI-based study, Irmakci et al. used DL algorithms on multiview MRI scans to identify meniscus and anterior cruciate ligament tears as well as other abnormalities in the knee. In addition to X-ray, CT, and MRI, ultrasonography is another common radiological technique that is used for diagnosing a wide variety of joint and soft-tissue disorders. It holds the advantage of having a short diagnosis time as well as easy applicability for a host of patients. Musculoskeletal soft-tissue damage is identified as disorganized and hyperechoic patterns in ultrasonography images. These differences in the visual appearance of damaged musculoskeletal tissue can also be quantified in the form of tissue thickness, cross-sectional area, and mean echogenicity. However, the identification of visual differences is highly dependent on the experience level of the radiologist, and even with the same experience level, it can still vary from one expert to another.[6,28] Moreover, segmentation of musculoskeletal tissues such as tendons can be difficult in ultrasound images. Some AI studies have attempted to overcome these challenges by improving on image analysis.
To address the challenge of radiologist-to-radiologist difference in interpretation, Gupta et al. developed an AI-based automatic segmentation approach using ultrasonographs of the rotator cuff supraspinatus tendon in the shoulder. In another study on the supraspinatus tendon, Chang et al. developed a computer-aided diagnosis system to improve the diagnosis of rotator cuff lesions and their classification into relevant pathological categories such as calcific tendonitis, inflammation, or thickness tears [Table 1]. They recommended that their model could be used as a suggestive diagnostic tool by radiologists whose diagnostic accuracy may be lower than an orthopedic surgeon with relevant experience in shoulder surgeries.
Natural language processing (NLP) in musculoskeletal radiology
In addition to the literature discussed above, some studies provide a unique perspective on AI applications in musculoskeletal radiology using AI models indirectly on textual reports and protocols created by radiologists instead of directly on radiological images.[7,29] For example, Wang et al. developed a NLP algorithm that successfully extracted skeletal fracture related information from radiology reports of osteoporosis patients. Annarumma et al. developed an NLP processing system to classify adult chest X-ray radiology reports as “critical,” “urgent,” “non-urgent,” and “normal.” This classification was followed by the use of a CNN algorithm to analyze the chest X-ray images and further prioritize them based on the abnormalities detected. This way, AI-based NLP processing in concert with image analysis could be used for triage (that is, classification of patients based on the urgency of treatment so as to optimize the functioning of a medical facility).
Some other studies in the area of NLP have used AI methods to determine which musculoskeletal imaging protocols should be used for MRI (regular or tumor oriented) and whether or not intravenous contrast is needed when performing MRI.[7,31,32]
Based on the studies discussed in this section, it is clear that all commonly used modalities for musculoskeletal radiology can be improved in terms of accuracy, speed, and resolution with the help of AI-based techniques, which is the foremost advantage of using AI in this field [Table 1]. By improving image pre-processing, image acquisition, image analysis, as well as interpretation of textual radiological reports, AI-based methods can either assist radiologists in their interpretation of radiological data or perform automatic diagnoses to varying degrees of accuracy.
As indicated by the volume of research focused on the use of AI in musculoskeletal radiology, the future of this collaboration can be expected to be fruitful. Some orthopedic departments in hospitals across the world, including the Cleveland Clinic, have already started laying special focus on AI for musculoskeletal-specific applications by setting up ML laboratories. The hope is that AI can help provide personalized musculoskeletal health care to patients. AI-based approaches can also help to expand the current applications of musculoskeletal radiology. For example, in the future, cognitive maps developed using AI can help radiologists accurately detect and diagnose malignant bone tumors such as osteosarcoma and Ewing’s sarcoma in children. AI-based image analysis techniques can also help quickly triage patients based on the severity of their musculoskeletal damage.
To ensure that AI and ML algorithms can be efficiently trained, it is also important to overcome the dearth of well-annotated musculoskeletal image data sets. The X-ray image database MURA (part of a competition organized by Stanford ML Group; https://stanfordmlgroup.github.io/competitions/mura/) and the MRI scan database FastMRI (part of a competition organized by Facebook AI Research and NYU Langone Health; https://fastmri.org/) are proving to be fruitful steps toward providing well-annotated musculoskeletal imaging data to AI specialists.[2,19,34,35] The FastMRI competition can also open new avenues for enhancing the speed of MRI data acquisition in the future.
Orthopedics is a major field of health care that stands to benefit immensely from AI and ML, particularly in the area of musculoskeletal radiology. This review article discussed many of the studies that highlight significant achievements in the development of AI-based approaches that supplement the radiological assessment of musculoskeletal damage or dysfunction. For AI and ML to be incorporated into regular radiological analyses, it is important that AI-based approaches are able to generate an impeccable trust in radiologists, clinicians, hospital managements, and patients alike. The approaches discussed in this article suggest that this is an achievable goal. In the future, it is important to increase the size and quality of the radiological datasets that are used to train AI models for diagnosing musculoskeletal problems. This can help expand the applicability and accuracy of radiological techniques for the diagnosis of musculoskeletal diseases and injuries.
Declaration of patient consent
Patient’s consent not required as there are no patients in this study.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
- Radial basis function neural networks: A review. Comput Rev J. 2018;1:52-74.
- [Google Scholar]
- MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs, arXiv No 171206957. 2017
- [Google Scholar]
- fastMRI: An Open Dataset and Benchmarks for Accelerated MRI, arXivNo 181108839. 2018
- [Google Scholar]