No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Learning to deep learning: statistics and a paradigm test in selecting a UNet architecture to enhance MRI
Abstract
This study aims to assess the statistical significance of training parameters in 240 dense UNets (DUNets) used for enhancing low Signal-to-Noise Ratio (SNR) and undersampled MRI in various acquisition protocols. The objective is to determine the validity of differences between different DUNet configurations and their impact on image quality metrics.
To achieve this, we trained all DUNets using the same learning rate and number of epochs, with variations in 5 acquisition protocols, 24 loss function weightings, and 2 ground truths. We calculated evaluation metrics for two metric regions of interest (ROI). We employed both Analysis of Variance (ANOVA) and Mixed Effects Model (MEM) to assess the statistical significance of the independent parameters, aiming to compare their efficacy in revealing differences and interactions among fixed parameters.
ANOVA analysis showed that, except for the acquisition protocol, fixed variables were statistically insignificant. In contrast, MEM analysis revealed that all fixed parameters and their interactions held statistical significance. This emphasizes the need for advanced statistical analysis in comparative studies, where MEM can uncover finer distinctions often overlooked by ANOVA.
These findings highlight the importance of utilizing appropriate statistical analysis when comparing different deep learning models. Additionally, the surprising effectiveness of the UNet architecture in enhancing various acquisition protocols underscores the potential for developing improved methods for characterizing and training deep learning models. This study serves as a stepping stone toward enhancing the transparency and comparability of deep learning techniques for medical imaging applications.
... Beyond human perceptual studies, advanced evaluation methods, such as eye-tracking and physiological metrics like heart rate variability, will be explored to gain deeper insights into user responses to transformed scenes. Statistical analyses will also be performed to validate the significance of observed improvements and establish the practical relevance of the results such as done by Sharma et al. [43], [44] and Javadi et al. [45]. By addressing these aspects, future work aims to refine the proposed pipeline further and extend its applicability to a broader range of real-world scenarios and practical use cases. ...
Recent advances in GAN-based architectures have led to innovative methods for image transformation. The lack of diverse environmental data, such as different lighting conditions and seasons in public data, prevents researchers from effectively studying the difference in driver and road user behaviour under varying conditions. This study introduces a deep learning pipeline that combines CycleGAN-turbo and Real-ESRGAN to improve video transformations of the traffic scene. Evaluated using dashcam videos from London, Hong Kong, and Los Angeles, our pipeline shows a 7.97% improvement in T-SIMM for temporal consistency compared to CycleGAN-turbo for night-today transformation for Hong Kong. PSNR and VPQ scores are comparable, but the pipeline performs better in DINO structure similarity and KL divergence, with up to 153.49% better structural fidelity in Hong Kong compared to Pix2Pix and 107.32% better compared to ToDayGAN. This approach demonstrates better realism and temporal coherence in day-tonight , night-today , and clear-to-rainy transitions.
... Deep learning (DL) has emerged as a promising strategy for reducing noise and enhancing image quality in LFMRI scans [12][13][14][15][16][17][18][19][20][21][22]. A fundamental aspect of this progress is the development of convolutional neural networks (CNNs) that can be trained in three dimensions, enhancing their applicability and effectiveness in analyzing volumetric data. ...
... These methods follow DCNN models for essential feature learning and preserving the corresponding clean images. DL models can automatically learn by updating the model weight parameters and extracting image features to restore image quality (Sharma et al 2023). For this purpose, DL models were trained on a complete MRI dataset using high-power graphics processing units (GPUs) to achieve optimal performance (Khural et al 2022). ...
Magnetic resonance imaging (MRI) provides detailed structural information of the internal body organs and soft tissue regions of a patient in clinical diagnosis for disease detection, localization, and progress monitoring. MRI scanner hardware manufacturers incorporate various post-acquisition image-processing techniques into the scanner’s computer software tools for different post-processing tasks. These tools provide a final image of adequate quality and essential features for accurate clinical reporting and predictive interpretation for better treatment planning. Different post-acquisition image-processing tasks for MRI quality enhancement include noise removal, motion artifact reduction, magnetic bias field correction, and eddy electric current effect removal. Recently, deep learning (DL) methods have shown great success in many research fields, including image and video applications. DL-based data-driven feature-learning approaches have great potential for MR image denoising and image-quality-degrading artifact correction. Recent studies have demonstrated significant improvements in image-analysis tasks using DL-based convolutional neural network techniques. The promising capabilities and performance of DL techniques in various problem-solving domains have motivated researchers to adapt DL methods to medical image analysis and quality enhancement tasks. This paper presents a comprehensive review of DL-based state-of-the-art MRI quality enhancement and artifact removal methods for regenerating high-quality images while preserving essential anatomical and physiological feature maps without destroying important image information. Existing research gaps and future directions have also been provided by highlighting potential research areas for future developments, along with their importance and advantages in medical imaging.
... The statistical analysis of these improvements is crucial to determine their true significance. For instance [118], highlights the importance of using rigorous statistical methods such as Analysis of Variance (ANOVA) and Mixed Effects Models (MEM). ...
Medical imaging datasets for research are frequently collected from multiple imaging centers using different scanners, protocols, and settings. These variations affect data consistency and compatibility across different sources. Image harmonization is a critical step to mitigate the effects of factors like inherent differences between various vendors, hardware upgrades, protocol changes, and scanner calibration drift, as well as to ensure consistent data for medical image processing techniques. Given the critical importance and widespread relevance of this issue, a vast array of image harmonization methodologies have emerged, with deep learning-based approaches driving substantial advancements in recent times. The goal of this review paper is to examine the latest deep learning techniques employed for image harmonization by analyzing cutting-edge architectural approaches in the field of medical image harmonization, evaluating both their strengths and limitations. This paper begins by providing a comprehensive fundamental overview of image harmonization strategies, covering three critical aspects: established imaging datasets, commonly used evaluation metrics, and characteristics of different scanners. Subsequently, this paper analyzes recent structural MRI (Magnetic Resonance Imaging) harmonization techniques based on network architecture, network learning algorithm, network supervision strategy, and network output. The underlying architectures include U-Net, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), flow-based generative models, transformer-based approaches, as well as custom-designed network architectures. This paper investigates the effectiveness of Disentangled Representation Learning (DRL) as a pivotal learning algorithm in harmonization. Lastly, the review highlights the primary limitations in harmonization techniques, specifically the lack of comprehensive quantitative comparisons across different methods. The overall aim of this review is to serve as a guide for researchers and practitioners to select appropriate architectures based on their specific conditions and requirements. It also aims to foster discussions around ongoing challenges in the field and shed light on promising future research directions with the potential for significant advancements.
... This special issue does, however, provide valuable insights into a potentially crucial step in this process, the model evaluation. In the paper by Sharma et al., different statistical tests were used for the comparison of deep learning models trained with different hyperparameter settings and it was found that mixed effect models may be more sensitive than ANOVA for identifying such differences [49]. ...
Adversarial training has attracted much attention in enhancing the visual realism of images, but its efficacy in clinical imaging has not yet been explored. This work investigated adversarial training in a clinical context, by training 206 networks on the OASIS-1 dataset for improving low-resolution and low signal-to-noise ratio (SNR) magnetic resonance images. Each network corresponded to a different combination of perceptual and adversarial loss weights and distinct learning rate values. For each perceptual loss weighting, we identified its corresponding adversarial loss weighting that minimized structural disparity. Each optimally weighted adversarial loss yielded an average SSIM reduction of 1.5%. We further introduced a set of new metrics to assess other clinically relevant image features: Gradient Error (GE) to measure structural disparities; Sharpness to compute edge clarity; and Edge-Contrast Error (ECE) to quantify any distortion of the pixel distribution around edges. Including adversarial loss increased structural enhancement in visual inspection, which correlated with statistically consistent GE reductions (p-value << 0.05). This also resulted in increased Sharpness; however, the level of statistical significance was dependent on the perceptual loss weighting. Additionally, adversarial loss yielded ECE reductions for smaller perceptual loss weightings, while showing non-significant increases (p-value >> 0.05) when these weightings were higher, demonstrating that the increased Sharpness does not adversely distort the pixel distribution around the edges in the image. These studies clearly suggest that adversarial training significantly improves the performance of an MRI enhancement pipeline, and highlights the need for systematic studies of hyperparameter optimization and investigation of alternative image quality metrics.
Magnetic resonance imaging (MRI) provides high spatial resolution and excellent soft-tissue contrast without using harmful ionising radiation. Dynamic MRI is an essential tool for interventions to visualise movements or changes of the target organ. However, such MRI acquisitions with high temporal resolution suffer from limited spatial resolution - also known as the spatio-temporal trade-off of dynamic MRI. Several approaches, including deep learning based super-resolution approaches by treating each timepoint as individual volumes. This research addresses this issue by creating a deep learning model which attempts to learn both spatial and temporal relationships. A modified 3D UNet model, DDoS-UNet, is proposed - which takes the low-resolution volume of the current timepoint along with a prior image volume. Initially, the network is supplied with a static high-resolution planning scan as the prior image along with the low-resolution input to super-resolve the first timepoint. Then it continues step-wise by using the super-resolved timepoints as the prior image while super-resolving the subsequent timepoints. The model performance was tested with 3D dynamic data that was undersampled to different in-plane levels and achieved an average SSIM value of 0.951±0.017 while reconstructing only 4% of the k-space - which could result in a theoretical acceleration factor of 25. The proposed approach can be used to reduce the required scan-time while achieving high spatial resolution - consequently alleviating the spatio-temporal trade-off of dynamic MRI, by incorporating prior knowledge of spatio-temporal information from the available high-resolution planning scan and the existing temporal redundancy of time-series images into the network model.
MR scans of low-gamma X-nuclei, low-concentration metabolites, or standard imaging at very low field entail a challenging tradeoff between resolution, signal-to-noise, and acquisition duration. Deep learning (DL) techniques, such as UNets, can potentially be used to improve such “low-quality” (LQ) images. We investigate three UNets for upscaling LQ MRI: dense (DUNet), robust (RUNet), and anisotropic (AUNet). These were evaluated for two acquisition scenarios. In the same-subject High-Quality Complementary Priors (HQCP) scenario, an LQ and a high quality (HQ) image are collected and both LQ and HQ were inputs to the UNets. In the No Complementary Priors (NoCP) scenario, only the LQ images are collected and used as the sole input to the UNets. To address the lack of same-subject LQ and HQ images, we added data from the OASIS-1 database. The UNets were tested in upscaling 1/8, 1/4, and 1/2 undersampled images for both scenarios. As manifested by non-statically significant differences of matrices, also supported by subjective observation, the three UNets upscaled images equally well. This was in contrast to mixed effects statistics that clearly illustrated significant differences. Observations suggest that the detailed architecture of these UNets may not play a critical role. As expected, HQCP substantially improves upscaling with any of the UNets. The outcomes support the notion that DL methods may have merit as an integral part of integrated holistic approaches in advancing special MRI acquisitions; however, primary attention should be paid to the foundational step of such approaches, i.e., the actual data collected.
Dynamic MRI is an essential tool for interventions to visualise movements or changes in the target organ. However, such MRI acquisition with high temporal resolution suffers from limited spatial resolution-also known as the spatio-temporal trade-off. Several approaches, including deep learning based super-resolution approaches, have been proposed to mitigate this trade-off. Nevertheless, such an approach typically aims to super-resolve each time-point separately, treating them as individual volumes. This research addresses the problem by creating a deep learning model that attempts to learn spatial and temporal relationships. The performance was tested with 3D dynamic data that was undersampled to different in-plane levels. The proposed network achieved an average SSIM value of 0.951±0.017 while reconstructing the lowest resolution data (i.e. only 4% of the k-space acquired), resulting in a theoretical acceleration factor of 25.
Usage of effective classification techniques on Magnetic Resonance Imaging (MRI) helps in the proper diagnosis of brain tumors. Previous studies have focused on the classification of normal (nontumorous) or abnormal (tumorous) brain MRIs using methods such as Support Vector Machine (SVM) and AlexNet. In this paper, deep learning architectures are used to classify brain MRI images into normal or abnormal. Gender and age are added as higher attributes for more accurate and meaningful classification. A deep learning Convolutional Neural Network (CNN)-based technique and a Deep Neural Network (DNN) are also proposed for effective classification. Other deep learning architectures such as LeNet, AlexNet, ResNet, and traditional approaches such as SVM are also implemented to analyze and compare the results. Age and gender biases are found to be more useful and play a key role in classification, and they can be considered essential factors in brain tumor analysis. It is also worth noting that, in most circumstances, the proposed technique outperforms both existing SVM and AlexNet. The overall accuracy obtained is 88% (LeNet Inspired Model) and 80% (CNN-DNN) compared to SVM (82%) and AlexNet (64%), with best accuracy of 100%, 92%, 92%, and 81%, respectively.
MR image reconstruction techniques based on deep learning have shown their capacity for reducing MRI acquisition time and performance improvement compared to analytical methods. Despite the many challenges in training these rather large networks, novel methodologies have enhanced the capability for having clinical-grade MR image reconstruction in real-time. In recent literature, novel developments have facilitated the utilization of deep networks in various image processing inverse problems. In particular, it has been reported multiple times that the performance of deep networks can be improved by using short connections between layers. In this study, we introduce a novel MRI reconstruction method that utilizes such short connections. The dense connections are used inside densely connected residual blocks. Inside these blocks, the feature maps are concatenated to the subsequent layers. In this way, the extracted information is propagated until the last stage of the block. We have evaluated this densely connected residual block's efficiency in MRI reconstruction settings, by augmenting different types of effective deep network models with these blocks in novel structures. The quantitative and qualitative results indicate that this original introduction of the densely connected blocks to the MR image reconstruction problem improves the reconstruction performance significantly.
Understanding urban dynamics, such as estimating population, urban development, and several other uses, necessitates up-to-date large-scale building maps. Since aerial imagery provides enough textural and structural details, it has been utilized as a critical data source for building detection. However, accurate mapping of building objects from aerial imagery is a challenging task. This problem is attributed due to presence of vegetation and shadows in images that present similar spectral values and transparency as a building class. To deal with the issues mentioned above, we offer a new deep-learning structure named MultiRes-UNet network, which is an improved version of the original UNet network. In the proposed network, we utilized the MultiRes block to assimilate the features learned from the data at various scales and comprise some more spatial details. Also, we suggest the incorporation of several convolutional operations along with the skip connections to mitigate the differences between the encode–decoder features. Furthermore, we integrated semantic edge information with semantic polygons to solve the issue of irregular semantic polygons and enhance the boundary of semantic polygons. We tested our network on aerial images for roof segmentation dataset, and the experimental results exhibited that the proposed network can improve the quantitative results of Intersection Over Union to 0.78% after adding semantic edges. We also used state-of-the-art comparative models such as UNet, DeeplabV3, ResNet, and FractalNet networks to show the competency of the introduced network, and the results prove the success of the introduced network for building object extraction from aerial imagery.
In this work, we address the problem of image denoising using deep neural networks. Recent developments in convolutional neural networks provide a very potent alternative for image restoration applications and in particular for image denoising. A particularly popular deep network structure for image processing are the auto-encoders which include the U-Net as an important example. U-Nets contract and expand feature maps repeatedly, which leads to extraction of multi scale information as well as an increase in the effective receptive field when compared to conventional convolutional nets. In this paper, we propose the integration of a multi scale channel attention module through a U-Net structure as a novelty for the image denoising problem. The introduced network structure also utilizes multi scale inputs in the various substages of the encoder module in a novel manner. Simulation results demonstrate competitive and mostly superior performance when compared to some state of the art deep learning based image denoising methodologies. Qualitative results also indicate that the developed deep network framework has powerful detail preserving capability.
Purpose
To develop a deep‐learning–based method to quantify multiple parameters in the brain from conventional contrast‐weighted images.
Methods
Eighteen subjects were imaged using an MR Multitasking sequence to generate reference T1 and T2 maps in the brain. Conventional contrast‐weighted images consisting of T1 MPRAGE, T1 GRE, and T2 FLAIR were acquired as input images. A U‐Net–based neural network was trained to estimate T1 and T2 maps simultaneously from the contrast‐weighted images. Six‐fold cross‐validation was performed to compare the network outputs with the MR Multitasking references.
Results
The deep‐learning T1/T2 maps were comparable with the references, and brain tissue structures and image contrasts were well preserved. A peak signal‐to‐noise ratio >32 dB and a structural similarity index >0.97 were achieved for both parameter maps. Calculated on brain parenchyma (excluding CSF), the mean absolute errors (and mean percentage errors) for T1 and T2 maps were 52.7 ms (5.1%) and 5.4 ms (7.1%), respectively. ROI measurements on four tissue compartments (cortical gray matter, white matter, putamen, and thalamus) showed that T1 and T2 values provided by the network outputs were in agreement with the MR Multitasking reference maps. The mean differences were smaller than ±1%, and limits of agreement were within ±5% for T1 and within ±10% for T2 after taking the mean differences into account.
Conclusion
A deep‐learning–based technique was developed to estimate T1 and T2 maps from conventional contrast‐weighted images in the brain, enabling simultaneous qualitative and quantitative MRI without modifying clinical protocols.
Recent years have seen a resurgence of interest in inexpensive low magnetic field (< 0.3 T) MRI systems mainly due to advances in magnet, coil and gradient set designs. Most of these advances have focused on improving hardware and signal acquisition strategies, and far less on the use of advanced image reconstruction methods to improve attainable image quality at low field. We describe here the use of our end-to-end deep neural network approach (AUTOMAP) to improve the image quality of highly noise-corrupted low-field MRI data. We compare the performance of this approach to two additional state-of-the-art denoising pipelines. We find that AUTOMAP improves image reconstruction of data acquired on two very different low-field MRI systems: human brain data acquired at 6.5 mT, and plant root data acquired at 47 mT, demonstrating SNR gains above Fourier reconstruction by factors of 1.5- to 4.5-fold, and 3-fold, respectively. In these applications, AUTOMAP outperformed two different contemporary image-based denoising algorithms, and suppressed noise-like spike artifacts in the reconstructed images. The impact of domain-specific training corpora on the reconstruction performance is discussed. The AUTOMAP approach to image reconstruction will enable significant image quality improvements at low-field, especially in highly noise-corrupted environments.
Purpose
Deep learning has had success with MRI reconstruction, but previously published works use real‐valued networks. The few works which have tried complex‐valued networks have not fully assessed their impact on phase. Therefore, the purpose of this work is to fully investigate end‐to‐end complex‐valued convolutional neural networks (CNNs) for accelerated MRI reconstruction and in several phase‐based applications in comparison to 2‐channel real‐valued networks.
Methods
Several complex‐valued activation functions for MRI reconstruction were implemented, and their performance was compared. Complex‐valued convolution was implemented and tested on an unrolled network architecture and a U‐Net–based architecture over a wide range of network widths and depths with knee, body, and phase‐contrast datasets.
Results
Quantitative and qualitative results demonstrated that complex‐valued CNNs with complex‐valued convolutions provided superior reconstructions compared to real‐valued convolutions with the same number of trainable parameters for both an unrolled network architecture and a U‐Net–based architecture, and for 3 different datasets. Complex‐valued CNNs consistently had superior normalized RMS error, structural similarity index, and peak SNR compared to real‐valued CNNs.
Conclusion
Complex‐valued CNNs can enable superior accelerated MRI reconstruction and phase‐based applications such as fat–water separation, and flow quantification compared to real‐valued convolutional neural networks.
Chest radiographs are among the most frequently acquired images in radiology and are often the subject of computer vision research. However, most of the models used to classify chest radiographs are derived from openly available deep neural networks, trained on large image datasets. These datasets differ from chest radiographs in that they are mostly color images and have substantially more labels. Therefore, very deep convolutional neural networks (CNN) designed for ImageNet and often representing more complex relationships, might not be required for the comparably simpler task of classifying medical image data. Sixteen different architectures of CNN were compared regarding the classification performance on two openly available datasets, the CheXpert and COVID-19 Image Data Collection. Areas under the receiver operating characteristics curves (AUROC) between 0.83 and 0.89 could be achieved on the CheXpert dataset. On the COVID-19 Image Data Collection, all models showed an excellent ability to detect COVID-19 and non-COVID pneumonia with AUROC values between 0.983 and 0.998. It could be observed, that more shallow networks may achieve results comparable to their deeper and more complex counterparts with shorter training times, enabling classification performances on medical image data close to the state-of-the-art methods even when using limited hardware.
Background Cardiac MRI is limited by long acquisition times, yet faster acquisition of smaller-matrix images reduces spatial detail. Deep learning (DL) might enable both faster acquisition and higher spatial detail via super-resolution. Purpose To explore the feasibility of using DL to enhance spatial detail from small-matrix MRI acquisitions and evaluate its performance against that of conventional image upscaling methods. Materials and Methods Short-axis cine cardiac MRI examinations performed between January 2012 and December 2018 at one institution were retrospectively collected for algorithm development and testing. Convolutional neural networks (CNNs), a form of DL, were trained to perform super resolution in image space by using synthetically generated low-resolution data. There were 70%, 20%, and 10% of examinations allocated to training, validation, and test sets, respectively. CNNs were compared against bicubic interpolation and Fourier-based zero padding by calculating the structural similarity index (SSIM) between high-resolution ground truth and each upscaling method. Means and standard deviations of the SSIM were reported, and statistical significance was determined by using the Wilcoxon signed-rank test. For evaluation of clinical performance, left ventricular volumes were measured, and statistical significance was determined by using the paired Student t test. Results For CNN training and retrospective analysis, 400 MRI scans from 367 patients (mean age, 48 years ± 18; 214 men) were included. All CNNs outperformed zero padding and bicubic interpolation at upsampling factors from two to 64 (P < .001). CNNs outperformed zero padding on more than 99.2% of slices (9828 of 9907). In addition, 10 patients (mean age, 51 years ± 22; seven men) were prospectively recruited for super-resolution MRI. Super-resolved low-resolution images yielded left ventricular volumes comparable to those from full-resolution images (P > .05), and super-resolved full-resolution images appeared to further enhance anatomic detail. Conclusion Deep learning outperformed conventional upscaling methods and recovered high-frequency spatial information. Although training was performed only on short-axis cardiac MRI examinations, the proposed strategy appeared to improve quality in other imaging planes. © RSNA, 2020 Online supplemental material is available for this article.
Purpose
To develop a deep learning‐based Bayesian estimation for MRI reconstruction.
Methods
We modeled the MRI reconstruction problem with Bayes’s theorem, following the recently proposed PixelCNN++ method. The image reconstruction from incomplete k‐space measurement was obtained by maximizing the posterior possibility. A generative network was utilized as the image prior, which was computationally tractable, and the k‐space data fidelity was enforced by using an equality constraint. The stochastic backpropagation was utilized to calculate the descent gradient in the process of maximum a posterior, and a projected subgradient method was used to impose the equality constraint. In contrast to the other deep learning reconstruction methods, the proposed one used the likelihood of prior as the training loss and the objective function in reconstruction to improve the image quality.
Results
The proposed method showed an improved performance in preserving image details and reducing aliasing artifacts, compared with GRAPPA, ℓ1‐ESPRiT, model‐based deep learning architecture for inverse problems (MODL), and variational network (VN), last two were state‐of‐the‐art deep learning reconstruction methods. The proposed method generally achieved more than 3 dB peak signal‐to‐noise ratio improvement for compressed sensing and parallel imaging reconstructions compared with the other methods.
Conclusions
The Bayesian estimation significantly improved the reconstruction performance, compared with the conventional ℓ1‐sparsity prior in compressed sensing reconstruction tasks. More importantly, the proposed reconstruction framework can be generalized for most MRI reconstruction scenarios.
Convolutional neural networks (CNNs) demonstrate excellent performance when employed to reconstruct the images obtained by compressed-sensing magnetic resonance imaging (CS-MRI). Our study aimed to enhance image quality by developing a novel iterative reconstruction approach that utilizes image-based CNNs and k-space correction to preserve original k-space data. In the proposed method, CNNs represent a priori information concerning image spaces. First, the CNNs are trained to map zero-filling images onto corresponding full-sampled images. Then, they recover the zero-filled part of the k-space data. Subsequently, k-space corrections, which involve the replacement of unfilled regions by original k-space data, are implemented to preserve the original k-space data. The above-mentioned processes are used iteratively. The performance of the proposed method was validated using a T2-weighted brain-image dataset, and experiments were conducted with several sampling masks. Finally, the proposed method was compared with other noniterative approaches to demonstrate its effectiveness. The aliasing artifacts in the reconstructed images obtained using the proposed approach were reduced compared to those using other state-of-the-art techniques. In addition, the quantitative results obtained in the form of the peak signal-to-noise ratio and structural similarity index demonstrated the effectiveness of the proposed method. The proposed CS-MRI method enhanced MR image quality with high-throughput examinations.
Cardiac magnetic resonance (CMR) imaging is an important tool for the non-invasive assessment of cardiovascular disease. However, CMR suffers from long acquisition times due to the need of obtaining images with high temporal and spatial resolution, different contrasts, and/or whole-heart coverage. In addition, both cardiac and respiratory-induced motion of the heart during the acquisition need to be accounted for, further increasing the scan time. Several undersampling reconstruction techniques have been proposed during the last decades to speed up CMR acquisition. These techniques rely on acquiring less data than needed and estimating the non-acquired data exploiting some sort of prior information. Parallel imaging and compressed sensing undersampling reconstruction techniques have revolutionized the field, enabling 2-to 3-fold scan time accelerations to become standard in clinical practice. Recent scientific advances in CMR reconstruction hinge on the thriving field of artificial intelligence. Machine learning reconstruction approaches have been recently proposed to learn the non-linear optimization process employed in CMR reconstruction. Unlike analytical methods for which the reconstruction problem is explicitly defined into the optimization process, machine learning techniques make use of large data sets to learn the key reconstruction parameters and priors. In particular, deep learning techniques promise to use deep neural networks (DNN) to learn the reconstruction process from existing datasets in advance, providing a fast and efficient reconstruction that can be applied to all newly acquired data. However, before machine learning and DNN can realize their full potentials and enter widespread clinical routine for CMR image reconstruction, there are several technical hurdles that need to be addressed. In this article, we provide an overview of the recent developments in the area of artificial intelligence for CMR image reconstruction. The underlying assumptions of established techniques such as compressed sensing and low-rank reconstruction are briefly summarized, while a greater focus is given to recent advances in dictionary learning and deep learning based CMR reconstruction. In particular, approaches that exploit neural networks as implicit or explicit priors are discussed for 2D dynamic cardiac imaging and 3D whole-heart CMR imaging. Current limitations, challenges, and potential future directions of these techniques are also discussed.
Single image super-resolution (SISR) is a challenging ill-posed problem which aims to restore or infer a high-resolution image from a low-resolution one. Powerful deep learning-based techniques have achieved state-of-the art performance in SISR; however, they can underperform when handling images with non-stationary degradations, such as for the application of projector resolution enhancement. In this paper, a new UNet architecture that is able to learn the relationship between a set of degraded low-resolution images and their corresponding original high-resolution images is proposed. We propose employing a
degradation model on training images in a non-stationary way, allowing the construction of a robust UNet (RUNet) for image super-resolution (SR). Experimental results show
that the proposed RUNet improves the visual quality of the obtained super-resolution images while maintaining a low reconstruction error.
Purpose
Neural networks have received recent interest for reconstruction of undersampled MR acquisitions. Ideally, network performance should be optimized by drawing the training and testing data from the same domain. In practice, however, large datasets comprising hundreds of subjects scanned under a common protocol are rare. The goal of this study is to introduce a transfer‐learning approach to address the problem of data scarcity in training deep networks for accelerated MRI.
Methods
Neural networks were trained on thousands (upto 4 thousand) of samples from public datasets of either natural images or brain MR images. The networks were then fine‐tuned using only tens of brain MR images in a distinct testing domain. Domain‐transferred networks were compared to networks trained directly in the testing domain. Network performance was evaluated for varying acceleration factors (4‐10), number of training samples (0.5‐4k), and number of fine‐tuning samples (0‐100).
Results
The proposed approach achieves successful domain transfer between MR images acquired with different contrasts (T1‐ and T2‐weighted images) and between natural and MR images (ImageNet and T1‐ or T2‐weighted images). Networks obtained via transfer learning using only tens of images in the testing domain achieve nearly identical performance to networks trained directly in the testing domain using thousands (upto 4 thousand) of images.
Conclusion
The proposed approach might facilitate the use of neural networks for MRI reconstruction without the need for collection of extensive imaging datasets.
Purpose
Magnetic resonance (MR) imaging with a long scan time can lead to degraded images due to patient motion, patient discomfort, and increased costs. For these reasons, the role of rapid MR imaging is important. In this study, we propose the joint reconstruction of multicontrast brain MR images from down‐sampled data to accelerate the data acquisition process using a novel deep‐learning network.
Methods
Twenty‐one healthy volunteers (female/male = 7/14, age = 26 ± 4 yr, range 22–35 yr) and 16 postoperative patients (female/male = 7/9, age = 49 ± 9 yr, range 37–62 yr) were scanned on a 3T whole‐body scanner for prospective and retrospective studies, respectively, using both T1‐weighted spin‐echo (SE) and T2‐weighted fast spin‐echo (FSE) sequences. We proposed a network which we term “X‐net” to reconstruct both T1‐ and T2‐weighted images from down‐sampled images as well as a network termed “Y‐net” which reconstructs T2‐weighted images from highly down‐sampled T2‐weighted images and fully sampled T1‐weighted images. Both X‐net and Y‐net are composed of two concatenated subnetworks. We investigate optimal sampling patterns, the optimal patch size for augmentation, and the optimal acceleration factors for network training. An additional Y‐net combined with a generative adversarial network (GAN) was also implemented and tested to investigate the effects of the GAN on the Y‐net performance. Single‐ and joint‐reconstruction parallel‐imaging and compressed‐sensing algorithms along with a conventional U‐net were also tested and compared with the proposed networks. For this comparison, the structural similarity (SSIM), normalized mean square error (NMSE), and Fréchet inception distance (FID) were calculated between the outputs of the networks and fully sampled images. The statistical significance of the performance was evaluated by assessing the interclass correlation and in paired t‐tests.
Results
The outputs from the two concatenated subnetworks were closer to the fully sampled images compared to those from one subnetwork, with this result showing statistical significance. Uniform down‐sampling led to a statically significant improvement in the image quality compared to random or central down‐sampling patterns. In addition, the proposed networks provided higher SSIM and NMSE values than U‐net, compressed‐sensing, and parallel‐imaging algorithms, all at statistically significant levels. The GAN‐based Y‐net showed a better FID and more realistic images compared to a non‐GAN‐based Y‐net. The performance capabilities of the networks were similar between normal subjects and patients.
Conclusions
The proposed X‐net and Y‐net effectively reconstructed full images from down‐sampled images, outperforming the conventional parallel‐imaging, compressed‐sensing and U‐net methods and providing more realistic images in combination with a GAN. The developed networks potentially enable us to accelerate multicontrast anatomical MR imaging in routine clinical studies including T1‐and T2‐weighted imaging.
Purpose
Calculation of sophisticated MR contrasts often requires complex mathematical modeling. Data evaluation is computationally expensive, vulnerable to artifacts, and often sensitive to fit algorithm parameters. In this work, we investigate whether neural networks can provide not only fast model fitting results, but also a quality metric for the predicted values, so called uncertainty quantification, investigated here in the context of multi‐pool Lorentzian fitting of CEST MRI spectra at 3T.
Methods
A deep feed‐forward neural network including a probabilistic output layer allowing for uncertainty quantification was set up to take uncorrected CEST‐spectra as input and predict 3T Lorentzian parameters of a 4‐pool model (water, semisolid MT, amide CEST, NOE CEST), including the B0 inhomogeneity. Networks were trained on data from 3 subjects with and without data augmentation, and applied to untrained data from 1 additional subject and 1 brain tumor patient. Comparison to conventional Lorentzian fitting was performed on different perturbations of input data.
Results
The deepCEST 3T networks provided fast and accurate predictions of all Lorentzian parameters and were robust to input perturbations because of noise or B0 artifacts. The uncertainty quantification detected fluctuations in input data by increase of the uncertainty intervals. The method generalized to unseen brain tumor patient CEST data.
Conclusions
The deepCEST 3T neural network provides fast and robust estimation of CEST parameters, enabling online reconstruction of sophisticated CEST contrast images without the typical computational cost. Moreover, the uncertainty quantification indicates if the predictions are trustworthy, enabling confident interpretation of contrast changes.
Magnetic resonance spectroscopic imaging (SI) is a unique imaging technique that provides biochemical information from in vivo tissues. The 1H spectra acquired from several spatial regions are quantified to yield metabolite concentrations reflective of tissue metabolism. However, since these metabolites are found in tissues at very low concentrations, SI is often acquired with limited spatial resolution. In this work, we test the hypothesis that deep learning is able to upscale low resolution SI, together with the T1-weighted (T1w) image, to reconstruct high resolution SI. We report on a novel densely connected UNet (D-UNet) architecture capable of producing super-resolution spectroscopic images. The inputs for the D-UNet are the T1w image and the low resolution SI image while the output is the high resolution SI. The results of the D-UNet are compared both qualitatively and quantitatively to simulated and in vivo high resolution SI. It is found that this deep learning approach can produce high quality spectroscopic images and reconstruct entire 1H spectra from low resolution acquisitions, which can greatly advance the current SI workflow.
Background:
To review and evaluate approaches to convolutional neural network (CNN) reconstruction for accelerated cardiac MR imaging in the real clinical context.
Methods:
Two CNN architectures, Unet and residual network (Resnet) were evaluated using quantitative and qualitative assessment by radiologist. Four different loss functions were also considered: pixel-wise (L1 and L2), patch-wise structural dissimilarity (Dssim) and feature-wise (perceptual loss). The networks were evaluated using retrospectively and prospectively under-sampled cardiac MR data.
Results:
Based on our assessments, we find that Resnet and Unet achieve similar image quality but that former requires only 100,000 parameters compared to 1.3 million parameters for the latter. The perceptual loss function performed significantly better than L1, L2 or Dssim loss functions as determined by the radiologist scores.
Conclusions:
CNN image reconstruction using Resnet yields comparable image quality to Unet with 10X the number of parameters. This has implications for training with significantly lower data requirements. Network training using the perceptual loss function was found to better agree with radiologist scoring compared to L1, L2 or Dssim loss functions.
Purpose
To determine the feasibility of employing the prior knowledge of well‐separated chemical exchange saturation transfer (CEST) signals in the 9.4 T Z‐spectrum to separate overlapping CEST signals acquired at 3 T, using a deep learning approach trained with 3 T and 9.4 T CEST spectral data from brains of the same subjects.
Methods
Highly spectrally resolved Z‐spectra from the same volunteer were acquired by 3D‐snapshot CEST MRI at 3 T and 9.4 T at low saturation power of B1 = 0.6 µT. The volume‐registered 3 T Z‐spectra‐stack was then used as input data for a three layer deep neural network with the volume‐registered 9.4 T fitted parameter stack as target data.
Results
An optimized neural net architecture could be found and verified in healthy volunteers. The gray‐/white‐matter contrast of the different CEST effects was predicted with only small deviations (Pearson R = 0.89). The 9.4 T prediction was less noisy compared to the directly measured CEST maps, although at the cost of slightly lower tissue contrast. Application to an unseen tumor patient measured at 3 T and 9.4 T revealed that tumorous tissue Z‐spectra and corresponding hyper‐/hypointensities of different CEST effects can also be predicted (Pearson R = 0.84).
Conclusion
The 9.4 T CEST signals acquired at low saturation power can be accurately estimated from CEST imaging at 3 T using a neural network trained with coregistered 3 T and 9.4 T data of healthy subjects. The deepCEST approach generalizes to Z‐spectra of tumor areas and might indicate whether additional ultrahigh‐field (UHF) scans will be beneficial.
Magnetic Resonance Imaging (MRI) produces excellent soft tissue contrast, albeit it is an inherently slow imaging modality. Promising deep learning methods have recently been proposed to reconstruct accelerated MRI scans. However, existing methods still suffer from various limitations regarding image fidelity, contextual sensitivity, and reliance on fully-sampled acquisitions for model training. To comprehensively address these limitations, we propose a novel self-supervised deep reconstruction model, named Self-Supervised Diffusion Reconstruction (SSDiffRecon). SSDiffRecon expresses a conditional diffusion process as an unrolled architecture that interleaves cross-attention transformers for reverse diffusion steps with data-consistency blocks for physics-driven processing. Unlike recent diffusion methods for MRI reconstruction, a self-supervision strategy is adopted to train SSDiffRecon using only undersampled k-space data. Comprehensive experiments on public brain MR datasets demonstrates the superiority of SSDiffRecon against state-of-the-art supervised, and self-supervised baselines in terms of reconstruction speed and quality. Implementation will be available at https://github.com/yilmazkorkmaz1/SSDiffRecon.
Deep MRI reconstruction is commonly performed with conditional models that de-alias undersampled acquisitions to recover images consistent with fully-sampled data. Since conditional models are trained with knowledge of the imaging operator, they can show poor generalization across variable operators. Unconditional models instead learn generative image priors decoupled from the operator to improve reliability against domain shifts related to the imaging operator. Recent diffusion models are particularly promising given their high sample fidelity. Nevertheless, inference with a static image prior can perform suboptimally. Here we propose the first adaptive diffusion prior for MRI reconstruction, AdaDiff, to improve performance and reliability against domain shifts. AdaDiff leverages an efficient diffusion prior trained via adversarial mapping over large reverse diffusion steps. A two-phase reconstruction is executed following training: a rapid-diffusion phase that produces an initial reconstruction with the trained prior, and an adaptation phase that further refines the result by updating the prior to minimize data-consistency loss. Demonstrations on multi-contrast brain MRI clearly indicate that AdaDiff outperforms competing conditional and unconditional methods under domain shifts, and achieves superior or on par within-domain performance.
Supervised reconstruction models are characteristically trained on matched pairs of undersampled and fully-sampled data to capture an MRI prior, along with supervision regarding the imaging operator to enforce data consistency. To reduce supervision requirements, the recent deep image prior framework instead conjoins untrained MRI priors with the imaging operator during inference. Yet, canonical convolutional architectures are suboptimal in capturing long-range relationships, and priors based on randomly initialized networks may yield suboptimal performance. To address these limitations, here we introduce a novel unsupervised MRI reconstruction method based on zero-Shot Learned Adversarial TransformERs (SLATER). SLATER embodies a deep adversarial network with cross-attention transformers to map noise and latent variables onto coil-combined MR images. During pre-training, this unconditional network learns a high-quality MRI prior in an unsupervised generative modeling task. During inference, a zero-shot reconstruction is then performed by incorporating the imaging operator and optimizing the prior to maximize consistency to undersampled data. Comprehensive experiments on brain MRI datasets clearly demonstrate the superior performance of SLATER against state-of-the-art unsupervised methods.
This work is an update of a previous paper on the same topic published a few years ago (Borji, 2019). With the dramatic progress in generative modeling, a suite of new quantitative and qualitative techniques to evaluate models has emerged. Although some measures such as Inception Score, Fréchet Inception Distance, Precision-Recall, and Perceptual Path Length are relatively more popular, GAN evaluation is not a settled issue and there is still room for improvement. Here, I describe new dimensions that are becoming important in assessing models (e.g. bias and fairness) and discuss the connection between GAN evaluation and deepfakes. These are important areas of concern in the machine learning community today and progress in GAN evaluation can help mitigate them.
Background
Prostate volume, as determined by magnetic resonance imaging (MRI), is a useful biomarker both for distinguishing between benign and malignant pathology and can be used either alone or combined with other parameters such as prostate‐specific antigen.
Purpose
This study compared different deep learning methods for whole‐gland and zonal prostate segmentation.
Study Type
Retrospective.
Population
A total of 204 patients (train/test = 99/105) from the PROSTATEx public dataset.
Field strength/Sequence
A 3 T, TSE T2‐weighted.
Assessment
Four operators performed manual segmentation of the whole‐gland, central zone + anterior stroma + transition zone (TZ), and peripheral zone (PZ). U‐net, efficient neural network (ENet), and efficient residual factorized ConvNet (ERFNet) were trained and tuned on the training data through 5‐fold cross‐validation to segment the whole gland and TZ separately, while PZ automated masks were obtained by the subtraction of the first two.
Statistical Tests
Networks were evaluated on the test set using various accuracy metrics, including the Dice similarity coefficient (DSC). Model DSC was compared in both the training and test sets using the analysis of variance test (ANOVA) and post hoc tests. Parameter number, disk size, training, and inference times determined network computational complexity and were also used to assess the model performance differences. A P < 0.05 was selected to indicate the statistical significance.
Results
The best DSC (P < 0.05) in the test set was achieved by ENet: 91% ± 4% for the whole gland, 87% ± 5% for the TZ, and 71% ± 8% for the PZ. U‐net and ERFNet obtained, respectively, 88% ± 6% and 87% ± 6% for the whole gland, 86% ± 7% and 84% ± 7% for the TZ, and 70% ± 8% and 65 ± 8% for the PZ. Training and inference time were lowest for ENet.
Data Conclusion
Deep learning networks can accurately segment the prostate using T2‐weighted images.
Evidence Level
4
Technical Efficacy
Stage 2
Purpose
To demonstrate the utility of compressed sensing with parallel imaging (Compressed SPEEDER) and AiCE compared with that of conventional parallel imaging (SPEEDER) for shortening examination time and improving image quality of women’s pelvic MRI.
Method
Thirty consecutive patients with women’s pelvic diseases (mean age 50 years) underwent T2-weighted imaging using Compressed SPEEDER as well as conventional SPEEDER reconstructed with and without AiCE. The examination times were recorded, and signal-to-noise ratio (SNR) was calculated for every patient. Moreover, overall image quality was assessed using a 5-point scoring system, and final scores for all patients were determined by consensus of two readers. Mean examination time, SNR and overall image quality were compared among the four data sets by Wilcoxon signed–rank test.
Results
Examination times for Compressed SPEEDER with and without AiCE were significantly shorter than those for conventional SPEEDER with and without AiCE (with AiCE: p < 0.0001, without AiCE: p < 0.0001). SNR of Compressed SPEEDER and of SPEEDER with AiCE was significantly superior to that of Compressed SPEEDER without AiCE (vs. Compressed SPEEDER, p = 0.01; vs. SPEEDER, p = 0.009). Overall image quality of Compressed SPEEDER with AiCE and of SPEEDER with and without AiCE was significantly higher than that of Compressed SPEEDER without AiCE (vs. Compressed SPEEDER with AiCE, p < 0.0001; vs. SPEEDER with AiCE, p < 0.0001; SPEEDER without AiCE, p = 0.0003).
Conclusion
Image quality and shorten examination time for T2-weighted imaging in women’s pelvic MRI can be significantly improved by using Compressed SPEEDER with AiCE in comparison with conventional SPEEDER, although other sequences were not tested.
The accurate reconstruction of under-sampled magnetic resonance imaging (MRI) data using modern deep learning technology, requires significant effort to design the necessary complex neural network architectures. The cascaded network architecture for MRI reconstruction has been widely used, while it suffers from the “vanishing gradient” problem when the network becomes deep. In addition, the homogeneous architecture degrades the representation capacity of the network. In this work, we present an enhanced MRI reconstruction network using a residual in residual basic block. For each cell in the basic block, we use the differentiable neural architecture search (NAS) technique to automatically choose the optimal operation among eight variants of the dense block. This new heterogeneous network is evaluated on two publicly available datasets and outperforms all current state-of-the-art methods, which demonstrates the effectiveness of our proposed method.
MR images scanned at low magnetic field (<1 T) have lower resolution in the slice direction and lower contrast, due to a relatively small signal-to-noise ratio (SNR) than those from high field (typically 1.5T and 3T). We adapt the recent idea of Image Quality Transfer (IQT) to enhance very low-field structural images aiming to estimate the resolution, spatial coverage, and contrast of high-field images. Analogous to many learning-based image enhancement techniques, IQT generates training data from high-field scans alone by simulating low-field images through a pre-defined decimation model. However, the ground truth decimation model is not well-known in practice, and lack of its specification can bias the trained model, aggravating performance on the real low-field scans. In this paper we propose a probabilistic decimation simulator to improve robustness of model training. It is used to generate and augment various low-field images whose parameters are random variables and sampled from an empirical distribution related to tissue-specific SNR on a 0.36T scanner. The probabilistic decimation simulator is model-agnostic, that is, it can be used with any super-resolution networks. Furthermore we propose a variant of U-Net architecture to improve its learning performance. We show promising qualitative results from clinical low-field images confirming the strong efficacy of IQT in an important new application area: epilepsy diagnosis in sub-Saharan Africa where only low-field scanners are normally available.
Background:
Multiphoton microscopy (MPM) offers a feasible approach for the biopsy in clinical medicine, but it has not been used in clinical applications due to the lack of efficient image processing methods, especially the automatic segmentation technology. Segmentation technology is still one of the most challenging assignments of the MPM imaging technique.
Methods:
The MPM imaging segmentation model based on deep learning is one of the most effective methods to address this problem. In this paper, the practicability of using a convolutional neural network (CNN) model to segment the MPM image of skin cells in vivo was explored. A set of MPM in vivo skin cells images with a resolution of 128×128 was successfully segmented under the Python environment with TensorFlow. A novel deep-learning segmentation model named Dense-UNet was proposed. The Dense-UNet, which is based on U-net structure, employed the dense concatenation to deepen the depth of the network architecture and achieve feature reuse. This model included four expansion modules (each module consisted of four down-sampling layers) to extract features.
Results:
Sixty training images were taken from the dorsal forearm using a femtosecond Ti:Sa laser running at 735 nm. The resolution of the images is 128×128 pixels. Experimental results confirmed that the accuracy of Dense-UNet (92.54%) was higher than that of U-Net (88.59%), with a significantly lower loss value of 0.1681. The 90.60% Dice coefficient value of Dense-UNet outperformed U-Net by 11.07%. The F1-Score of Dense-UNet, U-Net, and Seg-Net was 93.35%, 90.02%, and 85.04%, respectively.
Conclusions:
The deepened down-sampling path improved the ability of the model to capture cellular fined-detailed boundary features, while the symmetrical up-sampling path provided a more accurate location based on the test result. These results were the first time that the segmentation of MPM in vivo images had been adopted by introducing a deep CNN to bridge this gap in Dense-UNet technology. Dense-UNet has reached ultramodern performance for MPM images, especially for in vivo images with low resolution. This implementation supplies an automatic segmentation model based on deep learning for high-precision segmentation of MPM images in vivo.
Purpose
To enable fast reconstruction of undersampled motion-compensated whole-heart 3D coronary magnetic resonance angiography (CMRA) by learning a multi-scale variational neural network (MS-VNN) which allows the acquisition of high-quality 1.2 × 1.2 × 1.2 mm isotropic volumes in a short and predictable scan time.
Methods
Eighteen healthy subjects and one patient underwent free-breathing 3D CMRA acquisition with variable density spiral-like Cartesian sampling, combined with 2D image navigators for translational motion estimation/compensation. The proposed MS-VNN learns two sets of kernels and activation functions for the magnitude and phase images of the complex-valued data. For the magnitude, a multi-scale approach is applied to better capture the small calibre of the coronaries. Ten subjects were considered for training and validation. Prospectively undersampled motion-compensated data with 5-fold and 9-fold accelerations, from the remaining 9 subjects, were used to evaluate the framework. The proposed approach was compared to Wavelet-based compressed-sensing (CS), conventional VNN, and to an additional fully-sampled (FS) scan.
Results
The average acquisition time (m:s) was 4:11 for 5-fold, 2:34 for 9-fold acceleration and 18:55 for fully-sampled. Reconstruction time with the proposed MS-VNN was ~14 s. The proposed MS-VNN achieves higher image quality than CS and VNN reconstructions, with quantitative right coronary artery sharpness (CS:43.0%, VNN:43.9%, MS-VNN:47.0%, FS:50.67%) and vessel length (CS:7.4 cm, VNN:7.7 cm, MS-VNN:8.8 cm, FS:9.1 cm) comparable to the FS scan.
Conclusion
The proposed MS-VNN enables 5-fold and 9-fold undersampled CMRA acquisitions with comparable image quality that the corresponding fully-sampled scan. The proposed framework achieves extremely fast reconstruction time and does not require tuning of regularization parameters, offering easy integration into clinical workflow.
Magnetic resonance imaging (MRI) is widely used for screening, diagnosis, image-guided therapy, and scientific research. A significant advantage of MRI over other imaging modalities such as computed tomography (CT) and nuclear imaging is that it clearly shows soft tissues in multi-contrasts. Compared with other medical image super-resolution methods that are in a single contrast, multi-contrast super-resolution studies can synergize multiple contrast images to achieve better super-resolution results. In this paper, we propose a one-level nonprogressive neural network for low up-sampling multi-contrast super-resolution and a two-level progressive network for high upsampling multi-contrast super-resolution. The proposed networks integrate multi-contrast information in a high-level feature space and optimize the imaging performance by minimizing a composite loss function, which includes mean-squared-error, adversarial loss, perceptual loss, and textural loss. Our experimental results demonstrate that 1) the proposed networks can produce MRI super-resolution images with good image quality and outperform other multi-contrast super-resolution methods in terms of structural similarity and peak signal-to-noise ratio; 2) combining multi-contrast information in a high-level feature space leads to a signicantly improved result than a combination in the lowlevel pixel space; and 3) the progressive network produces a better super-resolution image quality than the non-progressive network, even if the original low-resolution images were highly down-sampled.
Artificial intelligence (AI) shows tremendous promise in the field of medical imaging, with recent breakthroughs applying deep‐learning models for data acquisition, classification problems, segmentation, image synthesis, and image reconstruction. With an eye towards clinical applications, we summarize the active field of deep‐learning‐based MR image reconstruction. We review the basic concepts of how deep‐learning algorithms aid in the transformation of raw k‐space data to image data, and specifically examine accelerated imaging and artifact suppression. Recent efforts in these areas show that deep‐learning‐based algorithms can match and, in some cases, eclipse conventional reconstruction methods in terms of image quality and computational efficiency across a host of clinical imaging applications, including musculoskeletal, abdominal, cardiac, and brain imaging. This article is an introductory overview aimed at clinical radiologists with no experience in deep‐learning‐based MR image reconstruction and should enable them to understand the basic concepts and current clinical applications of this rapidly growing area of research across multiple organ systems.
Compressed sensing (CS) reconstruction methods leverage sparse structure in underlying signals to recover high-resolution images from highly undersampled measurements. When applied to magnetic resonance imaging (MRI), CS has the potential to dramatically shorten MRI scan times, increase diagnostic value, and improve the overall patient experience. However, CS has several shortcomings that limit its clinical translation. These include 1) artifacts arising from inaccurate sparse modeling assumptions, 2) extensive parameter tuning required for each clinical application, and 3) clinically infeasible reconstruction times. Recently, CS has been extended to incorporate deep neural networks as a way of learning complex image priors from historical exam data. Commonly referred to as unrolled neural networks, these techniques have proven to be a compelling and practical approach to address the challenges of sparse CS. In this tutorial, we review the classical CS formulation and outline steps needed to transform this formulation into a deep-learning-based reconstruction framework. Supplementary open-source code in Python is used to demonstrate this approach with open databases. Further, we discuss considerations in applying unrolled neural networks in the clinical setting.
Magnetic resonance imaging (MRI) is a widely used medical imaging modality. However, due to the limitations in hardware, scan time, and throughput, it is often clinically challenging to obtain high-quality MR images. The super-resolution approach is potentially promising to improve MR image quality without any hardware upgrade. In this paper, we propose an ensemble learning and deep learning framework for MR image super-resolution. In our study, we first enlarged low resolution images using 5 commonly used super-resolution algorithms and obtained differentially enlarged image datasets with complementary priors. Then, a generative adversarial network (GAN) is trained with each dataset to generate super-resolution MR images. Finally, another GAN is used for ensemble learning that synergizes the outputs of GANs into the final MR super-resolution images. According to our results, the ensemble learning results outperform any single GAN output component. Compared with some state-of-the-art deep learning-based super-resolution methods, our approach is advantageous in suppressing artifacts and keeping more image details.
MR images scanned at low magnetic field (<1T) have lower resolution in the slice direction and lower contrast, due to a relatively small signal-to-noise ratio (SNR) than those from high field (typically 1.5T and 3T). We adapt the recent idea of Image Quality Transfer (IQT) to enhance very low-field structural images aiming to estimate the resolution, spatial coverage, and contrast of high-field images. Analogous to many learning-based image enhancement techniques, IQT generates training data from high-field scans alone by simulating low-field images through a pre-defined decimation model. However, the ground truth decimation model is not well-known in practice, and lack of its specification can bias the trained model, aggravating performance on the real low-field scans. In this paper we propose a probabilistic decimation simulator to improve robustness of model training. It is used to generate and augment various low-field images whose parameters are random variables and sampled from an empirical distribution related to tissue-specific SNR on a 0.36T scanner. The probabilistic decimation simulator is model-agnostic, that is, it can be used with any super-resolution networks. Furthermore we propose a variant of U-Net architecture to improve its learning performance. We show promising qualitative results from clinical low-field images confirming the strong efficacy of IQT in an important new application area: epilepsy diagnosis in sub-Saharan Africa where only low-field scanners are normally available.
Compressive sensing enables fast magnetic resonance imaging (MRI) reconstruction with undersampled k-space data. However, in most existing MRI reconstruction models, the whole MR image is targeted and reconstructed without taking specific tissue regions into consideration. This may fails to emphasize the reconstruction accuracy on important and region-of-interest (ROI) tissues for diagnosis. In some ROI-based MRI reconstruction models, the ROI mask is extracted by human experts in advance, which is laborious when the MRI datasets are too large. In this paper, we propose a deep neural network architecture for ROI MRI reconstruction called ROIRecNet to improve reconstruction accuracy of the ROI regions in under-sampled MRI. In the model, we obtain the ROI masks by feeding an initially reconstructed MRI from a pre-trained MRI reconstruction network (RecNet) to a pre-trained MRI segmentation network (ROINet). Then we fine-tune the RecNet with a binary weighted ℓ2 loss function using the produced ROI mask. The resulting ROIRecNet can offer more focus on the ROI. We test the model on the MRBrainS13 dataset with different brain tissues being ROIs. The experiment shows the proposed ROIRecNet can significantly improve the reconstruction quality of the region of interest.
The annihilating filter-based low-rank Hankel matrix approach (ALOHA) is one of the state-of-the-art compressed sensing approaches that directly interpolates the missing k-space data using low-rank Hankel matrix completion. The success of ALOHA is due to the concise signal representation in the k-space domain thanks to the duality between structured low-rankness in the k-space domain and the image domain sparsity. Inspired by the recent mathematical discovery that links convolutional neural networks to Hankel matrix decomposition using datadriven framelet basis, here we propose a fully data-driven deep learning algorithm for k-space interpolation. Our network can be also easily applied to non-Cartesian k-space trajectories by simply adding an additional regridding layer. Extensive numerical experiments show that the proposed deep learning method consistently outperforms the existing image-domain deep learning approaches.
Photoacoustic imaging is an emerging imaging modality that is based upon the photoacoustic effect. In photoacoustic tomography (PAT), the induced acoustic pressure waves are measured by an array of detectors and used to reconstruct an image of the initial pressure distribution. A common challenge faced in PAT is that the measured acoustic waves can only be sparsely sampled. Reconstructing sparsely sampled data using standard methods results in severe artifacts that obscure information within the image. We propose a modified convolutional neural network (CNN) architecture termed Fully Dense UNet (FD-UNet) for removing artifacts from 2D PAT images reconstructed from sparse data and compare the proposed CNN with the standard UNet in terms of reconstructed image quality.
Video super-resolution (VSR) has become one of the most critical problems in video processing. In the deep learning literature, recent works have shown the benefits of using adversarial-based and perceptual losses to improve the performance on various image restoration tasks; however, these have yet to be applied for video super-resolution. In this work, we propose a Generative Adversarial Network(GAN)-based formulation for VSR. We introduce a new generator network optimized for the VSR problem, named VSRResNet, along with a new discriminator architecture to properly guide VSRResNet during the GAN training. We further enhance our VSR GAN formulation with two regularizers, a distance loss in featurespace and pixel-space, to obtain our final VSRResFeatGAN model. We show that pre-training our generator with the Mean- Squared-Error loss only quantitatively surpasses the current state-of-the-art VSR models. Finally, we employ the PercepDist metric ([2]) to compare state-of-the-art VSR models. We show that this metric more accurately evaluates the perceptual quality of SR solutions obtained from neural networks, compared with the commonly used PSNR/SSIM metrics. Finally, we show that our proposed model, the VSRResFeatGAN model, outperforms current state-of-the-art SR models, both quantitatively and qualitatively.