No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Super Resolution in Medical Imaging
... It can offer the much-needed imaging modality or data for specific clinical purposes without the added expense or risk of conducting an actual acquisition. However, these types of cross-modality medical image-based translation or synthesis is much difficult to directly solve due to the ill-posed and high dimensionality nature of the mapping between the source image and the target image [5][6][7][8]. ...
Brain magnetic resonance imaging (MRI) offers intricate soft tissue contrasts that are essential for diagnosing diseases and conducting neuroscience research. At 7 Tesla (7T) magnetic field intensity, MRI enables increased resolution, enhanced tissue contrast, and improved SNR, compared to MRI collected from the commonly employed 3 Tesla (3T) MRI scanners. However, the exorbitant expenses associated with 7T MRI scanners hinder their broad use in research and clinical facilities. Efforts are underway to develop algorithms that can generate 7T MRI from 3T MRI to achieve better image quality without the need for 7T MRI machines. In this study, we have adopted a cycle consistent generative adversarial network (CycleGAN)-based approach for 3T MRI to 7T MRI translation, and vice versa, using a recently published dataset of paired T1-weighted MR images collected at 3T and 7T from a total of ten subjects. Various CycleGAN architectures were experimented with and compared on this dataset. The best performing CycleGAN architecture successfully produced the reconstructed images with a high level of accuracy based on different quantitative and qualitative evaluation criteria. Utilizing a post-processing technique, the best performing model generated 7T MRI from 3T MRI with a structural similarity index measure (SSIM) of 83.80%, peak SNR (PSNR) of 26.25, normalized mean squared error (NMSE) of 0.0088 and normalized mean absolute error (NMAE) of 0.0630. Utilizing CycleGAN to convert images from 3T to 7T MRI has shown a substantial improvement in MRI resolution, setting the stage for advancements in more informative and precise diagnostic imaging. INDEX TERMS Cycle consistent generative adversarial network, image-to-image translation, magnetic resonance imaging, paired dataset, T1-weighted MRI
High-resolution images are really helpful in various applications like medical diagnosis and hence the need for super-resolution has also increased significantly. Increasing the image resolution on various medical images like a chest X-ray or cell images can improve the accuracy of diagnosis by revealing previously unseen details. Using Image super-resolution also reduces the number of X-ray radiations required to render ultra high-quality imaging. Hence we applied super-resolution on X-rays using fine-tuned Swift-SRGAN architecture, which significantly improved the details on the chest X-rays. This helps in rendering super-resolution images from low-resolution images with less computational requirements. The proposed approach achieves a Structural Similarity Index Measure(SSIM) of 0.893 and a Peak Signal-to-Noise Ratio (PSNR) of 32.10. 2.
Magnetic resonance imaging (MRI) is an important medical imaging modality, but its acquisition speed is quite slow due to the physiological limitations. Recently, super-resolution methods have shown excellent performance in accelerating MRI. In some circumstances, it is difficult to obtain high-resolution images even with prolonged scan time. Therefore, we proposed a novel super-resolution method that uses a generative adversarial network with cyclic loss and attention mechanism to generate high-resolution MR images from low-resolution MR images by upsampling factors of 2× and 4×. We implemented our model on pelvic images from healthy subjects as training and validation data, while those data from patients were used for testing. The MR dataset was obtained using different imaging sequences, including T2, T2W SPAIR, and mDIXON-W. Four methods, i.e., BICUBIC, SRCNN, SRGAN, and EDSR were used for comparison. Structural similarity, peak signal to noise ratio, root mean square error, and variance inflation factor were used as calculation indicators to evaluate the performances of the proposed method. Various experimental results showed that our method can better restore the details of the high-resolution MR image as compared to the other methods. In addition, the reconstructed high-resolution MR image can provide better lesion textures in the tumor patients, which is promising to be used in clinical diagnosis.
In the realm of image processing, enhancing the quality of the images is known as a super-resolution problem (SR). Among SR methods, a super-resolution generative adversarial network, or SRGAN, has been introduced to generate SR images from low-resolution images. As it is of the utmost importance to keep the size and the shape of the images, while enlarging the medical images, we propose a novel super-resolution model with a generative adversarial network to generate SR images with finer details and higher quality to encourage less blurring. By widening residual blocks and using a self-attention layer, our model becomes robust and generalizable as it is able to extract the most important part of the images before up-sampling. We named our proposed model as wide-attention SRGAN (WA-SRGAN). Moreover, we have applied improved Wasserstein with a Gradient penalty to stabilize the model while training. To train our model, we have applied images from Camylon 16 database and enlarged them by 2×, 4×, 8×, and 16× upscale factors with the ground truth of the size of 256×256×3. Furthermore, two normalization methods, including batch normalization, and weight normalization have been applied and we observed that weight normalization is an enabling factor to improve metric performance in terms of SSIM. Moreover, several evaluation metrics, such as PSNR, MSE, SSIM, MS-SSIM, and QILV have been applied for having a comprehensive objective comparison with other methods, including SRGAN, A-SRGAN, and bicubial. Also, we performed the job of classification by using a deep learning model called ResNeXt-101 (32x8d) for super-resolution, high-resolution, and low-resolution images and compared the outcomes in terms of accuracy score. Finally, the results on breast cancer histopathology images show the superiority of our model by using weight normalization and a batch size of one in terms of restoration of the color and the texture details.
Generative Adversarial Models (GANs) have been quite popular and are currently and active area of research. They can be used for generative new data and study adversarial samples and attacks. We have used the similar approach to apply super-resolution to medical images. In Radiology MRI is a commonly used method to produce medical imaging but the limitations of lab equipment and health hazard of being in an MRI radiation environment to obtain good quality scans lead to lower quality scans and also it takes a lot of time to get a high-resolution data. This problem can be solved by using super-resolution using deep learning as a post-processing step to improve the resolution of the scans. Super-resolution is a process of generating higher resolution images from lower resolution data. For this, we are proposing a generative adversarial network architecture which is a dual neural network designed to generate lifelike images. In this deep learning algorithm, two neural networks compete with each other to improve alternatively. Given a training set, this technique learns to generate new data with the same statistics as the training set. To apply this technique to our problem statement we are using generator as the network to improve the resolution and discriminator as a network to train generator better. We used transfer learning in our generative neural network and training our discriminator from scratch and using the perceptual loss [1] to train our network. This will help in improving the performance of the network. We are using Lung MRI scans of tuberculosis with a set of 216 MRI samples containing around 60-130 channels each and each channel having 512x512 dimensions.
Super-resolution (SR) in medical imaging is an emerging application in medical imaging due to the needs of high quality images acquired with limited radiation dose, such as low dose Computer Tomography (CT), low field magnetic resonance imaging (MRI). However, because of its complexity and higher visual requirements of medical images, SR is still a challenging task in medical imaging. In this study, we developed a deep learning based method called Medical Images SR using Generative Adversarial Networks (MedSRGAN) for SR in medical imaging. A novel convolutional neural network, Residual Whole Map Attention Network (RWMAN) was developed as the generator network for our MedSRGAN in extracting the useful information through different channels, as well as paying more attention on meaningful regions. In addition, a weighted sum of content loss, adversarial loss, and adversarial feature loss were fused to form a multi-task loss function during the MedSRGAN training. 242 thoracic CT scans and 110 brain MRI scans were collected for training and evaluation of MedSRGAN. The results showed that MedSRGAN not only preserves more texture details but also generates more realistic patterns on reconstructed SR images. A mean opinion score (MOS) test on CT slices scored by five experienced radiologists demonstrates the efficiency of our methods.
The face super-resolution method is used for generating high-resolution images from low-resolution ones for better visualization. The super-resolution generative adversarial network (SRGAN) can generate a single super-resolution image with realistic textures, which is a groundbreaking work. Based on SRGAN, we proposed improved face super-resolution generative adversarial networks. The super-resolution image details generated by SRGAN usually have undesirable artifacts. To further improve visual quality, we delve into the key components of the SRGAN network architecture and improve each part to achieve a more powerful SRGAN. First, the SRGAN employs residual blocks as the core of the very deep generator network G. In this paper, we decide to employ dense convolutional network blocks (dense blocks), which connect each layer to every other layer in a feed-forward fashion as our very deep generator networks. Moreover, in the past few years, generative adversarial networks (GANs) have been applied to solve various problems. Despite its superior performance, it is difficult to train. A simple and effective regularization method called spectral normalization GAN is used to solve this problem. We have experimentally confirmed that our proposed method is superior to the other existing method in training stability and visual improvements.
Computed Tomography (CT) scanners that are commonly-used in hospitals and medical centers nowadays produce low-resolution images, e.g. one voxel in the image corresponds to at most one-cubic millimeter of tissue. In order to accurately segment tumors and make treatment plans, radiologists and oncologists need CT scans of higher resolution. The same problem appears in Magnetic Resonance Imaging (MRI). In this paper, we propose an approach for the single-image super-resolution of 3D CT or MRI scans. Our method is based on deep convolutional neural networks (CNNs) composed of 10 convolutional layers and an intermediate upscaling layer that is placed after the first 6 convolutional layers. Our first CNN, which increases the resolution on two axes (width and height), is followed by a second CNN, which increases the resolution on the third axis (depth). Different from other methods, we compute the loss with respect to the ground-truth high-resolution image right after the upscaling layer, in addition to computing the loss after the last convolutional layer. The intermediate loss forces our network to produce a better output, closer to the ground-truth. A widely-used approach to obtain sharp results is to add Gaussian blur using a fixed standard deviation. In order to avoid overfitting to a fixed standard deviation, we apply Gaussian smoothing with various standard deviations, unlike other approaches. We evaluate the proposed method in the context of 2D and 3D super-resolution of CT and MRI scans from two databases, comparing it to related works from the literature and baselines based on various interpolation schemes, using 2× and 4× scaling factors. The empirical study shows that our approach attains superior results to all other methods. Moreover, our subjective image quality assessment by human observers reveals that both doctors and regular annotators chose our method in favor of Lanczos interpolation in 97.55% cases for an upscaling factor of 2× and in 96.69% cases for an upscaling factor of 4×. In order to allow others to reproduce our state-of-the-art results, we provide our code as open source at https://github.com/lilygeorgescu/3d-super-res-cnn.
Details of small anatomical landmarks and pathologies, such as small changes of the microvasculature and soft exudates, are critical to accurate disease analysis. However, actual medical images always suffer from limited spatial resolution, due to imaging equipment and imaging parameters (e.g. scanning time of CT images). Recently, machine learning, especially deep learning techniques, have brought revolution to image super resolution reconstruction. Motivated by these achievements, in this paper, we propose a novel super resolution method for medical images based on an improved generative adversarial networks. To obtain useful image details as much as possible while avoiding the fake information in high frequency, the original squeeze and excitation block is improved by strengthening important features while weakening non-important ones. Then, by embedding the improved squeeze and excitation block in a simplified EDSR model, we build a new image super resolution network. Finally, a new fusion loss that can further strengthen the constraints on low-level features is designed for training our model. The proposed image super resolution model has been validated on the public medical images, and the results show that visual effects of the reconstructed images by our method, especially in the case of high upscaling factors, outperform state-of-the-art deep learning-based methods such as SRGAN, EDSR, VDSR and D-DBPN.
Generative adversarial networks (GANs) are used for image enhancement such as single image super-resolution (SISR) and deblurring. The conventional GANs-based image enhancement suffers from two drawbacks that cause a quality degradation due to a loss of detailed information. First, the conventional discriminator network adopts strided convolution layers which cause a reduction in the resolution of the feature map, and thereby resulting in a loss of detailed information. Second, the previous GANs for image enhancement use the feature map of the visual geometry group (VGG) network for generating a content loss, which also causes visual artifacts because the maxpooling layers in the VGG network result in a loss of detailed information. To overcome these two drawbacks, this paper presents a proposal of a new resolutionpreserving discriminator network architecture which removes the strided convolution layers, and a new content loss generated from the VGG network without maxpooling layers. The proposed discriminator network is applied to the super-resolution generative adversarial network (SRGAN), which is called a resolution-preserving SRGAN (RPSRGAN). Experimental results show that RPSRGAN generates more realistic super-resolution images than SRGAN does, and consequently, RPSRGAN with the new content loss improves the average peak signal-to-noise ratio (PSNR) by 0.75 dB and 0.32 dB for super-resolution images with the scale factors of 2 and 4, respectively. For deblurring, the visual appearance is also significantly improved, and the average PSNR is increased by 1.54 dB when the proposed discriminator and content loss are applied to the deblurring adversarial network.
In this work we propose an adversarial learning approach to generate high resolution MRI scans from low resolution images. The architecture, based on the SRGAN model, adopts 3D convolutions to exploit volumetric information. For the discriminator, the adversarial loss uses least squares in order to stabilize the training. For the generator, the loss function is a combination of a least squares adversarial loss and a content term based on mean square error and image gradients in order to improve the quality of the generated images. We explore different solutions for the upsampling phase. We present promising results that improve classical interpolation, showing the potential of the approach for 3D medical imaging super-resolution.
High-resolution (HR) magnetic resonance images (MRI) provide detailed anatomical information important for clinical application and quantitative image analysis. However, HR MRI conventionally comes at the cost of longer scan time, smaller spatial coverage, and lower signal-to-noise ratio (SNR). Recent studies have shown that single image super-resolution (SISR), a technique to recover HR details from one single low-resolution (LR) input image, could provide high quality image details with the help of advanced deep convolutional neural networks (CNN). However, deep neural networks consume memory heavily and run slowly, especially in 3D settings. In this paper, we propose a novel 3D neural network design, namely a multi-level densely connected super-resolution network (mDCSRN) with generative adversarial network (GAN)–guided training. The mDCSRN trains and inferences quickly, and the GAN promotes realistic output hardly distinguishable from original HR images. Our results from experiments on a dataset with 1,113 subjects shows that our new architecture outperforms other popular deep learning methods in recovering 4x resolution-downgraded images and runs 6x faster.
Recent attempts at Super-Resolution for medical images used deep learning techniques such as Generative Adversarial Networks (GANs) to achieve perceptually realistic single image Super-Resolution. Yet, they are constrained by their inability to generalise to different scale factors. This involves high storage and energy costs as every integer scale factor involves a separate neural network. A recent paper has proposed a novel meta-learning technique that uses a Weight Prediction Network to enable Super-Resolution on arbitrary scale factors using only a single neural network. In this paper, we propose a new network that combines that technique with SRGAN, a state-of-the-art GAN-based architecture, to achieve arbitrary scale, high fidelity Super-Resolution for medical images. By using this network to perform arbitrary scale magnifications on images from the Multimodal Brain Tumor Segmentation Challenge (BraTS) dataset, we demonstrate that it is able to outperform traditional interpolation methods by up to 20 on SSIM scores whilst retaining generalisability on brain MRI images. We show that performance across scales is not compromised, and that it is able to achieve competitive results with other state-of-the-art methods such as EDSR whilst being fifty times smaller than them. Combining efficiency, performance, and generalisability, this can hopefully become a new foundation for tackling Super-Resolution on medical images.
The Super-Resolution Generative Adversarial Network (SRGAN) is a seminal work that is capable of generating realistic textures during single image super-resolution. However, the hallucinated details are often accompanied with unpleasant artifacts. To further enhance the visual quality, we thoroughly study three key components of SRGAN – network architecture, adversarial loss and perceptual loss, and improve each of them to derive an Enhanced SRGAN (ESRGAN). In particular, we introduce the Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit. Moreover, we borrow the idea from relativistic GAN to let the discriminator predict relative realness instead of the absolute value. Finally, we improve the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery. Benefiting from these improvements, the proposed ESRGAN achieves consistently better visual quality with more realistic and natural textures than SRGAN and won the first place in the PIRM2018-SR Challenge (region 3) with the best perceptual index. The code is available at https://github.com/xinntao/ESRGAN.
Edge Profile Super Resolution
- Lee Jiun
- Yun Inyong
- Kim Jaekwang