Conference Paper

MATNet: MRI Super-Resolution with Multiple Attention Mechanisms

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Magnetic resonance imaging (MRI) provides high spatial resolution and excellent soft-tissue contrast without using harmful ionising radiation. Dynamic MRI is an essential tool for interventions to visualise movements or changes of the target organ. However, such MRI acquisitions with high temporal resolution suffer from limited spatial resolution - also known as the spatio-temporal trade-off of dynamic MRI. Several approaches, including deep learning based super-resolution approaches by treating each timepoint as individual volumes. This research addresses this issue by creating a deep learning model which attempts to learn both spatial and temporal relationships. A modified 3D UNet model, DDoS-UNet, is proposed - which takes the low-resolution volume of the current timepoint along with a prior image volume. Initially, the network is supplied with a static high-resolution planning scan as the prior image along with the low-resolution input to super-resolve the first timepoint. Then it continues step-wise by using the super-resolved timepoints as the prior image while super-resolving the subsequent timepoints. The model performance was tested with 3D dynamic data that was undersampled to different in-plane levels and achieved an average SSIM value of 0.951±0.017 while reconstructing only 4% of the k-space - which could result in a theoretical acceleration factor of 25. The proposed approach can be used to reduce the required scan-time while achieving high spatial resolution - consequently alleviating the spatio-temporal trade-off of dynamic MRI, by incorporating prior knowledge of spatio-temporal information from the available high-resolution planning scan and the existing temporal redundancy of time-series images into the network model.
Article
Full-text available
Although robust PCA has been increasingly adopted to extract vessels from X-ray coronary angiography (XCA) images, challenging problems such as inefficient vessel-sparsity modelling, noisy and dynamic background artefacts, and high computational cost still remain unsolved. Therefore, we propose a novel robust PCA unrolling network with sparse feature selection for super-resolution XCA vessel imaging. Being embedded within a patch-wise spatiotemporal super-resolution framework that is built upon a pooling layer and a convolutional long short-term memory network, the proposed network can not only gradually prune complex vessel-like artefacts and noisy backgrounds in XCA during network training but also iteratively learn and select the high-level spatiotemporal semantic information of moving contrast agents flowing in the XCA-imaged vessels. The experimental results show that the proposed method significantly outperforms state-of-the-art methods, especially in the imaging of the vessel network and its distal vessels, by restoring the intensity and geometry profiles of heterogeneous vessels against complex and dynamic backgrounds. The source code is available at https://github.com/Binjie-Qin/RPCA-UNet</uri
Conference Paper
Full-text available
Dynamic MRI is an essential tool for interventions to visualise movements or changes in the target organ. However, such MRI acquisition with high temporal resolution suffers from limited spatial resolution-also known as the spatio-temporal trade-off. Several approaches, including deep learning based super-resolution approaches, have been proposed to mitigate this trade-off. Nevertheless, such an approach typically aims to super-resolve each time-point separately, treating them as individual volumes. This research addresses the problem by creating a deep learning model that attempts to learn spatial and temporal relationships. The performance was tested with 3D dynamic data that was undersampled to different in-plane levels. The proposed network achieved an average SSIM value of 0.951±0.017 while reconstructing the lowest resolution data (i.e. only 4% of the k-space acquired), resulting in a theoretical acceleration factor of 25.
Article
Full-text available
This paper proposes aconvolutional neural network (CNN)-based efficient medical image super-resolution (SR) method in the shearlet domain. Because of differences between imaging mechanisms optimized for natural images and medical images, the design begins with building a medical image dataset for medical image SR and extracting effective areas to remarkably enhance the training effects of the CNN-based method. Then, a new medical image SR network structure—deep medical super-resolution network (DMSRN)—has been designed in which local residual learning is implemented through a recursive network and combined with global residual learning to heighten the depth of the network on the ground with no parameter increase. This effectively fixes the long-term dependency problem, which causes the prior state layers to barely have any effect on the following state layers. Last, the design addresses the problem of too-smooth reconstruction effects in the CNN-based method in the image space domain; shearlet transform is introduced to DMSRN to restore global topology through low-frequency sub-bands and restore local edge detail information through high-frequency sub-bands. Experimental results show that the proposed method is better than other state-of-the-art methods for medical image SR, which significantly promotes the restoration ability of texture structure and edge details.
Article
Full-text available
The diagnosis of brain pathologies usually involves imaging to analyze the condition of the brain. Magnetic resonance imaging (MRI) technology is widely used in brain disorder diagnosis. The image quality of MRI depends on the magnetostatic field strength and scanning time. Scanners with lower field strengths have the disadvantages of a low resolution and high imaging cost, and scanning takes a long time. The traditional super-resolution reconstruction method based on MRI generally states an optimization problem in terms of prior information. It solves the problem using an iterative approach with a large time cost. Many methods based on deep learning have emerged to replace traditional methods. MRI super-resolution technology based on deep learning can effectively improve MRI resolution through a three-dimensional convolutional neural network; however, the training costs are relatively high. In this paper, we propose the use of two-dimensional super-resolution technology for the super-resolution reconstruction of MRI images. In the first reconstruction, we choose a scale factor of 2 and simulate half the volume of MRI slices as input. We utilize a receiving field block enhanced super-resolution generative adversarial network (RFB-ESRGAN), which is superior to other super-resolution technologies in terms of texture and frequency information. We then rebuild the super-resolution reconstructed slices in the MRI. In the second reconstruction, the image after the first reconstruction is composed of only half of the slices, and there are still missing values. In our previous work, we adopted the traditional interpolation method, and there was still a gap in the visual effect of the reconstructed images. Therefore, we propose a noise-based super-resolution network (nESRGAN). The noise addition to the network can provide additional texture restoration possibilities. We use nESRGAN to further restore MRI resolution and high-frequency information. Finally, we achieve the 3D reconstruction of brain MRI images through two super-resolution reconstructions. Our proposed method is superior to 3D super-resolution technology based on deep learning in terms of perception range and image quality evaluation standards.
Article
Full-text available
General image super-resolution techniques have difficulties in recovering detailed face structures when applying to low resolution face images. Recent deep learning based methods tailored for face images have achieved improved performance by jointly trained with additional task such as face parsing and landmark prediction. However, multi-task learning requires extra manually labeled data. Besides, most of the existing works can only generate relatively low resolution face images ( e.g ., 128×128128\times 128 ), and their applications are therefore limited. In this paper, we introduce a novel SPatial Attention Residual Network (SPARNet) built on our newly proposed Face Attention Units (FAUs) for face super-resolution. Specifically, we introduce a spatial attention mechanism to the vanilla residual blocks. This enables the convolutional layers to adaptively bootstrap features related to the key face structures and pay less attention to those less feature-rich regions. This makes the training more effective and efficient as the key face structures only account for a very small portion of the face image. Visualization of the attention maps shows that our spatial attention network can capture the key face structures well even for very low resolution faces ( e.g ., 16×1616\times 16 ). Quantitative comparisons on various kinds of metrics (including PSNR, SSIM, identity similarity, and landmark detection) demonstrate the superiority of our method over current state-of-the-arts. We further extend SPARNet with multi-scale discriminators, named as SPARNetHD, to produce high resolution results ( i.e ., 512×512512\times 512 ). We show that SPARNetHD trained with synthetic data can not only produce high quality and high resolution outputs for synthetically degraded face images, but also show good generalization ability to real world low quality face images. Codes are available at https://github.com/chaofengc/Face-SPARNet .
Conference Paper
Full-text available
This paper presents a generative model for super-resolution in routine clinical magnetic resonance images (MRI), of arbitrary orientation and contrast. The model recasts the recovery of high resolution images as an inverse problem, in which a forward model simulates the slice-select profile of the MR scanner. The paper introduces a prior based on multi-channel total variation for MRI super-resolution. Bias-variance trade-off is handled by estimating hyper-parameters from the low resolution input scans. The model was validated on a large database of brain images. The validation showed that the model can improve brain segmentation, that it can recover anatomical information between images of different MR contrasts, and that it generalises well to the large variability present in MR images of different subjects.
Article
Full-text available
Objective: In magnetic resonance imaging (MRI), the super-resolution technology has played a great role in improving image quality. The aim of this work is to improve edges of brain MRI by incorporating the gradient information of another contrast high-resolution image. Methods: Multi-contrast images are assumed to possess the same gradient direction in a local pattern. We proposed to establish a relation model of gradient value between different contrast images, to restore a high-resolution image from its input low-resolution version. The similarity of image patches is employed to estimate intensity parameters, leading a more accurate reconstructed image. Then, iterative back-projection filter is applied to the reconstructed image to further increase image quality. Results: The new approach is verified on synthetic and real brain MRI images, and achieves higher visual quality and higher objective quality criteria than the compared state-of-the-art super-resolution approaches. Conclusion: The gradient information of the multi-contrast MRI images is very useful. With a proper relation model, the proposed method enhances image edges in MRI image super-resolution. Significance: Improve the MRI image resolution from very low-resolution observations is challenging. We tackle this problem by first modeling the relation of gradient value in multi-contrast MRI and then performing fast supper-resolution methods. This relation model may be helpful for other MRI reconstruction problems.
Article
Full-text available
In this paper, a single-computed tomography (CT) image super-resolution (SR) reconstruction scheme is proposed. This SR reconstruction scheme is based on sparse representation theory and dictionary learning of low- and high-resolution image patch pairs to improve the poor quality of low-resolution CT images obtained in clinical practice using low-dose CT technology. The proposed strategy is based on the idea that image patches can be well represented by sparse coding of elements from an overcomplete dictionary. To obtain similarity of the sparse representations, two dictionaries of low- and high-resolution image patches are jointly trained. Then, sparse representation coefficients extracted from the low-resolution input patches are used to reconstruct the high-resolution output. Sparse representation is used such that the trained dictionary pair can reduce computational costs. Combined with several appropriate iteration operations, the reconstructed high-resolution image can attain better image quality. The effectiveness of the proposed method is demonstrated using both clinical CT data and simulation image data. Image quality evaluation indexes (root mean squared error (RMSE) and peak signal-to-noise ratio (PSNR)) indicate that the proposed method can effectively improve the resolution of a single CT image.
Article
Full-text available
Magnetic resonance image (MRI) in high spatial resolution provides detailed anatomical information and is often necessary for accurate quantitative analysis. However, high spatial resolution typically comes at the expense of longer scan time, less spatial coverage, and lower signal to noise ratio (SNR). Single Image Super-Resolution (SISR), a technique aimed to restore high-resolution (HR) details from one single low-resolution (LR) input image, has been improved dramatically by recent breakthroughs in deep learning. In this paper, we introduce a new neural network architecture, 3D Densely Connected Super-Resolution Networks (DCSRN) to restore HR features of structural brain MR images. Through experiments on a dataset with 1,113 subjects, we demonstrate that our network outperforms bicubic interpolation as well as other deep learning methods in restoring 4x resolution-reduced images.
Article
Full-text available
We propose a deep learning method for single image super-resolution (SR). Our method directly learns an end-to-end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art restoration quality, and achieves fast speed for practical on-line usage. We explore different network structures and parameter settings to achieve trade-offs between performance and speed. Moreover, we extend our network to cope with three color channels simultaneously, and show better overall reconstruction quality.
Article
Full-text available
Laser ablation electrospray ionization (LAESI) is a novel method for the direct imaging of biological tissues by mass spectrometry. By performing ionization in the ambient environment, this technique enables in vivo studies with potential for single-cell analysis. A unique aspect of LAESI mass spectrometric imaging (MSI) is depth profiling that, in combination with lateral imaging, permits 3D molecular imaging for the first time under native conditions. With current lateral and depth resolutions of approximately 100 and approximately 40 microm, respectively, LAESI MSI helps to explore the molecular architecture of live tissues.
Chapter
Unsupervised real world super resolution (USR) aims to restore high-resolution (HR) images given low-resolution (LR) inputs, and its difficulty stems from the absence of paired dataset. One of the most common approaches is synthesizing noisy LR images using GANs (i.e., degradation generators) and utilizing a synthetic dataset to train the model in a supervised manner. Although the goal of training the degradation generator is to approximate the distribution of LR images given a HR image, previous works have heavily relied on the unrealistic assumption that the conditional distribution is a delta function and learned the deterministic mapping from the HR image to a LR image. In this paper, we show that we can improve the performance of USR models by relaxing the assumption and propose to train the probabilistic degradation generator. Our probabilistic degradation generator can be viewed as a deep hierarchical latent variable model and is more suitable for modeling the complex conditional distribution. We also reveal the notable connection with the noise injection of StyleGAN. Furthermore, we train multiple degradation generators to improve the mode coverage and apply collaborative learning for ease of training. We outperform several baselines on benchmark datasets in terms of PSNR and SSIM and demonstrate the robustness of our method on unseen data distribution. Code is available at https://github.com/sangyun884/MSSR.
Article
Hyperspectral image produces high spectral resolution at the sacrifice of spatial resolution. Without reducing the spectral resolution, improving the resolution in the spatial domain is a very challenging problem. Motivated by the discovery that hyperspectral image exhibits high similarity between adjacent bands in a large spectral range, in this paper, we explore a new structure for hyperspectral image super-resolution (DualSR), leading to a dual-stage design, i.e., coarse stage and fine stage. In coarse stage, five bands with high similarity in a certain spectral range are divided into three groups, and the current band is guided to study the potential knowledge. Under the action of alternative spectral fusion mechanism, the coarse SR image is super-resolved in band-by-band. In order to build model from a global perspective, an enhanced back-projection method via spectral angle constraint is developed in fine stage to learn the content of spatial-spectral consistency, dramatically improving the performance gain. Extensive experiments demonstrate the effectiveness of the proposed coarse stage and fine stage. Besides, our network produces state-of-the-art results against existing works in terms of spatial reconstruction and spectral fidelity. Our code is publicly available at https://github.com/qianngli/DualSR .
Chapter
Recently, transformer-based methods have demonstrated impressive results in various vision tasks, including image super-resolution (SR), by exploiting the self-attention (SA) for feature extraction. However, the computation of SA in most existing transformer based models is very expensive, while some employed operations may be redundant for the SR task. This limits the range of SA computation and consequently limits the SR performance. In this work, we propose an efficient long-range attention network (ELAN) for image SR. Specifically, we first employ shift convolution (shift-conv) to effectively extract the image local structural information while maintaining the same level of complexity as 1×\,\times \,1 convolution, then propose a group-wise multi-scale self-attention (GMSA) module, which calculates SA on non-overlapped groups of features using different window sizes to exploit the long-range image dependency. A highly efficient long-range attention block (ELAB) is then built by simply cascading two shift-conv with a GMSA module, which is further accelerated by using a shared attention mechanism. Without bells and whistles, our ELAN follows a fairly simple design by sequentially cascading the ELABs. Extensive experiments demonstrate that ELAN obtains even better results against the transformer-based SR models but with significantly less complexity. The source codes of ELAN can be found at https://github.com/xindongzhang/ELAN.KeywordsSuper-resolutionLong-range attentionTransformer
Article
Multimodal medical images are widely used by clinicians and physicians to analyze and retrieve complementary information from high-resolution images in a non-invasive manner. Loss of corresponding image resolution adversely affects the overall performance of medical image interpretation. Deep learning-based single image super resolution (SISR) algorithms have revolutionized the overall diagnosis framework by continually improving the architectural components and training strategies associated with convolutional neural networks (CNN) on low-resolution images. However, existing work lacks in two ways: i) the SR output produced exhibits poor texture details, and often produce blurred edges, ii) most of the models have been developed for a single modality, hence, require modification to adapt to a new one. This work addresses (i) by proposing generative adversarial network (GAN) with deep multi-attention modules to learn high-frequency information from low-frequency data. Existing approaches based on the GAN have yielded good SR results; however, the texture details of their SR output have been experimentally confirmed to be deficient for medical images particularly. The integration of wavelet transform (WT) and GANs in our proposed SR model addresses the aforementioned limitation concerning textons. While the WT divides the LR image into multiple frequency bands, the transferred GAN uses multi-attention and upsample blocks to predict high-frequency components. Additionally, we present a learning method for training domain-specific classifiers as perceptual loss functions. Using a combination of multi-attention GAN loss and a perceptual loss function results in an efficient and reliable performance. Applying the same model for medical images from diverse modalities is challenging, our work addresses (ii) by training and performing on several modalities via transfer learning. Using two medical datasets, we validate our proposed SR network against existing state-of-the-art approaches and achieve promising results in terms of structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR).
Chapter
The core problem of Magnetic Resonance Imaging (MRI) is the trade off between acceleration and image quality. Image reconstruction and super-resolution are two crucial techniques in Magnetic Resonance Imaging (MRI). Current methods are designed to perform these tasks separately, ignoring the correlations between them. In this work, we propose an end-to-end task transformer network (T2^2Net) for joint MRI reconstruction and super-resolution, which allows representations and feature transmission to be shared between multiple task to achieve higher-quality, super-resolved and motion-artifacts-free images from highly undersampled and degenerated MRI data. Our framework combines both reconstruction and super-resolution, divided into two sub-branches, whose features are expressed as queries and keys. Specifically, we encourage joint feature learning between the two tasks, thereby transferring accurate task information. We first use two separate CNN branches to extract task-specific features. Then, a task transformer module is designed to embed and synthesize the relevance between the two tasks. Experimental results show that our multi-task model significantly outperforms advanced sequential methods, both quantitatively and qualitatively.
Chapter
We propose Convolutional Block Attention Module (CBAM), a simple yet effective attention module for feed-forward convolutional neural networks. Given an intermediate feature map, our module sequentially infers attention maps along two separate dimensions, channel and spatial, then the attention maps are multiplied to the input feature map for adaptive feature refinement. Because CBAM is a lightweight and general module, it can be integrated into any CNN architectures seamlessly with negligible overheads and is end-to-end trainable along with base CNNs. We validate our CBAM through extensive experiments on ImageNet-1K, MS COCO detection, and VOC 2007 detection datasets. Our experiments show consistent improvements in classification and detection performances with various models, demonstrating the wide applicability of CBAM. The code and models will be publicly available.
Conference Paper
This paper introduces new algorithms for enhancing the resolution of interpolated images. The aim of this algorithm is to have a high resolution image from a single low resolution image. The new algorithms used the interpolation technique with the two dimensional filter that is specifically designed for the image by maximizing the peak signal to noise ratio and structural similarity. The processed images are assessed by comparing them with the ground truth images. The algorithm performance is assessed using the peak signal to noise ratio (PSNR) and structural similarity (SSIM). The new algorithm gives better result than the previous 2D filters algorithms.
Medical image super-resolution method based on dense blended attention network
  • K Liu
  • Y Ma
  • H Xiong
  • Z Yan
  • Z Zhou
  • P Fang
Global attention mechanism: Retain information to enhance channel-spatial interactions
  • Y Liu
  • Z Shao
  • N Hoffmann
Bam: Bottleneck attention module
  • J Park
  • S Woo
  • J.-Y Lee
  • I S Kweon
Cbam: Convolutional block attention module[C]
  • S Woo
  • J Y Park
  • Lee
Practical blind denoising via swin-convunet and data synthesis
  • Zhang
Global attention mechanism: Retain information to enhance channel-spatial interactions
  • Liu
Medical image super-resolution method based on dense blended attention network
  • Liu