ArticlePDF Available

Abstract and Figures

Image super-resolution (SR) aims to recover high-resolution images from their low-resolution counterparts for improving image analysis and visualization. Interpolation methods, widely used for this purpose, often result in images with blurred edges and blocking effects. More advanced methods such as total variation (TV) retain edge sharpness during image recovery. However, these methods only utilize information from local neighborhoods, neglecting useful information from remote voxels. In this paper, we propose a novel image SR method that integrates both local and global information for effective image recovery. This is achieved by, in addition to TV, low-rank regularization that enables utilization of information throughout the image. The optimization problem can be solved effectively via alternating direction method of multipliers (ADMM). Experiments on MR images of both adult and pediatric subjects demonstrate that the proposed method enhances the details in the recovered high-resolution images, and outperforms methods such as the nearest-neighbor interpolation, cubic interpolation, iterative back projection (IBP), non-local means (NLM), and TV-based up-sampling.
Content may be subject to copyright.
IEEE TMI-2015-0061
AbstractImage super-resolution (SR) aims to recover
high-resolution images from their low-resolution counterparts for
improving image analysis and visualization. Interpolation
methods, widely used for this purpose, often result in images with
blurred edges and blocking effects. More advanced methods such
as total variation (TV) retain edge sharpness during image
recovery. However, these methods only utilize information from
local neighborhoods, neglecting useful information from remote
voxels. In this paper, we propose a novel image SR method that
integrates both local and global information for effective image
recovery. This is achieved by, in addition to TV, low-rank
regularization that enables utilization of information throughout
the image. The optimization problem can be solved effectively via
alternating direction method of multipliers (ADMM).
Experiments on MR images of both adult and pediatric subjects
demonstrate that the proposed method enhances the details in the
recovered high-resolution images, and outperforms methods such
as the nearest-neighbor interpolation, cubic interpolation,
iterative back projection (IBP), non-local means (NLM), and
TV-based up-sampling.
Index Terms Image enhancement, spatial resolution, image
sampling, matrix completion, sparse learning
IGH-resolution (HR) medical images provide rich
structural details that are critical for accurate image
post-processing and pathological assessment of bodily organs
[1-10]. However, image resolution is limited by factors such as
imaging hardware, signal to noise ratio (SNR), and time
constraints. Image SNR is proportional to voxel size and the
Manuscript received January 22, 2015; accepted May 17, 2015. This work
was supported in part by National Institutes of Health grants MH100217,
EB006733, EB008374, EB009634, AG041721, AG042599, MH088520, and
the National Research Foundation under Grant 2012-005741 from the Korean
F. Shi and J. Cheng contributed equally to this work.
corresponding author.
F. Shi, L. Wang, and P.-T. Yap are with the Department of Radiology and
Biomedical Research Imaging Center, University of North Carolina, Chapel
Hill, NC 27599, USA (e-mails: fengshi, li_wang,
J. Cheng is with Department of Radiology and Biomedical Research
Imaging Center, University of North Carolina, Chapel Hill, NC 27599, USA,
and also with Section on Tissue Biophysics and Biomimetics, PPITS, National
Institute of Child Health and Human Development, NIH, Bethesda, MD, USA
D. Shen is with the Department of Radiology and Biomedical Research
Imaging Center (BRIC), University of North Carolina at Chapel Hill, NC
27599 USA, and also with the Department of Brain and Cognitive Engineering,
Korea University, Seoul 136-713, Korea (
square root of the number of averages in the voxel. Reducing
the voxel size from 2 mm
to 1 mm
will require 64
averages for similar SNR [11]. This requires significantly
longer scanning time, which may not be practical clinically.
A possible alternative approach to this problem is image
post-processing. For this, interpolation methods (nearest
neighbor, linear, and spine) are generally employed due to their
simplicity. However, as pointed out in [1], interpolation
methods generally blur the sharp edges, introduce blocking
artifacts in lines, and are unable to recover fine details. In view
of this, we take a super-resolution (SR) approach for resolution
enhancement of LR images [3]. Interpolation methods are not
considered as SR methods since they do not consider the image
degradation process (e.g., blurring, and down-sampling).
Multi-frame SR algorithms reconstruct a HR image from
multiple LR images [12, 13]. These LR images are typically
acquired repeatedly with slightly shifted field of view (FOV).
In this paper, we propose a single-frame SR algorithm that
requires the acquisition of only one LR image. A number of
methods have been proposed for single-image SR [3]. For
example, iterative back-projection (IBP) was proposed to
estimate the HR image by back projecting the difference
between the LR image simulated based on the estimated HR
image via imaging blur and the input LR image [14]. This
process is repeated to minimize the energy of the difference.
Non-local means (NLM) is a method proposed to take
advantage of image self-similarity [15]. Specifically, the input
LR image is first denoised and the similar patches are used to
reconstruct to a HR image. A correction step is then applied to
ensure that the down-sampled HR image is close to the
denoised LR image. The reconstruction and correction steps are
iterated in a multi-scale manner. In another work, NLM was
employed to enhance the resolution of a single LR T2 image
with the guidance from an HR T1 image [16].
Matrix completion algorithms have recently been shown to
be effective in estimating missing values in a matrix from a
small sample of known entries [17-19]. For instance, it has been
applied to the famous Netflix problem where one needs to infer
user preference for unrated movies based on only a small
number of rated movies [20]. Matrix completion methods
assume that the recovered matrix has low rank and then uses
this property as a constraint or regularization to minimize the
difference between the given incomplete matrix and the
estimated matrix. Candes et al. proved that, low-rank matrices
can be perfectly recovered from a small number of given entries
LRTV: MR Image Super-Resolution with
Low-Rank and Total Variation Regularizations
Feng Shi
, Jian Cheng
, Li Wang, Pew-Thian Yap, Senior Member, IEEE, and Dinggang Shen
, Senior
Member, IEEE
IEEE TMI-2015-0061
under some conditions [18]. In image processing, one
advantage of matrix completion is that the remote information
from the whole image can be utilized in image recovery.
Besides, matrix completion has been widely applied to
image/video in-painting and decoding problems. However,
low-rank regularization is limited when the matrix to be
recovered contain rows or columns that are entirely missing. As
shown in Fig. 1, although voxels occluded by the words “Low
Rank” can be recovered very well, the voxels masked by white
horizontal and vertical lines cannot be recovered. In this case,
each missing row or column will be simply filled by an
arbitrary combination of other known rows or columns to meet
the low rank requirement. This precludes the application of
low-rank regularization to SR problem where missing values
for many rows and columns need to be recovered in the process
of recovering a HR image from a LR image.
The limitation of low-rank regularization in SR problem
could be remedied by imposing the additional local spatial
consistency. Although local information may not be useful in
applications such as the Netflix problem, where different rows
(e.g., users) can be considered independently, it is valuable in
recovering images. One possible choice is total variation (TV)
[21], defined as the integral of the absolute gradients of the
image. Recently, TV has been applied to avoid the ringing and
smearing caused by interpolation [22].
In this paper, we propose a novel low-rank total variation
method, referred to as LRTV, for recovering a HR image from a
LR image. Our method 1) explicitly models the blurring effects
when an image is down-sampled, 2) combines both low-rank
and TV regularizations for more effective image recovery, and
3) works for 3D or 4D images by a tensor formulation [19]. A
preliminary version of this work was presented at a conference
[23]. The work is significantly extended in this article with
more methodological details, validations, and discussions.
Experiments on MR images of both adults and pediatric
subjects are conducted, and the proposed method is compared
to various interpolation methods, as well as the NLM and
TV-based up-sampling methods.
We first describe how image degradation processes such as
blurring and down-sampling effects are modeled. We then
describe the solution for the inverse problem of recovering the
HR image from the LR image, using low-rank and TV
A. Super-Resolution Image Reconstruction Framework
As illustrated in Fig. 2, the acquired image is affected by
factors such as motion blur, field inhomogeneity, acquisition
time, and noise.
The observation model could be mathematically formulated
   (1)
where T denotes the observed LR image, D is a down-sampling
operator, S is a blurring operator, X is the HR image that we
want to recover, and n represents the observation noise. In the
case of single-image SR, the HR image can be estimated using
this observation model by minimizing the following cost
 
  
This is a data fidelity term for penalizing the difference
between the degraded HR image X and the observed LR image
T. Since this is an ill-posed inverse problem, regularization
terms are added to stabilize the solution. The cost function is
thus rewritten as:
 
  
 
is the regularization term often defined based on
prior knowledge. The parameter λ is used to balance the
contributions of the fidelity term and the regularization term.
B. Low-Rank Total-Variation (LRTV) Method
The proposed LRTV method is formulated as follows:
 
  
 
  
where the regularization can be separated into low-rank and
total variation terms.
are the respective tuning
parameters for those two terms.
Fig. 1. Recovering the missing values using low-rank matrix completion [14].
The red arrows mark the horizontal and vertical lines that the algorithm fails to
Fig. 2. Observation model in MR image acquisition process.
IEEE TMI-2015-0061
1) Low-Rank Regularization. The rank of a matrix is a
measure of nondegenerateness of the matrix, calculated by the
maximum number of linearly independent rows or columns in
the matrix. The low-rank property implies that some rows or
columns in the matrix can be linearly represented by other rows
or columns, indicating redundant information in the matrix.
Low-rank prior can be used in matrix completion when only a
subset of elements is known [18]. Since the rank of a matrix
is a nonconvex function of , a common approach is to
approximate it using the trace norm
, which leads to a
convex optimization problem. Recently, Liu et al. extended the
low-rank regularization to higher dimensional images and
further referred to as tensor completion [19]. Basically, a
N-dimensional image
can be seen as a high-order tensor.
Since it is an NP-hard problem to compute the rank of a
high-order tensor
, the rank is then approximated as the
combination of trace norms of all matrices unfolded along each
where is the number of image dimensionality. In this study,
we use 3D MR images and thus .
are parameters
is the unfolded along
the -th dimension: 
. For example, a 3D
image with size of   can be unfolded into three 2D
matrices, with sizes of   ,   , and
  , respectively. 
is the trace norm of the matrix
2) Total-Variation Regularization. Total variation was
proposed as a regularization approach to remove noise and
handle proper edges in images [21]. It is defined as the integral
of the absolute gradients of an image: 
Minimizing TV will enforce local spatial consistency in image
recovery, remove noise, and preserve edges.
C. LRTV Optimization
We use the alternating direction method of multipliers
(ADMM) algorithm to solve the cost function in Eq. (4).
ADMM is proven to be efficient for solving optimization
problems with multiple non-smooth terms in the cost function
[24]. First, we introduce redundant variables
simulate in each dimension , by requiring that the unfolded
along the -th dimension
should be equal to the unfolded
along this dimension
. The new cost function is as
  
 
, subject to
,  (6)
Based on ADMM in [24], the augmented Lagrangian of the
above cost function is written in below, where
Lagrangian parameters:
  
 
 
 
We further set
, and combine the last two linear
and quadratic terms for simplicity:
  
 
 
 
According to ADMM [24], we break Eq. (8) into three
sub-problems below that could be solved by iteratively
updating the variables. In the below, denotes the current
iteration step.
Subproblem 1: Update
by minimizing:
  
  
This subproblem can be solved by gradient descent, where
the gradient of TV term is obtained from the associated
Euler-Lagrange equation [22].
Subproblem 2: Update 
by minimizing:
which can be solved using a close-form solution according to
 
 (11)
where 
 is the inverse operator of 
 , i.e.,
. 
is the Singular Value Thresholding
operator [17] using
as the shrinkage parameter.
Subproblem 3: Update 
 
 
Algorithm 1. Low-Rank Total Variation (LRTV) for MR Image
Input: Low-resolution image T;
Output: Reconstructed high-resolution image X;
Initialize: 
, 
1. Update X based on Eq. (9);
2. Update M based on Eq. (11);
3. Update Y based on Eq. (12);
4. Until difference in the cost function (Eq. (8)) is less than ;
The 
operator is implemented by nearest-neighbor
IEEE TMI-2015-0061
A. Low-Rank Approximation for Brain Images
We first evaluated whether brain images can be sufficiently
characterized using their low-rank approximation. We selected
a representative 2D axial slice from the T1 MR phantom in
, which has a size of 181×181 with in-plane
resolution of 1 mm (Fig. 3). We then performed singular value
decomposition (SVD) on this image and obtain 174 non-zero
singular values. As shown in Fig. 3, singular values decrease
dramatically, with most values being close to zero. Next, we
remove the small singular values and use the top 30, 60, 90, and
120 singular values to reconstruct the image. Note that the
number of used singular values equals to the rank of the
recovered image, implying that the recovered image is a
low-rank approximation of the original image. The
reconstructed images are shown in Fig. 3 for visual inspection
and the difference maps between original and reconstructed
images are also provided. Signal-to-noise ratio (SNR) in
decibels (dB) is used to evaluate the quality of reconstruction:
  
 
, where is the original HR
image and is the recovered HR image.
The results show that, by using the top 60 singular values, the
reconstructed image has high SNR (34.0db), although small
edge information in the brain boundary is lost. When using the
top 90 or 120 singular values (out of 174), the resulting image
does not show visual differences with respect to the original
image. For the 3D Brainweb image with size 181×217×181, it
has three ranks for its three unfolded matrices that are less than
its longest image size 217. These ranks are relatively low in
comparison to the total number of elements, suggesting brain
images could be represented using their low-rank
approximations with a relatively high SNR.
B. Experimental Settings
We applied our method to a set of down-sampled and blurred
3D brain images and evaluated whether our method can
successfully recover the original high-resolution images. To do
that, we use HR images as ground truth, and simulate LR
images as shown in Fig. 4. Blurring was implemented using a
Gaussian kernel with a standard deviation of 1 voxel. The
blurred image was then down-sampled by averaging every 8
voxels (to simulate the partial volume effect), resulting in half
of the original resolution. The quality of reconstruction of all
methods from the input LR images was evaluated by comparing
with their corresponding original HR images.
A number of comparison methods were also employed,
including nearest-neighbor interpolation (NN), spline
interpolation (Spline), IBP based up-sampling [14], NLM
based up-sampling [15], and TV based up-sampling [22]. The
estimated HR images from all methods were compared with the
original HR image for accuracy of image recovery by using
SNR. Note that, for NLM, we used the implementation made
available by the authors
. TV-based up-sampling is realized by
and in the proposed method and
solving only the subproblem 1. Other methods were
implemented by in-house tools.
Parameters were optimized based on a small dataset,
consisting of 5 adult and 5 pediatric images as described in the
Real Data Evaluation section. In particular,
are weights
to combine unfolded matrices along each spatial dimension in
rank computation. All dimensions are assumed to be equally
important, i.e.,
. The difference between
iterations was measured by
 
, and the
program was stopped when this difference is less than
  . Since TV is a major component of the proposed
method, we first optimized TV by setting
from a group of
candidate values (Fig. 5). We thus chose
. Then, we
optimized the proposed method by employing the same
while setting
from a wide range of candidate values (Fig.
5). We chose the rank regularization as
. As for
Fig. 3. Low-rank approximations of brain images. Top row shows the original
image, singular-value plot, and zoomed singular-value plot of indices from 141
to 181. Bottom row shows the four reconstructed images and their differences
with the original image by using top 30, 60, 90, and 120 singular values,
Fig. 5. Parameter optimization based on ten images. SNR was used to evaluate
the reconstruction performance.
Fig. 4. Simulation of low-resolution image from high-resolution image.
IEEE TMI-2015-0061
the input arguments, the blurring kernel (such as Gaussian
distribution with 1 voxel standard deviation) was used in IBP,
TV, and the proposed method. On the other hand, the default
parameters in NLM implementation were used [15]. Both NN
and Spline do not have any free parameters that require tuning.
We evaluate the performance of the proposed and
comparison methods by comparing the recovered HR images
with the original HR image. Besides SNR, we also employ
another image quality measurement named Structural
Similarity Index (SSIM), which is considered to be correlated
with the quality perception of the human visual system (HVS)
[25]. SSIM has been used in many image SR studies [15,
26-28]. SSIM is defined as:
 
 
 
 
 
 
 ,
are the mean values respectively in the
original HR image and recovered image ,
are the
is the covariance of two images,
 and
, and is the
dynamic range of voxel values [25]. SSIM ranges from 0 to 1,
and 1 means perfect recovery.
C. Phantom Data Evaluation
We employed the T1 MR phantom from Brainweb to
evaluate the recovery performance of the proposed and
comparison methods in cases of no noise and with noise. The
phantom has image size of 181×217×181 and spatial resolution
of 1×1 mm
. We downloaded images both without noise and
with noise at different levels of 1%, 3%, 5%, 7%, and 9% of the
maximum intensity. The noise in the phantom images has
Rayleigh statistics in the background and Rician statistics in the
signal regions. For our experiment, we generated images with
2 mm
resolution using the above-mentioned LR image
simulation pipeline (Fig. 4), and upsample it again to 1 mm
isotropic resolution using the proposed and comparison
methods, respectively.
Fig. 6 demonstrates the results when using noisy data at level
of 3% as input image. A typical slice is shown for each of
coronal, sagittal, and axial views, and the frontal region in
sagittal view is zoomed up for better visual inspection.
Compared to the original HR image, results of NN, Spline, IBP,
and NLM appear blurry. TV provides better image
reconstruction while the proposed method shows more fine
details. Fig. 7 shows the SNR and SSIM measurements for all
methods while changing the noise level from no noise to 9%
noise level. The proposed method outperforms all other
comparison methods in all noise levels for both SNR and SSIM.
Another observation is that, the SNR improvement of the
proposed method over other methods generally reduces when
noise level increases, while the SSIM improvement maintains a
similar level when the noise level increases.
Fig. 6. Illustration of upsampling results for simulated data. LR data with 2 mm
resolution was upsampled to 1 mm
. A typical slice for each of coronal,
sagittal, and axial views is shown, and a zoom-up of frontal region in sagittal view is also provided.
Fig. 7. Results of upsampling on simulate data with different noise levels. (A)
shows the SNR result, and (B) shows the SSIM result.
IEEE TMI-2015-0061
D. Real Data Evaluation
We further evaluated the proposed method on two publicly
available datasets. First, we randomly selected 45 adult subjects
from ADNI
, with 15 from Alzheimer's disease (AD), 15 from
mild cognitive impairment (MCI), and 15 from normal controls
(NC). Their ages were 75±8 years old at MRI scan. T1 MR
images were acquired with 166 sagittal slices at the resolution
of 0.95×0.95×1.2 mm
. Second, we also randomly selected 45
pediatric subjects from NDAR (, with age
of 11±3 years old at MRI scan. T1 MR images were acquired
with 124 sagittal slices at the resolution of 0.90.94×1.3 mm
Fig. 8 shows the representative image SR results of an adult
scan (upper panel) and a pediatric scan (lower panel). From left
to right, the first row of each panel shows the input image,
original HR image, the results of NN, Spline, IBP, NLM, TV,
and the proposed LRTV method. The close-up views of
selected regions are also shown for better visualization. It can
Fig. 8. Results for upsampling an adult image (upper panel) and a pediatric image (lower panel) with different methods. In each panel, the first row shows the input
image and the results by various methods, while the second row shows the close-up views of selected regions in the first row.
Fig. 9. Boxplot of SNR and SSIM results for recovering adult data and pediatric
data using different methods. The proposed LRTV method significantly
outperforms all other comparison methods (p<0.01 using two-sample t-tests).
Fig. 10. Boxplot of SNR and SSIM results for recovering adult data in groups of
AD, MCI, and NC.
IEEE TMI-2015-0061
be observed that the results of NN and Spline interpolation
methods show severe blurring artifacts. The contrast is
enhanced in the results of IBP, NLM, and TV up-sampling
methods, while the proposed LRTV method best preserves
edges and achieves the highest SNR values.
Quantitative results on the images of 45 adults and 45
pediatric subjects are shown in Fig. 9. Note that the original
images were used as ground truth and the input was LR image
simulated following the pipeline of Fig. 4. The proposed
method significantly outperforms all comparison methods
(p<0.01 using two-sample t-tests). Results on adult subjects
demonstrate less variance and higher accuracy than those on
pediatric subjects, which may be because the image quality is
higher in the matured brain and also clearer gyri/sulci patterns
appear in the adult images. No significant difference was found
between the adult subjects of AD, MCI, and NC, as shown in
Fig. 10.
In addition, we applied our method directly to images with
typical imaging resolution. Fig. 11 shows the results of a typical
AD subject for visual inspection, along with the close-up views
of selected representative regions. As it can be observed, the
proposed method recovers fine details and also preserves image
E. Computational Time
All programs were run in Linux environment on a standard
PC using a single thread of an Inter® Xeo CPU (E5630 1.6
GHz). Interpolation methods are computationally efficient as it
takes around 2 seconds for NN and 10 seconds for Spline for
one 3D image. For the up-sampling methods, IBP takes 1 min,
NLM takes 16 mins, TV takes 5 mins, and the proposed method
takes about 30 mins.
We have presented a novel super-resolution method for
recovering high-resolution image from a single low-resolution
image. For the first time, we show that combining low-rank and
total-variation regularizations is a viable solution to the SR
problem. This combination brings together global and local
information for effective recovery of the high-dimensional
image. Experimental results indicate that the proposed method
is able to remedy the partial volume effect and recover the fine
brain structure details from both adult and pediatric images.
Quantitative comparisons show that the proposed method
outperforms other popular methods such as the interpolation
methods based on NN and Spline, and the up-sampling
methods based on IBP, NLM, and TV.
Fig. 11. Upsampling of an image from a typical AD subject. The resolution was thus increased from 0.95×0.91.2 mm
to 0.475×0.475×0.6 mm
. The first row
shows the reconstructed results and other rows show the close-up views of the regions selected from the first row.
IEEE TMI-2015-0061
The proposed method is computationally expensive
compared with other methods. The major reason is that the
program spends most of the time solving TV, which is
implemented using gradient decent [22] in MATLAB. In future
work, we will implement a faster TV solver, such as [29], and
also accelerate the program using C++. On the other hand, we’d
like to clarify that our main purpose is to show that the
TV-based MR image reconstruction can be improved by
combining low-rank regularization. When a better TV solver is
available, the performance of the proposed method could also
be improved.
The proposed method is mainly developed for single-image
SR. In the future, we will extend the proposed method to use
multiple LR images [12, 13], training datasets [30], and also for
more applications such as 4D image recovery in functional
MRI or diffusion MRI. The code for the proposed method will
be released at our website
[1] J. Van Ouwerkerk, "Image super-resolution survey," Image and
Vision Computing, vol. 24, pp. 1039-1052, 2006.
[2] H. Greenspan, "Super-resolution in medical imaging," The
Computer Journal, vol. 52, pp. 43-63, 2009.
[3] S. C. Park, M. K. Park, and M. G. Kang, "Super-resolution image
reconstruction: a technical overview," IEEE Signal Processing
Magazine, vol. 20, pp. 21-36, 2003.
[4] I. Bankman, Handbook of medical image processing and analysis:
academic press, 2008.
[5] A. Buades, B. Coll, and J.-M. Morel, "A review of image denoising
algorithms, with a new one," Multiscale Modeling & Simulation,
vol. 4, pp. 490-530, 2005.
[6] K. Doi, "Computer-aided diagnosis in medical imaging: historical
review, current status and future potential," Computerized medical
imaging and graphics, vol. 31, pp. 198-211, 2007.
[7] B. Zitova and J. Flusser, "Image registration methods: a survey,"
Image and vision computing, vol. 21, pp. 977-1000, 2003.
[8] F. Shi, Y. Fan, S. Tang, J. H. Gilmore, W. Lin, and D. Shen,
"Neonatal brain image segmentation in longitudinal MRI studies,"
Neuroimage, vol. 49, pp. 391-400, Jan 1 2010.
[9] F. Shi, L. Wang, G. Wu, G. Li, J. H. Gilmore, W. Lin, et al.,
"Neonatal atlas construction using sparse representation," Human
brain mapping, 2014.
[10] P.-T. Yap, H. An, Y. Chen, and D. Shen, "Fiber-driven resolution
enhancement of diffusion-weighted images," NeuroImage, vol. 84,
pp. 939-950, 2014.
[11] B. Scherrer, A. Gholipour, and S. K. Warfield, "Super-resolution
reconstruction to increase the spatial resolution of diffusion
weighted images from orthogonal anisotropic acquisitions,"
Medical image analysis, vol. 16, pp. 1465-1476, 2012.
[12] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, "Advances and
challenges in super resolution," International Journal of Imaging
Systems and Technology, vol. 14, pp. 47-57, 2004.
[13] S. Borman and R. L. Stevenson, "Super-resolution from image
sequences-a review," in Circuits and Systems, 1998. Proceedings.
1998 Midwest Symposium on, 1998, pp. 374-378.
[14] M. Irani and S. Peleg, "Improving resolution by image registration,"
CVGIP: Graphical models and image processing, vol. 53, pp.
231-239, 1991.
[15] J. V. Manjón, P. Coupé, A. Buades, V. Fonov, D. Louis Collins, and
M. Robles, "Non-local MRI upsampling," Med Image Anal, vol. 14,
pp. 784-792, 2010.
[16] F. Rousseau, "A non-local approach for image super-resolution
using intermodality priors," Med Image Anal, vol. 14, p. 594, 2010.
[17] J.-F. Cai, E. J. Candès, and Z. Shen, "A singular value thresholding
algorithm for matrix completion," SIAM Journal on Optimization,
vol. 20, pp. 1956-1982, 2010.
[18] E. J. Candès and B. Recht, "Exact matrix completion via convex
optimization," Foundations of Computational mathematics, vol. 9,
pp. 717-772, 2009.
[19] J. Liu, P. Musialski, P. Wonka, and J. Ye, "Tensor Completion for
Estimating Missing Values in Visual Data," IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 35, pp. 208-220,
[20] Netflix, Netflix prize webpage,, 2007.
[21] L. I. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation
based noise removal algorithms," Physica D: Nonlinear
Phenomena, vol. 60, pp. 259-268, 1992.
[22] A. Marquina and S. J. Osher, "Image super-resolution by
TV-regularization and Bregman iteration," Journal of Scientific
Computing, vol. 37, pp. 367-382, 2008.
[23] F. Shi, J. Cheng, L. Wang, P.-T. Yap, and D. Shen, "Low-Rank
Total Variation for Image Super-Resolution," in Medical Image
Computing and Computer-Assisted InterventionMICCAI 2013, ed:
Springer, 2013, pp. 155-162.
[24] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, "Distributed
optimization and statistical learning via the alternating direction
method of multipliers," Foundations and Trends in Machine
Learning, vol. 3, pp. 1-122, 2011.
[25] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image
quality assessment: from error visibility to structural similarity,"
Image Processing, IEEE Transactions on, vol. 13, pp. 600-612,
[26] I. Begin and F. P. Ferrie, "Comparison of super-resolution
algorithms using image quality measures," in Computer and Robot
Vision, 2006. The 3rd Canadian Conference on, 2006, pp. 72-72.
[27] S. Yang, Z. Liu, M. Wang, F. Sun, and L. Jiao, "Multitask dictionary
learning and sparse representation based single-image
super-resolution reconstruction," Neurocomputing, vol. 74, pp.
3193-3203, 2011.
[28] Q. Yuan, L. Zhang, and H. Shen, "Multiframe super-resolution
employing a spatially weighted total variation model," Circuits and
Systems for Video Technology, IEEE Transactions on, vol. 22, pp.
379-392, 2012.
[29] T. Goldstein and S. Osher, "The split Bregman method for
L1-regularized problems," SIAM Journal on Imaging Sciences, vol.
2, pp. 323-343, 2009.
[30] J. Yang, J. Wright, T. S. Huang, and Y. Ma, "Image super-resolution
via sparse representation," Image Processing, IEEE Transactions
on, vol. 19, pp. 2861-2873, 2010.
... This is the same situation as case 1, we n m-dimensional base vectors { } all the base vectors are independent, it can form an m-dimensional space. So, A must be a full rank matrix[17] [18]. That is, rank(A) = m = n, A is invertible. ...
... If the original volume dimensions are (d 1 , d 2 , d 3 ), dimensions of HR image downsampled will be (d 1 /N, d 2 /N, d 3 /N) with N as the selected scaling factor. This way of deriving the LR counterparts is in consonance with previous works in the state of the art (Rueda et al., 2013;Shi et al., 2015;Pham et al., 2017Pham et al., , 2019Zeng et al., 2018). Low resolution downsampled images are then interpolated to HR image size. ...
Full-text available
Pediatric medical imaging represents a real challenge for physicians, as children who are patients often move during the examination, and it causes the appearance of different artifacts in the images. Thus, it is not possible to obtain good quality images for this target population limiting the possibility of evaluation and diagnosis in certain pathological conditions. Specifically, magnetic resonance imaging (MRI) is a technique that requires long acquisition times and, therefore, demands the use of sedation or general anesthesia to avoid the movement of the patient, which is really damaging in this specific population. Because ALARA (as low as reasonably achievable) principles should be considered for all imaging studies, one of the most important reasons for establishing novel MRI imaging protocols is to avoid the harmful effects of anesthesia/sedation. In this context, ground-breaking concepts and novel technologies, such as artificial intelligence, can help to find a solution to these challenges while helping in the search for underlying disease mechanisms. The use of new MRI protocols and new image acquisition and/or pre-processing techniques can aid in the development of neuroimaging studies for children evaluation, and their translation to pediatric populations. In this paper, a novel super-resolution method based on a convolutional neural network (CNN) in two and three dimensions to automatically increase the resolution of pediatric brain MRI acquired in a reduced time scheme is proposed. Low resolution images have been generated from an original high resolution dataset and used as the input of the CNN, while several scaling factors have been assessed separately. Apart from a healthy dataset, we also tested our model with pathological pediatric MRI, and it successfully recovers the original image quality in both visual and quantitative ways, even for available examples of dysplasia lesions. We hope then to establish the basis for developing an innovative free-sedation protocol in pediatric anatomical MRI acquisition.
... With paired input/output samples, MR modality conversion could be implemented as a special case of super-resolution, where one or multiple low-resolution images are combined to generate images with higher spatial resolution. Traditional super-resolution solutions include reconstruction-based methods [25], [27], [30], [35], and example-based methods [3], [10], [14], [20], [24], [36], [39]. Under the deep learning framework, numerous new super-resolution solutions have recently been developed, in both the computer vision [7], [26] and medical image computing [2], [1], [8], [18], [28], [40] communities. ...
Full-text available
In this paper, we explore the capabilities of a number of deep neural network models in generating whole-brain 3T-like MR images from clinical 1.5T MRIs. The models include a fully convolutional network (FCN) method and three state-of-the-art super-resolution solutions, ESPCN [26], SRGAN [17] and PRSR [7]. The FCN solution, U-Convert-Net, carries out mapping of 1.5T-to-3T slices through a U-Net-like architecture, with 3D neighborhood information integrated through a multi-view ensemble. The pros and cons of the models, as well the associated evaluation metrics, are measured with experiments and discussed in depth. To the best of our knowledge, this study is the first work to evaluate multiple deep learning solutions for whole-brain MRI conversion, as well as the first attempt to utilize FCN/U-Net-like structure for this purpose.
... The super-resolution problem is often solved by regularized inversion. Besides Tikhonov regularization [10], total variation [11] or Beltrami energy regularizers [8] have been proposed, as well as constraints on the rank of the matricized image (this method was applied to single-frame super-resolution) [12], and patch-based regularization methods [13]. The last two methods are intended to exploit the low-rank structure of the images. ...
Conference Paper
Full-text available
In this paper, we address the multi-frame super-resolution MRI problem. We formulate the reconstruction problem as a coupled tensor multilinear approximation. We prove that exact recovery of the high-resolution 3D isotropic image is achievable for a variety of multilinear ranks. We propose a simple algorithm based on Tikhonov regularization to perform the reconstruction. Our simulations on real datasets illustrates the good performance of the proposed approach, with a lower computation time than state-of-the-art methods.
... Des méthodes de reconstruction basées sur la régularisation de Tikhonov [21], de Beltrami [9], ou par variation totale [18] ont d'abordété proposées. D'autres approches [2] utilisent une factorisation matricielle de rang faible 1 [16,2]. Le coût de calcul associéà ces méthodes matricielles est souventélevé. ...
Conference Paper
Full-text available
Ce papier résout le problème de super-résolution à partir de plusieurs images IRM. Le problème de reconstruction est formulé comme une approximation multilinéaire tensorielle couplée. Nos résultats théoriques prouvent qu'il est possible de reconstruire exactement l'image à haute résolution. Nous proposons également un algorithme simple basé sur une régularisation de Tikhonov. Nos simulations sur données réelles illustrent les performances de cette approche à un coût de calcul réduit par rapport à l'état de l'art.
Magnetic Resonance (MR) images of the brain play key role in exploiting pathological changes and non-invasive investigation of many neuro-degenerative diseases. Computer Aided Diagnosis (CAD) systems assist radiologists in interpreting MR images and classifying them into “normal” and “abnormal” categories. However, reduced strength of the used magnet in the machine or involuntary motions of the patients may lead to degraded MR images, which can negatively affect the performance of CAD system compromising the classification accuracy. This work aims at modeling these types of situations via out-of-focus blur, motion blur, effect of variation in resolution, and a combination of these on brain MR images for validating the impact of image quality on classification performance. To validate this, this article mathematically models the blurs (both individually and simultaneously) by varying the strength of image quality covariates and afterwards Deep Convolutional Neural Networks (DCNN) are employed to train and classify the distorted brain MR images. Besides, a single DCNN is experimented with a good mix of image quality and characteristics to test the reliability of the model for real-life scenario. The CNN models are validated through comprehensive evaluation on both original and degraded versions of brain MR images from two benchmark datasets DS-75 and DS-160 collected by Harvard Medical School as well as a self-collected dataset NITR-DHH. This study reveals that the models are able to classify distorted MR images and hence can be used for assisting the clinicians.
Background and objective Magnetic resonance imaging can present the precise anatomic structure in clinical applications. Nevertheless, due to the limited scanning equipment cost, scanning time and so on, high-resolution knee MR images are difficult to obtain. So the super-resolution technique is developed to improve the image quality. Unfortunately, conventional CNN-based methods cannot explicitly learn the long-range dependencies in images and simply integrate the auxiliary contrast without considering the characteristics of medical images. To tackle this issue, our approach aims to adaptively capture and fuse the significant auxiliary information of the multi-contrast images to improve the knee magnetic resonance image quality. Methods We propose a multi-scale deformable transformer network (MSDT) for multi-contrast knee magnetic resonance imaging super-resolution. First, we aggregate multi-scale patch embedding from the multi-contrast knee MR images to effectively preserve the local contextual details and global structure information. Then, the deformable transformer architecture is designed to learn the data-dependent sparse attention of the knee MR image, which can adaptively obtain the high-frequency foreground details according to the image content. Results The proposed method is evaluated on the fastMRI dataset under 2× and 4× enlargements. Our MSDT achieves higher PSNR of 31.98 and SSIM of 0.713 at 2× upsampling factor and PSNR of 30.38 and SSIM of 0.615 at 4× upsampling factor. Moreover, our method can generate clear tissue structures and fine details. Conclusions The experimental results show superior performance in comparison to the state-of-the-art super-resolution methods. This indicates that the MSDT can effectively reconstruct the high-quality knee MR images.
Conference Paper
Full-text available
Most natural images can be approximated using their low-rank components. This fact has'been successfully exploited in recent advancements of matrix completion algorithms for image recovery. However, a major limitation of low-rank matrix completion algorithms is that they cannot recover the case where a whole row or column is missing. The missing row or column will be simply filled as an arbitrary combination of other rows or columns with known values. This precludes the application of matrix completion to problems such as super-resolution (SR) where missing values in many rows and columns need to be recovered in the process of up-sampling a low-resolution image. Moreover, low-rank regularization considers information globally from the whole image and does not take proper consideration of local spatial consistency. Accordingly, we propose in this paper a solution to the SR problem via simultaneous (global) low-rank and (local) total variation (TV) regularization. We solve the respective cost function using the alternating direction method of multipliers (ADMM). Experiments on MR images of adults and pediatric subjects demonstrate that the proposed method enhances the details of the recovered high-resolution images, and outperforms the nearest-neighbor interpolation, cubic interpolation, non-local means, and TV-based up-sampling.
Full-text available
Total variation (TV) has been used as a popular and effective image prior model in regularization-based image processing fields, such as denoising, deblurring, super-resolution (SR), and others, because of its ability to preserve edges. However, as the TV model favors a piecewise constant solution, the processing results in the flat regions of the image being poor, and it cannot automatically balance the processing strength between different spatial property regions in the image. In this paper, we propose a spatially weighted TV image SR algorithm, in which the spatial information distributed in different image regions is added to constrain the SR process. A newly proposed and effective spatial information indicator called difference curvature is used to identify the spatial property of each pixel, and a weighted parameter determined by the difference curvature information is added to constrain the regularization strength of the TV regularization at each pixel. Meanwhile, a majorization–minimization algorithm is used to optimize the proposed spatially weighted TV SR model. Finally, a significant amount of simulated and real data experimental results show that the proposed spatially weighted TV SR algorithm not only efficiently reduces the “artifacts” produced with a TV model in fat regions of the image, but also preserves the edge information, and the reconstruction results are less sensitive to the regularization parameters than the TV model, because of the consideration of the spatial information constraint.
Full-text available
In this paper we formulate a new time dependent convolutional model for super-resolution based on a constrained variational model that uses the total variation of the signal as a regularizing functional. We propose an iterative refinement procedure based on Bregman iteration to improve spatial resolution. The model uses a dataset of low resolution images and incorporates a downsampling operator to relate the high resolution scale to the low resolution one. We present an algorithm for the model and we perform a series of numerical experiments to show evidence of the good behavior of the numerical scheme and quality of the results.
Objective methods for assessing perceptual image quality have traditionally attempted to quantify the visibility of errors between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a Structural Similarity Index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MatLab implementation of the proposed algorithm is available online at
"The Handbook of Medical Image Processing and Analysis is a comprehensive compilation of concepts and techniques used for processing and analyzing medical images after they have been generated or digitized. The Handbook is organized into six sections that relate to the main functions: enhancement, segmentation, quantification, registration, visualization, and compression, storage and communication." "The second edition is extensively revised and updated throughout, reflecting new technology and research, and includes new chapters on: higher order statistics for tissue segmentation; tumor growth modeling in oncological image analysis; analysis of cell nuclear features in fluorescence microscopy images; imaging and communication in medical and public health informatics; and dynamic mammogram retrieval from web-based image libraries." "For those looking to explore advanced concepts and access essential information, this second edition of Handbook of Medical Image Processing and Analysis is an invaluable resource. It remains the most complete single volume reference for biomedical engineers, researchers, professionals and those working in medical imaging and medical image processing."--BOOK JACKET.
Atlas construction generally includes first an image registration step to normalize all images into a common space and then an atlas building step to fuse the information from all the aligned images. Although numerous atlas construction studies have been performed to improve the accuracy of the image registration step, unweighted or simply weighted average is often used in the atlas building step. In this article, we propose a novel patch-based sparse representation method for atlas construction after all images have been registered into the common space. By taking advantage of local sparse representation, more anatomical details can be recovered in the built atlas. To make the anatomical structures spatially smooth in the atlas, the anatomical feature constraints on group structure of representations and also the overlapping of neighboring patches are imposed to ensure the anatomical consistency between neighboring patches. The proposed method has been applied to 73 neonatal MR images with poor spatial resolution and low tissue contrast, for constructing a neonatal brain atlas with sharp anatomical details. Experimental results demonstrate that the proposed method can significantly enhance the quality of the constructed atlas by discovering more anatomical details especially in the highly convoluted cortical regions. The resulting atlas demonstrates superior performance of our atlas when applied to spatially normalizing three different neonatal datasets, compared with other start-of-the-art neonatal brain atlases. Hum Brain Mapp, 2014. © 2014 Wiley Periodicals, Inc.
Diffusion-weighted imaging (DWI), while giving rich information about brain circuitry, is often limited by insufficient spatial resolution and low signal-to-noise ratio (SNR). This paper describes an algorithm that will increase the resolution of DW images beyond the scan resolution, allowing for a closer investigation of fiber structures and more accurate assessment of brain connectivity. The algorithm is capable of generating a dense vector-valued field, consisting of diffusion data associated with the full set of diffusion-sensitizing gradients. The fundamental premise is that, to best preserve information, interpolation should always be performed along axonal fibers. To achieve this, at each spatial location, we probe neighboring voxels in various directions to gather diffusion information for data interpolation. Based on the fiber orientation distribution function (ODF), directions that are more likely to be traversed by fibers will be given greater weights during interpolation and vice versa. This ensures that data interpolation is only contributed by diffusion data coming from fibers that are aligned with a specific direction. This approach respects local fiber structures and prevents blurring resulting from averaging of data from significantly misaligned fibers. Evaluations suggest that this algorithm yields results with significantly less blocking artifacts, greater smoothness in anatomical structures, and markedly improved structural visibility.
Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a Structural Similarity Index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000.
Super-resolution reconstruction produces one or a set of high-resolution images from a se-quence of low-resolution frames. This paper reviews a variety of super-resolution methods proposed in the last twenty years, and provides some insight to, and a summary of, our recent contributions to the general super-resolution problem. In the process, a detailed study of several very important aspects of super-resolution, often ignored in the literature, is presented. Specifically, we discuss robustness, treatment of color, and dynamic operation modes. Novel methods for addressing these issues are accompanied by experimental results on simulated and real data. Finally, some future challenges in super-resolution are outlined and discussed.
Diffusion-weighted imaging (DWI) enables non-invasive investigation and characterization of the white matter but suffers from a relatively poor spatial resolution. Increasing the spatial resolution in DWI is challenging with a single-shot EPI acquisition due to the decreased signal-to-noise ratio and T2(∗) relaxation effect amplified with increased echo time. In this work we propose a super-resolution reconstruction (SRR) technique based on the acquisition of multiple anisotropic orthogonal DWI scans. DWI scans acquired in different planes are not typically closely aligned due to the geometric distortion introduced by magnetic susceptibility differences in each phase-encoding direction. We compensate each scan for geometric distortion by acquisition of a dual echo gradient echo field map, providing an estimate of the field inhomogeneity. We address the problem of patient motion by aligning the volumes in both space and q-space. The SRR is formulated as a maximum a posteriori problem. It relies on a volume acquisition model which describes how the acquired scans are observations of an unknown high-resolution image which we aim to recover. Our model enables the introduction of image priors that exploit spatial homogeneity and enables regularized solutions. We detail our SRR optimization procedure and report experiments including numerical simulations, synthetic SRR and real world SRR. In particular, we demonstrate that combining distortion compensation and SRR provides better results than acquisition of a single isotropic scan for the same acquisition duration time. Importantly, SRR enables DWI with resolution beyond the scanner hardware limitations. This work provides the first evidence that SRR, which employs conventional single shot EPI techniques, enables resolution enhancement in DWI, and may dramatically impact the role of DWI in both neuroscience and clinical applications.