No file available
To read the file of this research,
you can request a copy directly from the authors.
Differentiable SVD based on Moore-Penrose Pseudoinverse for Inverse Imaging Problems
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
Low-rank regularization-based deep unrolling networks have achieved remarkable success in various inverse imaging problems (IIPs). However, the singular value decomposition (SVD) is non-differentiable when duplicated singular values occur, leading to severe numerical instability during training. In this paper, we propose a differentiable SVD based on the Moore-Penrose pseudoinverse to address this issue. To the best of our knowledge, this is the first work to provide a comprehensive analysis of the differentiability of the trivial SVD. Specifically, we show that the non-differentiability of SVD is essentially due to an underdetermined system of linear equations arising in the derivation process. We utilize the Moore-Penrose pseudoinverse to solve the system, thereby proposing a differentiable SVD. A numerical stability analysis in the context of IIPs is provided. Experimental results in color image compressed sensing and dynamic MRI reconstruction show that our proposed differentiable SVD can effectively address the numerical instability issue while ensuring computational precision. Code is available at https://github.com/yhao-z/SVD-inv.
ResearchGate has not been able to resolve any citations for this publication.
In dynamic magnetic resonance (MR) imaging, low-rank plus sparse (L+S) decomposition, or robust principal component analysis (PCA), has achieved stunning performance. However, the selection of the parameters of L+S is empirical, and the acceleration rate is limited, which are common failings of iterative compressed sensing MR imaging (CS-MRI) reconstruction methods. Many deep learning approaches have been proposed to address these issues, but few of them use a low-rank prior. In this paper, a model-based low-rank plus sparse network, dubbed L+S-Net, is proposed for dynamic MR reconstruction. In particular, we use an alternating linearized minimization method to solve the optimization problem with low-rank and sparse regularization. Learned soft singular value thresholding is introduced to ensure the clear separation of the L component and S component. Then, the iterative steps are unrolled into a network in which the regularization parameters are learnable. We prove that the proposed L+S-Net achieves global convergence under two standard assumptions. Experiments on retrospective and prospective cardiac cine datasets show that the proposed model outperforms state-of-the-art CS and existing deep learning methods and has great potential for extremely high acceleration factors (up to 24x).
Deep learning methods have achieved attractive performance in dynamic MR cine imaging. However, most of these methods are driven only by the sparse prior of MR images, while the important low-rank (LR) prior of dynamic MR cine images is not explored, which may limit further improvements in dynamic MR reconstruction. In this paper, a learned singular value thresholding (Learned-SVT) operator is proposed to explore low-rank priors in dynamic MR imaging to obtain improved reconstruction results. In particular, we put forward a model-based unrolling sparse and low-rank network for dynamic MR imaging, dubbed as SLR-Net. SLR-Net is defined over a deep network flow graph, which is unrolled from the iterative procedures in the iterative shrinkage-thresholding algorithm (ISTA) for optimizing a sparse and LR-based dynamic MRI model. Experimental results on a single-coil scenario show that the proposed SLR-Net can further improve the state-of-the-art compressed sensing (CS) methods and sparsity-driven deep learning-based methods with strong robustness to different undersampling patterns, both qualitatively and quantitatively. Besides, SLR-Net has been extended to a multi-coil scenario, and achieved excellent reconstruction results compared with a sparsity-driven multi-coil deep learning-based method under a high acceleration. Prospective reconstruction results on an open real-time dataset further demonstrate the capability and flexibility of the proposed method on real-time scenarios.
In this paper, we consider the Tensor Robust Principal Component Analysis (TRPCA) problem, which aims to exactly recover the low-rank and sparse components from their sum. Our model is based on the recently proposed tensor-tensor product (or t-product) [13]. Induced by the t-product, we first rigorously deduce the tensor spectral norm, tensor nuclear norm, and tensor average rank, and show that the tensor nuclear norm is the convex envelope of the tensor average rank within the unit ball of the tensor spectral norm. These definitions, their relationships and properties are consistent with matrix cases. Equipped with the new tensor nuclear norm, we then solve the TRPCA problem by solving a convex program and provide the theoretical guarantee for the exact recovery. Our TRPCA model and recovery guarantee include matrix RPCA as a special case. Numerical experiments verify our results, and the applications to image recovery and background modeling problems demonstrate the effectiveness of our method.
Good tracking performance is in general attributed to accurate representation over previously obtained targets and/or reliable discrimination between the target and the surrounding background. In this work, a robust tracker is proposed by integrating the advantages of both approaches. A subspace is constructed to represent the target and the neighboring background, and their class labels are propagated simultaneously via the learned subspace. In addition, a novel criterion is proposed, by taking account of both the reliability of discrimination and the accuracy of representation, to identify the target from numerous target candidates in each frame. Thus, the ambiguity in the class labels of neighboring background samples, which influences the reliability of the discriminative tracking model, is effectively alleviated, while the training set still remains small. Extensive experiments demonstrate that the proposed approach outperforms most state-of-the-art trackers.
Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for ℓ1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.
Spectral computed tomography (CT) is an emerging technology, that generates a multienergy attenuation map for the interior of an object and extends the traditional image volume into a 4-D form. Compared with traditional CT based on energy-integrating detectors, spectral CT can make full use of spectral information, resulting in high resolution and providing accurate material quantification. Numerous model-based iterative reconstruction methods have been proposed for spectral CT reconstruction. However, these methods usually suffer from difficulties such as laborious parameter selection and expensive computational costs. In addition, due to the image similarity of different energy bins, spectral CT usually implies a strong low-rank prior, which has been widely adopted in current iterative reconstruction models. Singular value thresholding (SVT) is an effective algorithm to solve the low-rank constrained model. However, the SVT method requires a manual selection of thresholds, which may lead to suboptimal results. To relieve these problems, in this article, we propose a sparse and low-rank unrolling network (SOUL-Net) for spectral CT image reconstruction, that learns the parameters and thresholds in a data-driven manner. Furthermore, a Taylor expansion-based neural network backpropagation method is introduced to improve the numerical stability. The qualitative and quantitative results demonstrate that the proposed method outperforms several representative state-of-the-art algorithms in terms of detail preservation and artifact reduction.
Quantum state tomography aims to estimate the state of a quantum mechanical system which is described by a trace one, Hermitian positive semidefinite complex matrix, given a set of measurements of the state. Existing works focus on estimating the density matrix that represents the state, using a compressive sensing approach, with fewer measurements than that required for a tomographically complete set, with the assumption that the true state has a low rank. One very popular method to estimate the state is the use of the Singular Value Thresholding (SVT) algorithm. In this work, we present a machine learning approach to estimate the quantum state of n-qubit systems by unrolling the iterations of SVT which we call Learned Quantum State Tomography (LQST). As merely unrolling SVT may not ensure that the output of the network meets the constraints required for a quantum state, we design and train a custom neural network whose architecture is inspired from the iterations of SVT with additional layers to meet the required constraints. We show that our proposed LQST with very few layers reconstructs the density matrix with much better fidelity than the SVT algorithm which takes many hundreds of iterations to converge. We also demonstrate the reconstruction of the quantum Bell state from an informationally incomplete set of noisy measurements.
As an emerging paradigm for signal acquisition and reconstruction, compressive sensing (CS) achieves high-speed sampling and compression jointly and has found its way into many applications. With the fast growth of deep learning in computer vision, various methods of applying neural networks (NNs) in CS imaging tasks have been proposed. One category of them, named the deep unrolling network, is inspired by the physical sampling model and combines the merits of both optimization model- and data-driven methods, becoming the mainstream of this realm. In this review article, we first review the inverse imaging model and optimization algorithms encountered in the CS research and then provide the recent representative developments of CS networks, which are grouped into deep physics-free and physics-inspired approaches with respect to the utilization of sampling matrix and measurement information. Following this, we analyze the conceptual connections and relationships among various existing methods and present our perspectives on recent advances and trends for future research.
Clouds, together with their shadows, usually occlude ground-cover features in optical remote sensing images. This hinders the utilization of these images for a range of applications such as earth observation, land-cover classification and urban planning. In this work, we propose a deep unfolded and prior-aided robust principal component analysis (DUPA-RPCA) network for removing clouds and recovering ground-cover information in multi-temporal satellite images. We model these cloud-contaminated images as a sum of low rank and sparse elements and then unfold an iterative RPCA algorithm that has been designed for reweighted
minimization. As a result, the activation function in DUPA-RPCA adapts for every input at each layer of the network. Our experimental results on both Landsat and Sentinel images indicate that our method gives better accuracy and efficiency when compared with existing state of the art methods.
The major challenge in high dynamic range (HDR) imaging for dynamic scenes is suppressing ghosting artifacts caused by large object motions or poor exposures. Whereas recent deep learning-based approaches have shown significant synthesis performance, interpretation and analysis of their behaviors are difficult and their performance is affected by the diversity of training data. In contrast, traditional model-based approaches yield inferior synthesis performance to learning-based algorithms despite their theoretical thoroughness. In this paper, we propose an algorithm unrolling approach to ghost-free HDR image synthesis algorithm that unrolls an iterative low-rank tensor completion algorithm into deep neural networks to take advantage of the merits of both learning- and model-based approaches while overcoming their weaknesses. First, we formulate ghost-free HDR image synthesis as a low-rank tensor completion problem by assuming the low-rank structure of the tensor constructed from low dynamic range (LDR) images and linear dependency among LDR images. We also define two regularization functions to compensate for modeling inaccuracy by extracting hidden model information. Then, we solve the problem efficiently using an iterative optimization algorithm by reformulating it into a series of subproblems. Finally, we unroll the iterative algorithm into a series of blocks corresponding to each iteration, in which the optimization variables are updated by rigorous closed-form solutions and the regularizers are updated by learned deep neural networks. Experimental results on different datasets show that the proposed algorithm provides better HDR image synthesis performance with superior robustness compared with state-of-the-art algorithms, while using significantly fewer training samples.
Synthetic aperture radar (SAR) imaging with sub-Nyquist sampled echo is a challenging task. Compressed sensing (CS) has been widely applied in this case to reconstruct the unambiguous image. The CS-based methods need to set the iterative parameters manually, but the appropriate parameters are usually difficult to obtain. Besides, such methods require a large number of iterations to obtain satisfactory results, which seriously restricts their practical applications. Moreover, the observation scene of SAR is not sparse in some cases. In this article, we aim at proposing an efficient and effective imaging method for nonsparse observation scenes with reduced data. First, considering the characteristics of nonsparse observation scenes in SAR imaging, we model the SAR imaging problem as a joint low-rank and sparse matrices recovery problem. After that, the iterative alternating direction method of multipliers (ADMMs) to solve the above problem is unrolled into a layer-fixed deep neural network with trainable parameters, in which the learnable parameters are layer-varied. The threshold parameters, as well as the weight parameter between the sparse part and low-rank part of each layer, are learned adaptively instead of manually tuned. Experiments prove that the proposed low-rank and sparse recovery LRSR-ADMM-Net is capable of reconstructing the nonsparse observed scene with high efficiency and precision. Particularly, the proposed LRSR-ADMM-Net yields better reconstruction performance while maintaining high computational efficiency compared with the state-of-the-art iterative recovery methods and the trainable sparse-based network methods.
The low-rank matrix completion has gained rapidly increasing attention from researchers in recent years for its efficient recovery of the matrix in various fields. Numerous studies have exploited the popular neural networks to yield low-rank outputs under the framework of low-rank matrix factorization. However, due to the discontinuity and nonconvexity of rank function, it is difficult to directly optimize the rank function via back propagation technique. Although a large number of studies have attempted to find relaxations of the rank function, e.g., the extensively applied Schatten-p norm, they still face the following issues when updating parameters via back propagation: (1) These methods or surrogate functions are still non-differentiable, bringing obstacles to deriving the gradients of trainable variables. (2) Most of these surrogate functions perform singular value decomposition upon the original matrix at each iteration, which is time-consuming and blocks the propagation of gradients. To address these problems, in this paper, we develop an efficient block-wise model dubbed differentiable low-rank learning (DLRL) framework that adopts back propagation technique to optimize the Multi-Schatten-
p
norm Surrogate (MSS) function. Different from the original optimization of this surrogate function, the proposed framework avoids singular value decomposition to admit the gradient propagation and builds a block-wise learning schema to minimize values of Schatten-p norms. Accordingly, it speeds up the computations and makes all parameters in the proposed framework learnable according to a predefined loss function. Finally, we conduct substantial experiments in terms of image recovery and collaborative filtering. The experimental results verify the superiority of both runtimes and learning performances of the proposed framework compared with other state-of-the-art low-rank optimization methods.
Eigendecomposition of symmetric matrices is at the heart of many computer vision algorithms. However, the derivatives of the eigenvectors tend to be numerically unstable, whether using the SVD to compute them analytically or using the Power Iteration (PI) method to approximate them. This instability arises in the presence of eigenvalues that are close to each other. This makes integrating eigendecomposition into deep networks difficult and often results in poor convergence, particularly when dealing with large matrices. While this can be mitigated by partitioning the data into small arbitrary groups, doing so has no theoretical basis and makes it impossible to exploit the full power of eigendecomposition. In previous work, we mitigated this using SVD during the forward pass and PI to compute the gradients during the backward pass. However, the iterative deflation procedure required to compute multiple eigenvectors using PI tends to accumulate errors and yield inaccurate gradients. Here, we show that the Taylor expansion of the SVD gradient is theoretically equivalent to the gradient obtained using PI without relying in practice on an iterative process and thus yields more accurate gradients. We demonstrate the benefits of this increased accuracy for image classification and style transfer.
Deep neural networks provide unprecedented performance gains in many real-world problems in signal and image processing. Despite these gains, the future development and practical deployment of deep networks are hindered by their black-box nature, i.e., a lack of interpretability and the need for very large training sets. An emerging technique called algorithm unrolling, or unfolding, offers promise in eliminating these issues by providing a concrete and systematic connection between iterative algorithms that are widely used in signal processing and deep neural networks. Unrolling methods were first proposed to develop fast neural network approximations for sparse coding. More recently, this direction has attracted enormous attention, and it is rapidly growing in both theoretic investigations and practical applications. The increasing popularity of unrolled deep networks is due, in part, to their potential in developing efficient, high-performance (yet interpretable) network architectures from reasonably sized training sets.
Low Rank Regularization (LRR), in essence, involves introducing a low rank or approximately low rank assumption to target we aim to learn, which has achieved great success in many data analysis tasks. Over the last decade, much progress has been made in theories and applications. Nevertheless, the intersection between these two lines is rare. In order to construct a bridge between practical applications and theoretical studies, in this paper we provide a comprehensive survey for LRR. Specifically, we first review the recent advances in two issues that all LRR models are faced with: (1) rank-norm relaxation, which seeks to find a relaxation to replace the rank minimization problem; (2) model optimization, which seeks to use an efficient optimization algorithm to solve the relaxed LRR models. For the first issue, we provide a detailed summarization for various relaxation functions and conclude that the non-convex relaxations can alleviate the punishment bias problem compared with the convex relaxations. For the second issue, we summarize the representative optimization algorithms used in previous studies, and analyze their advantages and disadvantages. As the main goal of this paper is to promote the application of non-convex relaxations, we conduct extensive experiments to compare different relaxation functions. The experimental results demonstrate that the non-convex relaxations generally provide a large advantage over the convex relaxations. Such a result is inspiring for further improving the performance of existing LRR models.
Hyperspectral images (HSIs) are usually corrupted by various noises, e.g., Gaussian noise, impulse noise, stripes, dead lines, and many others. In this article, motivated by the good performance of the
nonconvex metric in image sparse structure exploitation, we first develop a 3-D
spatial–spectral total variation (
SSTV) regularization to globally represent the sparse prior in the gradient domain of HSIs. Then, we divide HSIs into local overlapping 3-D patches, and low-rank tensor recovery (LTR) is locally used to effectively separate the low-rank clean HSI patches from complex noise. The patchwise LTR can not only adapt to the local low-rank property of HSIs well but also significantly reduce the information loss caused by the global LTR. Finally, integrating the advantages of both the global
SSTV regularization and local LTR model, we propose a
SSTV regularized local LTR model for hyperspectral restoration. In the framework of the alternating direction method of multipliers, the difference of convex algorithm, the split Bregman iteration method, and tensor singular value decomposition method are adopted to solve the proposed model efficiently. Simulated and real HSI experiments show that the proposed model can reduce the dependence on noise independent and identical distribution hypotheses, and simultaneously remove various types of noise, even structure-related noise.
Contrast enhanced ultrasound is a radiation-free imaging modality which uses encapsulated gas microbubbles for improved visualization of the vascular bed deep within the tissue. It has recently been used to enable imaging with unprecedented subwavelength spatial resolution by relying on super-resolution techniques. A typical preprocessing step in super-resolution ultrasound is to separate the microbubble signal from the cluttering tissue signal. This step has a crucial impact on the final image quality. Here, we propose a new approach to clutter removal based on robust principle component analysis (PCA) and deep learning. We begin by modeling the acquired contrast enhanced ultrasound signal as a combination of low rank and sparse components. This model is used in robust PCA and was previously suggested in the context of ultrasound Doppler processing and dynamic magnetic resonance imaging. We then illustrate that an iterative algorithm based on this model exhibits improved separation of microbubble signal from the tissue signal over commonly practiced methods. Next, we apply the concept of deep unfolding to suggest a deep network architecture tailored to our clutter filtering problem which exhibits improved convergence speed and accuracy with respect to its iterative counterpart. We compare the performance of the suggested deep network on both simulations and in-vivo rat brain scans, with a commonly practiced deep-network architecture and with the fast iterative shrinkage algorithm. We show that our architecture exhibits better image quality and contrast.
Deep learning models have gained great success in many real-world applications. However, most existing networks are typically designed in heuristic manners, thus these approaches lack of rigorous mathematical derivations and clear interpretations. Several recent studies try to build deep models by unrolling a particular optimization model that involves task information. Unfortunately, due to the dynamic nature of network parameters, their resultant deep propagations do not possess the nice convergence property as the original optimization scheme does. In this work, we develop a generic paradigm to unroll nonconvex optimization for deep model design. Different from most existing frameworks, which just replace the iterations by network architectures, we prove in theory that the propagation generated by our proximally unrolled deep model can globally converge to the critical-point of the original optimization model. Moreover, even if the task information is only partially available (e.g., no prior regularization), we can still train a convergent deep propagations. We also extend these theoretical investigations on the more general multi-block models and thus a lot of real-world applications can be successfully handled by the proposed framework. Finally, we conduct experiments on various low-level vision tasks (i.e., non-blind deconvolution, dehazing, and low-light image enhancement) and demonstrate the superiority of our proposed framework, compared with existing state-of-the-art approaches.
This paper introduces an approach to simultaneously
estimate 3D shape, camera pose, and object and type of
deformation clustering, from partial 2D annotations in a
multi-instance collection of images. Furthermore, we can
indistinctly process rigid and non-rigid categories. This advances
existing work, which only addresses the problem for
one single object or, if multiple objects are considered, they
are assumed to be clustered a priori. To handle this broader
version of the problem, we model object deformation using
a formulation based on multiple unions of subspaces,
able to span from small rigid motion to complex deformations.
The parameters of this model are learned via Augmented
Lagrange Multipliers, in a completely unsupervised
manner that does not require any training data at all. Extensive
validation is provided in a wide variety of synthetic
and real scenarios, including rigid and non-rigid categories
with small and large deformations. In all cases our approach
outperforms state-of-the-art in terms of 3D reconstruction
accuracy, while also providing clustering results
that allow segmenting the images into object instances and
their associated type of deformation (or action the object is performing).
In this paper, we propose an efficient algorithm for dynamic magnetic resonance (MR) image reconstruction. With the total variation (TV) and the nuclear norm (NN) regularization, the TVNNR model can utilize both spatial and temporal redundancy in dynamic MR images. Such prior knowledge can help model dynamic MRI data significantly better than a low-rank or a sparse model alone. However, it is very challenging to efficiently minimize the energy function due to the non-smoothness and non-separability of both TV and NN terms. To address this issue, we propose an efficient algorithm by solving a primal-dual form of the original problem. We theoretically prove that the proposed algorithm achieves a convergence rate of O(1/N) for N iterations. In comparison with state-of-the-art methods, extensive experiments on single-coil and multi-coil dynamic MR data demonstrate the superior performance of the proposed method in terms of both reconstruction accuracy and time complexity.
Low-rank recovery models have shown potential for salient object detection, where a matrix is decomposed into a low-rank matrix representing image background and a sparse matrix identifying salient objects. Two deficiencies, however, still exist. First, previous work typically assumes the elements in the sparse matrix are mutually independent, ignoring the spatial and pattern relations of image regions. Second, when the low-rank and sparse matrices are relatively coherent, e.g., when there are similarities between the salient objects and background or when the background is complicated, it is difficult for previous models to disentangle them. To address these problems, we propose a novel structured matrix decomposition model with two structural regularizations: (1) a tree-structured sparsity-inducing regularization that captures the image structure and enforces patches from the same object to have similar saliency values, and (2) a Laplacian regularization that enlarges the gaps between salient objects and the background in feature space. Furthermore, high-level priors are integrated to guide the matrix decomposition and boost the detection. We evaluate our model for salient object detection on five challenging datasets including single object, multiple objects and complex scene images, and show competitive results as compared with 24 state-of-the-art methods in terms of seven performance metrics.
Whitening, or sphering, is a common preprocessing step in statistical
analysis to transform random variables to orthogonality. However, due to
rotational freedom there are infinitely many possible whitening procedures.
Consequently, there is a diverse range of sphering methods in use, for example
based on principal component analysis, Cholesky matrix decomposition and
Mahalanobis transformation, among others.
Here we provide an overview of the underlying theory and discuss five natural
whitening procedures. Subsequently, we demonstrate that investigating the
cross-covariance and the cross-correlation matrix between sphered and original
variables allows to break the rotational invariance of whitening and to
identify optimal transformations. As a result we recommended two particular
whitening approaches: CAT-CAR whitening to produce sphered variables that are
maximally similar to the original variables, and PCA-whitening based on the
correlation matrix to obtain maximally compressed whitened variables.
Spectral divide and conquer algorithms solve the eigenvalue problem for all the eigenvalues and eigenvectors by recursively computing an invariant subspace for a subset of the spectrum and using it to decouple the problem into two smaller subproblems. A number of such algorithms have been developed over the last 40 years, often motivated by parallel computing and, most recently, with the aim of achieving minimal communication costs. However, none of the existing algorithms has been proved to be backward stable, and they all have a significantly higher arithmetic cost than the standard algorithms currently used. We present new spectral divide and conquer algorithms for the symmetric eigenvalue problem and the singular value decomposition that are backward stable, achieve lower bounds on communication costs recently derived by Ballard, Demmel, Holtz, and Schwartz, and have operation counts within a small constant factor of those for the standard algorithms. The new algorithms are built on the polar decomposition and exploit the recently developed QR-based dynamically weighted Halley algorithm of Nakatsukasa, Bai, and Gygi, which computes the polar decomposition using a cubically convergent iteration based on the building blocks of QR factorization and matrix multiplication. The algorithms have great potential for efficient, numerically stable computations in situations where the cost of communication dominates the cost of arithmetic.
This paper is concerned with the problem of recovering an unknown matrix from a small fraction of its entries. This is known as the matrix completion problem, and comes up in a great number of applications, including the famous Netflix Prize and other similar questions in collaborative filtering. In general, accurate recovery of a matrix from a small number of entries is impossible, but the knowledge that the unknown matrix has low rank radically changes this premise, making the search for solutions meaningful. This paper presents optimality results quantifying the minimum number of entries needed to recover a matrix of rank r exactly by any method whatsoever (the information theoretic limit). More importantly, the paper shows that, under certain incoherence assumptions on the singular vectors of the matrix, recovery is possible by solving a convenient convex program as soon as the number of entries is on the order of the information theoretic limit (up to logarithmic factors). This convex program simply finds, among all matrices consistent with the observed entries, that with minimum nuclear norm. As an example, we show that on the order of nr log( n ) samples are needed to recover a random n x n matrix of rank r by any method, and to be sure, nuclear norm minimization succeeds as soon as the number of entries is of the form nr polylog( n ).
Spatiotemporal imaging, including both dynamic imaging and spectroscopic imaging, has a wide range of applications from functional neuroimaging, cardiac imaging to metabolic cancer imaging. A practical challenge lies in obtaining high spatiotemporal resolution because the amount of data required increases exponentially as the physical dimension increases (curse of dimensionality). This paper describes a new way for Spatiotemporal imaging using partially separable functions. This model admits highly sparse sampling of the data space, providing an effective way to achieve high Spatiotemporal resolution. Practical imaging data will also be presented to demonstrate the performance of the new method
Abstract This paper introduces a novel algorithm to approximate the matrix with minimum,nuclear norm among all matrices obeying a set of convex constraints. This problem may be understood as the convex relaxation of a rank minimization problem, and arises in many important applications as in the task of recovering a large matrix from a small subset of its entries (the famous Netix problem). O-the-shelf,algorithms such as interior point methods are not directly amenable to large problems of this kind with over a million unknown,entries. This paper develops a simple,rst-order and easy-to-implement algorithm that is extremely ecient,at addressing problems in which the optimal solution has low rank. The algorithm is iterative and produces a sequence of matricesfX,g is empirically nondecreasing. Both these facts allow the algorithm to make use of very minimal storage space and keep the computational cost of each iteration low. On the theoretical side, we provide a convergence analysis showing that the sequence of iterates converges. On the practical side, we provide numerical examples in which 1; 000 1; 000 matrices are recovered in less than a minute on a modest desktop computer. We also demonstrate that our approach is amenable to very large scale problems by recovering matrices of rank about 10 with nearly a billion unknowns from just about 0.4% of their sampled entries. Our methods are connected with the recent literature on linearized Bregman iterations for ‘1 minimization, and we develop a framework in which one can understand these algorithms in terms of well-known Lagrange multiplier algorithms. Keywords. Nuclear norm minimization, matrix completion, singular value thresholding, La-
We consider the class of iterative shrinkage-thresholding algorithms (ISTA) for solving linear inverse problems arising in signal/image processing. This class of methods, which can be viewed as an ex- tension of the classical gradient algorithm, is attractive due to its simplicity and thus is adequate for solving large-scale problems even with dense matrix data. However, such methods are also known to converge quite slowly. In this paper we present a new fast iterative shrinkage-thresholding algorithm (FISTA) which preserves the computational simplicity of ISTA but with a global rate of convergence which is proven to be significantly better, both theoretically and practically. Initial promising nu- merical results for wavelet-based image deblurring demonstrate the capabilities of FISTA which is shown to be faster than ISTA by several orders of magnitude.
This paper is about a curious phenomenon. Suppose we have a data matrix, which is the superposition of a low-rank component and a sparse component. Can we recover each component individually? We prove that under some suitable assumptions, it is possible to recover both the low-rank and the sparse components exactly by solving a very convenient convex program called Principal Component Pursuit; among all feasible decompositions, simply minimize a weighted combination of the nuclear norm and of the L1 norm. This suggests the possibility of a principled approach to robust principal component analysis since our methodology and results assert that one can recover the principal components of a data matrix even though a positive fraction of its entries are arbitrarily corrupted. This extends to the situation where a fraction of the entries are missing as well. We discuss an algorithm for solving this optimization problem, and present applications in the area of video surveillance, where our methodology allows for the detection of objects in a cluttered background, and in the area of face recognition, where it offers a principled way of removing shadows and specularities in images of faces.
TensorFlow: a system for Large-Scale machine learning
- M Abadi
- P Barham
- J Chen
- Z Chen
- A Davis
- J Dean
- M Devin
- S Ghemawat
- G Irving
- M Isard
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin,
M., Ghemawat, S., Irving, G., Isard, M., et al., 2016. TensorFlow: a
system for Large-Scale machine learning, in: 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp.
265-283.
OCMR (v1.0)-open-access multi-coil k-space dataset for cardiovascular magnetic resonance imaging
- C Chen
- Y Liu
- P Schniter
- M Tong
- K Zareba
- O Simonetti
- L Potter
- R Ahmad
Chen, C., Liu, Y., Schniter, P., Tong, M., Zareba, K., Simonetti, O.,
Potter, L., Ahmad, R., 2020. OCMR (v1.0)-open-access multi-coil
k-space dataset for cardiovascular magnetic resonance imaging. arXiv
preprint arXiv:2008.03410.
- D Liang
- J Cheng
- Z Ke
- L Ying
Liang, D., Cheng, J., Ke, Z., Ying, L., 2019. Deep MRI reconstruction:
unrolled optimization algorithms meet neural networks. arXiv preprint
arXiv:1907.11711.
Torch. manual_seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision
- D Picard
Picard, D., 2021. Torch. manual_seed (3407) is all you need: On the
influence of random seeds in deep learning architectures for computer
vision. arXiv preprint arXiv:2109.08203.
Differentiating the singular value decomposition
- J Townsend
Townsend, J., 2016. Differentiating the singular value decomposition.
Technical Report. Technical Report 2016, https://jtowns.github.io/papers/svd-derivative.pdf.
Backpropagation-friendly eigendecomposition
- W Wang
- Z Dang
- Y Hu
- P Fua
- M Salzmann
Wang, W., Dang, Z., Hu, Y., Fua, P., Salzmann, M., 2019.
Backpropagation-friendly eigendecomposition. Advances in Neural
Information Processing Systems 32.
Multi-feature discrete collaborative filtering for fast cold-start recommendation
- Y Xu
- L Zhu
- Z Cheng
- J Li
- J Sun
Xu, Y., Zhu, L., Cheng, Z., Li, J., Sun, J., 2020. Multi-feature
discrete collaborative filtering for fast cold-start recommendation, in:
Proceedings of the AAAI conference on artificial intelligence, pp.
270-278.