Figure 1 - uploaded by Jian Cheng
Content may be subject to copyright.
Source publication
Large neural networks are difficult to deploy on mobile devices because of intensive computation and storage. To alleviate it, we study ternarization, a balance between efficiency and accuracy that quantizes both weights and activations into ternary values. In previous ternarized neural networks, a hard threshold Δ is introduced to determine quanti...
Contexts in source publication
Context 1
... achieve effective optimization of PMD, we adopt a deep adversarial learning network using cross-entropy loss instead of the margin loss of Φ ρ according to . As shown in Fig. 1, it consists of a representation function ψ and two classifiers f and f with coincide to Definition 3.3. Theorem 3.1 implies searching for an empirical proxy distribution with large size of u is crucial. To achieve that, we assume thatˆQthatˆ thatˆQ is a better distribution that can be derived from noisy source ...
Context 2
... achieve effective optimization of PMD, we adopt a deep adversarial learning network using cross-entropy loss instead of the margin loss of Φ ρ according to . As shown in Fig. 1, it consists of a representation function ψ and two classifiers f and f with coincide to Definition 3.3. Theorem 3.1 implies searching for an empirical proxy distribution with large size of u is crucial. To achieve that, we assume thatˆQthatˆ thatˆQ is a better distribution that can be derived from noisy source ...
Similar publications
Edge computing has been getting a momentum with ever-increasing data at the edge of the network. In particular, huge amounts of video data and their real-time processing requirements have been increasingly hindering the traditional cloud computing approach due to high bandwidth consumption and high latency. Edge computing in essence aims to overcom...
Reconfigurable intelligent surfaces (RISs) have shown the potential to improve signal-to-interference-plus-noise ratio (SINR) related coverage, especially at high-frequency communications. However, assessing electromagnetic filed exposure (EMFE) and establishing EMFE regulations in RIS-assisted large-scale networks are still open issues. This paper...
Hyperspectral imaging technology, combining traditional imaging and spectroscopy technologies to simultaneously acquire spatial and spectral information, is deemed to be an intuitive medium for robust face recognition. However, the intrinsic structure of hyperspectral images is more complicated than ordinary gray-scale or RGB images, how to fully e...
A lightweight seedling detection model with improved YOLOv8s is proposed to address the seedling identification problem in the replenishment process In industrial vegetable seedling production, First, the CBS module in the network structure is replaced with depthwise separable convolution (DSC) to reduce the number of parameters and GFLOPS of the m...
Citations
... Various methods have been proposed to compress or accelerate deep networks, such as low-bit quantization (Rastegari et al. 2016;Wang et al. 2020;He et al. 2020;Xu et al. 2020;Chen et al. 2021), knowledge distillation (Gou et al. 2021), efficient convolution (Lavin and Gray 2016;Zhao et al. 2021) and specific hardware design Li et al. 2020Li et al. , 2021. ...
The 2:4 structured sparsity pattern released by NVIDIA Ampere architecture, requiring four consecutive values containing at least two zeros, enables doubling math throughput for matrix multiplications. Recent works mainly focus on inference speedup via 2:4 sparsity while training acceleration has been largely overwhelmed where backpropagation consumes around 70% of the training time. However, unlike inference, training speedup with structured pruning is nontrivial due to the need to maintain the fidelity of gradients and reduce the additional overhead of performing 2:4 sparsity online. For the first time, this article proposes fully sparse training (FST) where `fully' indicates that ALL matrix multiplications in forward/backward propagation are structurally pruned while maintaining accuracy. To this end, we begin with saliency analysis, investigating the sensitivity of different sparse objects to structured pruning. Based on the observation of spatial similarity among activations, we propose pruning activations with fixed 2:4 masks. Moreover, an Information Restoration block is proposed to retrieve the lost information, which can be implemented by efficient gradient-shift operation. Evaluation of accuracy and efficiency shows that we can achieve 2× training acceleration with negligible accuracy degradation on challenging large-scale classification and detection tasks.
Deep neural networks (DNNs) have recently achieved impressive success across a wide range of real-world vision and language processing tasks, spanning from image classification to many other downstream vision tasks, such as object detection, tracking, and segmentation. However, previous well-established DNNs, despite being able to maintain superior accuracy, have also been evolving to be deeper and wider and thus inevitably necessitate prohibitive computational resources for both training and inference. This trend further enlarges the computational gap between computation-intensive DNNs and resource-constrained embedded computing systems, making it challenging to deploy powerful DNNs upon real-world embedded computing systems towards ubiquitous embedded intelligence. To alleviate the above computational gap and enable ubiquitous embedded intelligence, we, in this survey, focus on discussing recent efficient deep learning infrastructures for embedded computing systems, spanning from training to inference , from manual to automated , from convolutional neural networks to transformers , from transformers to vision transformers , from vision models to large language models , from software to hardware , and from algorithms to applications . Specifically, we discuss recent efficient deep learning infrastructures for embedded computing systems from the lens of (1) efficient manual network design for embedded computing systems, (2) efficient automated network design for embedded computing systems, (3) efficient network compression for embedded computing systems, (4) efficient on-device learning for embedded computing systems, (5) efficient large language models for embedded computing systems, (6) efficient deep learning software and hardware for embedded computing systems, and (7) efficient intelligent applications for embedded computing systems. Furthermore, we also envision promising future directions and trends, which have the potential to deliver more ubiquitous embedded intelligence. We believe this survey has its merits and can shed light on future research, which can largely benefit researchers to quickly and smoothly get started in this emerging field.
When encountering the distribution shift between the source (training) and target (test) domains, domain adaptation attempts to adjust the classifiers to be capable of dealing with different domains. Previous domain adaptation research has achieved a lot of success both in theory and practice under the assumption that all the examples in the source domain are well-labeled and of high quality. However, the methods consistently lose robustness in noisy settings where data from the source domain have corrupted labels or features which is common in reality. Therefore, robust domain adaptation has been introduced to deal with such problems. In this paper, we attempt to solve two interrelated problems with robust domain adaptation: distribution shift across domains and sample noises of the source domain. To disentangle these challenges, an optimal transport approach with low-rank constraints is applied to guide the domain adaptation model training process to avoid noisy information influence. For the domain shift problem, the optimal transport mechanism can learn the joint data representations between the source and target domains using a measurement of discrepancy and preserve the discriminative information. The rank constraint on the transport matrix can help recover the corrupted subspace structures and eliminate the noise to some extent when dealing with corrupted source data. The solution to this relaxed and regularized optimal transport framework is a convex optimization problem that can be solved using the Augmented Lagrange Multiplier method, whose convergence can be mathematically proved. The effectiveness of the proposed method is evaluated through extensive experiments on both synthetic and real-world datasets.
Domain adaptation leverages labeled data from a source domain to learn an accurate classifier for an unlabeled target domain. Since the data collected in practical applications usually contain noise, the weakly-supervised domain adaptation algorithm has attracted widespread attention from researchers that tolerates the source domain with label noises or/and features noises. Several weakly-supervised domain adaptation methods have been proposed to mitigate the difficulty of obtaining the high-quality source domains that are highly related to the target domain. However, these methods assume to obtain the accurate noise rate in advance to reduce the negative transfer caused by noises in source domain, which limits the application of these methods in the real world where the noise rate is unknown. Meanwhile, since source data usually comes from multiple domains, the naive application of single-source domain adaptation algorithms may lead to sub-optimal results. We hence propose a universal and scalable weakly-supervised domain adaptation method called PDCAS to ease restraints of such assumptions and make it more general. Specifically, PDCAS includes two stages: progressive distillation and domain alignment. In progressive distillation stage, we iteratively distill out potentially clean samples whose annotated labels are highly consistent with the prediction of model and correct labels for noisy source samples. This process is non-supervision by exploiting intrinsic similarity to measure and extract initial corrected samples. In domain alignment stage, we consider Class-Aligned Sampling which balances the samples for both source and target domains along with the global feature distributions to alleviate the shift of label distributions. Finally, we apply PDCAS in multi-source noisy scenario and propose a novel multi-source weakly-supervised domain adaptation method called MSPDCAS, which shows the scalability of our framework. Extensive experiments on Office-31 and Office-Home datasets demonstrate the effectiveness and robustness of our method compared to state-of-the-art methods.
In recent years, great progress has been made in the field of cross-scene classification. However, most existing cross-scene methods assume that the source domain has massive and carefully annotated data, which is time-consuming and labour-intensive in practice. Datasets in real-life applications usually contain a large number of noisy labels, which will significantly affect cross-scene classification performance. How to perform more discriminative and generalized cross-scene classification in the presence of noisy samples needs to be urgently addressed. Apart from that, existing methods tend to implement global matching between domains, causing problems such as unbalanced adaptation and negative transfer, limiting the cross-scene performance of the model. For more effective and reliable cross-scene classification under noisy environment, Robust Adaptation with Noise (RAN) is proposed in this paper. RAN explores which samples are noiseless and transferable to enable positive and robust cross-scene transfer. The curriculum learning strategy is used to filter out noisy samples for better source supervised learning and cross-domain matching. To further improve the stability and effectiveness of cross-scene adaptation, the class weighting factor and the public weighting factor are introduced to consider the class information of the source and target domain. RAN is an efficient plug-and-play adaptation framework, which is easily implemented and can be embedded in existing methods. Experimental results demonstrate the proposed RAN can achieve remarkable performance on cross-scene classification tasks in noisy environments.
Unsupervised Domain Adaptation (UDA) intends to achieve excellent results by transferring knowledge from labeled source domains to unlabeled target domains in which the data or label distribution changes. Previous UDA methods have acquired great success when labels in source domain are pure. However, even the acquisition of a large scare clean labels in source domain needs plenty of cost as well. In the presence of label noise in source domain, the traditional UDA methods will be seriously degraded as they do not take appropriate measures against the label noise. In this paper, we propose an approach named Robust Self-training with Label Refinement (RSLR) to address the above issue. RSLR adopts the self-training framework by maintaining two Labeling Networks (LNets) on the source domain, which are used to provide confident pseudo-labeled to target samples, and a Target-specific Network (TNet) trained by the pseudo-labeled samples. To combat the effect of label noise, LNets progressively distinguish and refine the mislabeled source samples. RSLR leads to significant improvements over the state-of-the-art methods on extensive benchmark datasets.KeywordsMachine learningUnsupervised domain adaptationNoisy learningSelf-training
Domain adaptation (DA) aims to transfer knowledge from one source domain to another different but related target domain. The mainstream approach embeds adversarial learning into deep neural networks (DNNs) to either learn domain-invariant features to reduce the domain discrepancy or generate data to fill in the domain gap. However, these adversarial DA (ADA) approaches mainly consider the domain-level data distributions, while ignoring the differences among components contained in different domains. Therefore, components that are not related to the target domain are not filtered out. This can cause a negative transfer. In addition, it is difficult to make full use of the relevant components between the source and target domains to enhance DA. To address these limitations, we propose a general two-stage framework, named multicomponent ADA (MCADA). This framework trains the target model by first learning a domain-level model and then fine-tuning that model at the component-level. In particular, MCADA constructs a bipartite graph to find the most relevant component in the source domain for each component in the target domain. Since the nonrelevant components are filtered out for each target component, fine-tuning the domain-level model can enhance positive transfer. Extensive experiments on several real-world datasets demonstrate that MCADA has significant advantages over state-of-the-art methods.
Deep semi-supervised learning (SSL) methods aim to utilize abundant unlabeled data to improve the seen-class classification. However, in the open-world scenario, collected unlabeled data tend to contain unseen-class data, which would degrade the generalization to seen-class classification. Formally, we define the problem as safe deep semi-supervised learning with unseen-class unlabeled data. One intuitive solution is removing these unseen-class instances after detecting them during the SSL process. Nevertheless, the performance of unseen-class identification is limited by the lack of suitable score function, the uncalibrated model, and the small number of labeled data. To this end, we propose a safe SSL method called SAFE
R
-STUDENT from the teacher-student view. Firstly, to enhance the ability of teacher model to identify seen and unseen classes, we propose a general scoring framework called
D
iscrepancy with
R
aw (DR). Secondly, based on unseen-class data mined by teacher model from unlabeled data, we calibrate student model by newly proposed
U
nseen-class
E
nergy-bounded
C
alibration (UEC) loss. Thirdly, based on seen-class data mined by teacher model from unlabeled data, we propose
W
eighted
C
onfirmation
B
ias
E
limination (WCBE) loss to boost seen-class classification of student model. Extensive studies show that SAFE
R
-STUDENT remarkably outperforms the state-of-the-art, verifying the effectiveness of our method in the under-explored problem.
