Figure 2 - uploaded by Binghui Chen
Content may be subject to copyright.
Softmax activation function: 

Softmax activation function: 

Source publication
Conference Paper
Full-text available
Over the past few years, softmax and SGD have become a commonly used component and the default training strategy in CNN frameworks, respectively. However, when optimizing CNNs with SGD, the saturation behavior behind softmax always gives us an illusion of training well and then is omitted. In this paper, we first emphasize that the early saturation...

Context in source publication

Context 1
... simplify our analysis, we consider the problem of bi- nary classification 3 , where y i ∈ [1,2]. Under binary sce- nario, we plot the softmax activation for class 1 in Fig- ure 2. Intuitively, the softmax activation is totally like sig- moid function. ...

Citations

... The network output nodes apply the Softmax function for the number of the unordered classes. A Softmax function is defined in Eq. 5 [33]. ...
Article
Full-text available
Background Whole-body bone scan is the widely used tool for surveying bone metastases caused by various primary solid tumors including lung cancer. Scintigraphic images are characterized by low specificity, bringing a significant challenge to manual analysis of images by nuclear medicine physicians. Convolutional neural network can be used to develop automated classification of images by automatically extracting hierarchal features and classifying high-level features into classes. Results Using convolutional neural network, a multi-class classification model has been developed to detect skeletal metastasis caused by lung cancer using clinical whole-body scintigraphic images. The proposed method consisted of image aggregation, hierarchal feature extraction, and high-level feature classification. Experimental evaluations on a set of clinical scintigraphic images have shown that the proposed multi-class classification network is workable for automated detection of lung cancer-caused metastasis, with achieving average scores of 0.7782, 0.7799, 0.7823, 0.7764, and 0.8364 for accuracy, precision, recall, F-1 score, and AUC value, respectively. Conclusions The proposed multi-class classification model can not only predict whether an image contains lung cancer-caused metastasis, but also differentiate between subclasses of lung cancer (i.e., adenocarcinoma and non-adenocarcinoma). On the context of two-class (i.e., the metastatic and non-metastatic) classification, the proposed model obtained a higher score of 0.8310 for accuracy metric.
... As the dimensionality M increases, H r decreases, which leads to the saturation of softmax [21].ω r tends to only give non-zero firing levels to the rule with maximum H r and thus results in a poor classification performance for highdimensional input data sets. ...
Article
Full-text available
Gait analysis and evaluation are vital for disease diagnosis and rehabilitation. Current gait analysis technologies require wearable devices or high-resolution vision systems within a limited usage space. To facilitate gait analysis and quantitative walking-ability evaluation in daily environments without using wearable devices, a mobile gait analysis and evaluation system is proposed based on a cane robot. Two laser range finders (LRFs) are mounted to obtain the leg motion data. An effective high-dimensional Takagi-Sugeno-Kang (HTSK) fuzzy system, which is suitable for high-dimensional data by solving the saturation problem caused by softmax function in defuzzification, is proposed to recognize the walking states using only the motion data acquired from LRFs. The gait spatial-temporal parameters are then extracted based on the gait cycle segmented by different walking states. Besides, a quantitative walking-ability evaluation index is proposed in terms of the conventional Tinetti scale. The plantar pressure sensing system records the walking states to label training data sets. Experiments were conducted with seven healthy subjects and four patients. Compared with five classical classification algorithms, the proposed method achieves the average accuracy rate of 96.57%, which is improved more than 10%, compared with conventional Takagi-Sugeno-Kang (TSK) fuzzy system. Compared with the gait parameters extracted by the motion capture system OptiTrack, the average errors of step length and gait cycle are only 0.02 m and 1.23 s, respectively. The comparison between the evaluation results of the robot system and the scores given by the physician also validates that the proposed method can effectively evaluate the walking ability.
... D EEP learning has achieved considerable success in computer vision [1], [2], [3], [4], significantly improving the state-of-art of face recognition [5], [6], [7], [8], [9], [10], [11], [12], [13]. This ubiquitous technology is now used to create innovative applications for entertainment and commercial services. ...
... With the help of deep learning technologies, face recognition has developed with unprecedented success [5], [6], [7], [8], [9], [10], [11], [12], [13]. Face recognition models are trained on large-scale training databases [29], [37], [38], and used as feature extractors to test identities that are usually disjoint from the training set [8]. ...
Preprint
While convenient in daily life, face recognition technologies also raise privacy concerns for regular users on the social media since they could be used to analyze face images and videos, efficiently and surreptitiously without any security restrictions. In this paper, we investigate the face privacy protection from a technology standpoint based on a new type of customized cloak, which can be applied to all the images of a regular user, to prevent malicious face recognition systems from uncovering their identity. Specifically, we propose a new method, named one person one mask (OPOM), to generate person-specific (class-wise) universal masks by optimizing each training sample in the direction away from the feature subspace of the source identity. To make full use of the limited training images, we investigate several modeling methods, including affine hulls, class centers, and convex hulls, to obtain a better description of the feature subspace of source identities. The effectiveness of the proposed method is evaluated on both common and celebrity datasets against black-box face recognition models with different loss functions and network architectures. In addition, we discuss the advantages and potential problems of the proposed method. In particular, we conduct an application study on the privacy protection of a video dataset, Sherlock, to demonstrate the potential practical usage of the proposed method. Datasets and code are available at https://github.com/zhongyy/OPOM.
... D EEP learning has achieved considerable success in computer vision [1], [2], [3], [4], significantly improving the state-of-art of face recognition [5], [6], [7], [8], [9], [10], [11], [12], [13]. This ubiquitous technology is now used to create innovative applications for entertainment and commercial services. ...
... With the help of deep learning technologies, face recognition has developed with unprecedented success [5], [6], [7], [8], [9], [10], [11], [12], [13]. Face recognition models are trained on large-scale training databases [29], [37], [38], and used as feature extractors to test identities that are usually disjoint from the training set [8]. ...
Article
Full-text available
While convenient in daily life, face recognition technologies also raise privacy concerns for regular users on the social media since they could be used to analyze face images and videos, efficiently and surreptitiously without any security restrictions. In this paper, we investigate the face privacy protection from a technology standpoint based on a new type of customized cloak, which can be applied to all the images of a regular user, to prevent malicious face recognition systems from uncovering their identity. Specifically, we propose a new method, named one person one mask (OPOM), to generate person-specific (class-wise) universal masks by optimizing each training sample in the direction away from the feature subspace of the source identity. To make full use of the limited training images, we investigate several modeling methods, including affine hulls, class centers and convex hulls, to obtain a better description of the feature subspace of source identities. The effectiveness of the proposed method is evaluated on both common and celebrity datasets against black-box face recognition models with different loss functions and network architectures. In addition, we discuss the advantages and potential problems of the proposed method.
... Also, it is preferred to be a monotonic and differentiable function [97,98]. Whereas, the Sigmoid activation function SðxÞ ¼ 1 1þe Àx was used to estimate the relative probability distribution for continuous output between 0 and 1 [96,99]. The sum of squares error function was used to validate the model, as shown in Table 7. ...
Article
The contribution of this study is twofold. First, it calculates the depth, intensity, and degrees of energy poverty in developing countries using a multidimensional approach. The data analysis of 59 developing countries of Asia and Africa confirmed a widespread ‘severe’ energy poverty across multiple dimensions. The results revealed that Afghanistan, Yemen, Nepal, India, Bangladesh, and the Philippines in Asia and DR Congo, Chad, Madagascar, Niger, Sierre Leone, Tanzania, and Burundi in Africa were the most susceptible countries to extreme multidimensional energy poverty. Second, the study employed supervised machine learning algorithms to identify the most pertinent socioeconomic determinants of extreme multidimensional energy poverty in the developing world. The results of machine learning identified the accumulated wealth of a household, size and ownership status of a house, marital status of the main breadwinner, and place of residence of the main breadwinner to be the five most influential socioeconomic determinants of extreme multidimensional energy poverty. Therefore, the robust findings of an accurate assessment of extreme energy poverty and its socioeconomic determinants have policy significance to eradicate severe energy poverty by announcing additional incentives, allocating resources, and providing special assistance to those who are at the bottom.
... The softmax function can carry out the normalization process by limiting the function output to the interval [0, 1] and making the components add up to 1. A very confident event is denoted by 1, while an impossible event is denoted by 0 [32]. This process allows for the computation of class-membership probabilities for the 16 gestures, and the larger output components correspond to larger confidence [33]. ...
Article
Full-text available
As artificial intelligence and industrial automation are developing, human-robot collaboration (HRC) with advanced interaction capabilities has become an increasingly significant area of research. In this paper, we design and develop a real-time, multi-model HRC system using speech and gestures. A set of sixteen dynamic gestures is designed for communication from a human to an industrial robot. A data set of dynamic gestures is designed and constructed, and it will be shared with the community. A convolutional neural network (CNN) is developed to recognize the dynamic gestures in real time using the Motion History Image (MHI) and deep learning methods. An improved open-source speech recognizer is used for real-time speech recognition of the human worker. An integration strategy is proposed to integrate the gesture and speech recognition results, and a software interface is designed for system visualization. A multi-threading architecture is constructed for simultaneously operating multiple tasks, including gesture and speech data collection and recognition, data integration, robot control, and software interface operation. The various methods and algorithms are integrated to develop the HRC system, with a platform constructed to demonstrate the system performance. The experimental results validate the feasibility and effectiveness of the proposed algorithms and the HRC system.
... For the traditional face recognition systems [7][8][9][10][11], the recognition processes were totally performed in the plaintext state, which provided potential attackers to steal and tamper user's data, and traditional face recognition system cannot preserve user privacy any more. Therefore, the traditional face recognition system is not suitable for the rapid development of cloud technology today. ...
Article
Full-text available
Face recognition is playing an increasingly important role in present society, and suffers from the privacy leakage in plaintext. Therefore, a recognition system based on homomorphic encryption that supports privacy preservation is designed and implemented in this paper. This system uses the CKKS algorithm in the SEAL library, latest homomorphic encryption achievement, to encrypt the normalized face feature vectors, and uses the FaceNet neural network to learn on the image’s ciphertext to achieve face classification. Finally, face recognition in ciphertext is accomplished. After been tested, the whole process of extracting feature vectors and encrypting a face image takes only about 1.712s in the developed system. The average time to compare a group of images in ciphertext is about 2.06s, and a group of images can be effectively recognized within 30 degrees of face bias, with a recognition accuracy of 96.71%. Compared to the face recognition scheme based on the Advanced Encryption Standard encryption algorithm in ciphertext proposed by Wang et al. in 2019, our scheme improves the recognition accuracy by 4.21%. Compared to the image recognition scheme based on the Elliptical encryption algorithm in ciphertext proposed by Kumar S et al. in 2018, the total spent time in our system is decreased by 76.2%. Therefore, our scheme has better operational efficiency and practical value while ensuring the users’ personal privacy. Compared to the face recognition systems in plaintext presented in recent years, our scheme has almost the same level on recognition accuracy and time efficiency.
... As the activation function, we utilise dense layer with Softmax. Softmax function [77] is well-known for its ability to handle multi-class classification problems successfully. The Softmax function is used as an activation function in the output of neural network layers for predicting the multinormal probability. ...
Article
Full-text available
Information extraction from e-commerce platform is a challenging task. Due to significant increase in number of ecommerce marketplaces, it is difficult to gain good accuracy by using existing data mining techniques to systematically extract key information. The first step toward recognizing e-commerce entities is to design an application that detects the entities from unstructured text, known as the Named Entity Recognition (NER) application. The previous NER solutions are specific for recognizing entities such as people, locations, and organizations in raw text, but they are limited in e-commerce domain. We proposed a Bi-directional LSTM with CNN model for detecting e-commerce entities. The proposed model represents rich and complex knowledge about entities and groups of entities about products sold on the dark web. Different experiments were conducted to compare state-of-the-art baselines. Our proposed approach achieves the best performance accuracy on the Dark Web dataset and Conll-2003. Results show good accuracy of 96.20% and 92.90% for the Dark Web dataset and the Conll-2003 dataset, which show good performance compared to other cutting-edge approaches.
... The magnitude spectrum of acceleration on Z-axis [30] function which is mathematically defined in Eq. 5. ...
Article
Full-text available
In smartphone-based crowd/participatory sensing systems, it is necessary to identify the actual sensor data provider. In this context, this paper attempts to recognize the users’ identity based on their gait patterns (i.e. unique walking patterns). More specifically, a deep convolution neural network (CNN) model is proposed for the user identification with accelerometer data generated from users smartphone sensors. The proposed model is evaluated based on the real-world benchmark dataset (accelerometer biometric competition data) having a total of 387 users accelerometer sensor readings (60 million data samples). The performance of the proposed CNN-based approach is also compared with five baseline methods namely Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), Logistic Regression (LR), and K-Nearest Neighbours (KNN). It is observed that the proposed model achieves better results (accuracy = 98.8%, precision = 0.94, recall = 0.97, and F1-score = 0.95) as compared to the baseline methods.
... Research on the training objectives of face recognition (FR) has effectively improved the performance of deeplearning-based face recognition [32,34,39,40]. According to whether a proxy is used to represent a person's identity or a set of training samples, face recognition methods can be divided into proxy-free methods [4, 8, 12, 22-24, 27, 29, 30, 35, 42, 48] and proxy-based methods [3,5,15,17,20,31,33,[36][37][38]47]. The proxy-free methods directly compress the intra-class distance and expand the inter-class distance based on pair-wise learning [4,12,30,35] or triplet learning [8,22,23,27,29,42,48]. ...
Preprint
State-of-the-art face recognition methods typically take the multi-classification pipeline and adopt the softmax-based loss for optimization. Although these methods have achieved great success, the softmax-based loss has its limitation from the perspective of open set classification: the multi-classification objective in the training phase does not strictly match the objective of open set classification testing. In this paper, we derive a new loss named global boundary CosFace (GB-CosFace). Our GB-CosFace introduces an adaptive global boundary to determine whether two face samples belong to the same identity so that the optimization objective is aligned with the testing process from the perspective of open set classification. Meanwhile, since the loss formulation is derived from the softmax-based loss, our GB-CosFace retains the excellent properties of the softmax-based loss, and CosFace is proved to be a special case of the proposed loss. We analyze and explain the proposed GB-CosFace geometrically. Comprehensive experiments on multiple face recognition benchmarks indicate that the proposed GB-CosFace outperforms current state-of-the-art face recognition losses in mainstream face recognition tasks. Compared to CosFace, our GB-CosFace improves 1.58%, 0.57%, and 0.28% at TAR@FAR=1e-6, 1e-5, 1e-4 on IJB-C benchmark.