Fig 6 - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
Source publication
Natural language processing (NLP) may face the inexplicable “black-box” problem of parameters and unreasonable modeling for lack of embedding of some characteristics of natural language, while the quantum-inspired models based on quantum theory may provide a potential solution. However, the essential prior knowledge and pretrained text features are...
Contexts in source publication
Context 1
... ERNIE is used as an independent module to train the high-level features of sentences. Meanwhile, the quantum-inspired model is also trained to obtain the measurement probability features. The two parts of features are concatenated according to the last dimension and the output result is obtained by using the linear layer (corresponding to Fig. ...
Context 2
... letters, splitting words, removing stop words, and encoding tokens into vocabulary indexes. The training labels are converted to "float16" type of paddle tensor. We embed sentiment information into phase embedding layer, where we obtain feature parameters from a Vanilla quantuminspired model training on sentiment classification datasets. Fig. 6 shows the detail of QPFE-ERNIE for sentiment classification. The upper and lower parts of the graph are QPFE and ERNIE, respectively, and the features they learn are finally merged together through tensor concatenate operation. The upper part of the figure contains QPFE. First, words representing quantum particle pass through a complex ...
Similar publications
The increase in the popularity of code mixed languages has resulted in the need to engineer language models for the same. Unlike pure languages , code-mixed languages lack clear grammatical structures, leading to ambiguous sentence constructions. This ambiguity presents significant challenges for natural language processing tasks, including syntact...
Knowledge graphs show excellent application potential in natural language processing, but extracting overlapping relations of entities in their construction poses a significant challenge. This paper proposes an entity overlapping relation extraction algorithm based on a one-dimensional convolutional neural network, which combines the local features...
Citations
... The ability of deep learning models to extract and interpret these sophisticated patterns has led to groundbreaking advancements across various domains [21], [22], [23], [24], including NLP, computer vision, and robotics, among others [25], [26]. These achievements underscore the transformative impact of deep learning, solidifying its position as a key driver of innovation in AI and beyond [27], [28], [29]. ...
The acceptance of academic papers involves a complex peer-review process that requires substantial human and material resources and is susceptible to biases. With advancements in deep learning technologies, researchers have explored automated approaches for assessing paper acceptance. Existing automated academic paper rating methods primarily rely on the full content of papers to estimate acceptance probabilities. However, these methods are often inefficient and introduce redundant or irrelevant information. Additionally, while Bert can capture general semantic representations through pretraining on large-scale corpora, its performance on the automatic academic paper rating (AAPR) task remains suboptimal due to discrepancies between its pretraining corpus and academic texts. To address these issues, this study proposes LMCBert, a model that integrates large language models (LLMs) with momentum contrastive learning (MoCo). LMCBert utilizes LLMs to extract the core semantic content of papers, reducing redundancy and improving the understanding of academic texts. Furthermore, it incorporates MoCo to optimize Bert training, enhancing the differentiation of semantic representations and improving the accuracy of paper acceptance predictions. Empirical evaluations demonstrate that LMCBert achieves effective performance on the evaluation dataset, supporting the validity of the proposed approach. The code and data used in this article are publicly available at https://github.com/iioSnail/LMCBert.
... With the development and invention of NISQ devices to 1121 qubits, the performance and computational efficiency of QML models have been greatly enhanced. For example, Shi NLP disciplines [4]. Yu et al. introduced a simple and general reward design method based on quantum machine learning to effectively simulate quantum systems of different dimensions [5]. ...
Representing and learning from graphs is essential for developing effective machine learning models tailored to non-Euclidean data. While Graph Neural Networks (GNNs) strive to address the challenges posed by complex, high-dimensional graph data, Quantum Neural Networks (QNNs) present a compelling alternative due to their potential for quantum parallelism. However, much of the current QNN research tends to overlook the vital connection between quantum state encoding and graph structures, which limits the full exploitation of quantum computational advantages. To address these challenges, this paper introduces a quantum graph Hamiltonian neural network (QGHNN) to enhance graph representation and learning on noisy intermediate-scale quantum computers. Concretely, a quantum graph Hamiltonian learning method (QGHL) is first created by mapping graphs to the Hamiltonian of the topological quantum system. Then, QGHNN based on QGHL is presented, which trains parameters by minimizing the loss function and uses the gradient descent method to learn the graph. Experiments on the PennyLane quantum platform reveal that QGHNN outperforms all assessment metrics, achieving the lowest mean squared error of \textbf{0.004} and the maximum cosine similarity of \textbf{}, which shows that QGHNN not only excels in representing and learning graph information, but it also has high robustness ability. QGHNN can reduce the impact of quantum noise and has significant potential application in future research of quantum knowledge graphs and recommendation systems.
... Quantum theory not only describes the non-classical behaviors of microscopic particles in physics but also can be in principle applied to macroscopic world problems that require formalizing uncertainty. In recent years, researchers have been exploring the application of quantum theory in the field of natural language processing in various tasks such as information retrieval [20], question answering [21,22] and classification [10,23,24]. ...
Deep neural networks have a recognized susceptibility to diverse forms of adversarial attacks in the field of natural language processing and such a security issue poses substantial security risks and erodes trust in artificial intelligence applications among people who use them. Meanwhile, quantum theory-inspired models that represent word composition as a quantum mixture of words have modeled the non-linear semantic interaction. However, modeling without considering the non-linear semantic interaction between sentences in the current literature does not exploit the potential of the quantum probabilistic description for improving the robustness in adversarial settings. In the present study, a novel quantum theory-inspired inter-sentence semantic interaction model is proposed for enhancing adversarial robustness via fusing contextual semantics. More specifically, it is analyzed why humans are able to understand textual adversarial examples, and a crucial point is observed that humans are adept at associating information from the context to comprehend a paragraph. Guided by this insight, the input text is segmented into subsentences, with the model simulating contextual comprehension by representing each subsentence as a particle within a mixture system, utilizing a density matrix to model inter-sentence interactions. A loss function integrating cross-entropy and orthogonality losses is employed to encourage the orthogonality of measurement states. Comprehensive experiments are conducted to validate the efficacy of proposed methodology, and the results underscore its superiority over baseline models even commercial applications based on large language models in terms of accuracy across diverse adversarial attack scenarios, showing the potential of proposed approach in enhancing the robustness of neural networks under adversarial attacks.
Neural network language models (LMs) are confronted with significant challenges in generalization and robustness. Currently, many studies focus on improving either generalization or robustness in isolation, without methods addressing both aspects simultaneously, which presents a significant challenge in developing LMs that are both robust and generalized. In this paper, we propose a bi-stage optimization framework to uniformly enhance both the generalization and robustness of LMs, termed UEGR. Specifically, during the forward propagation stage, we enrich the output probability distributions of adversarial samples by adaptive dropout to generate diverse sub models, and incorporate JS divergence and adversarial losses of these output distributions to reinforce output stability. During backward propagation stage, we compute parameter saliency scores and selectively update only the most critical parameters to minimize unnecessary deviations and consolidate the model's resilience. Theoretical analysis shows that our framework includes gradient regularization to limit the model's sensitivity to input perturbations and selective parameter updates to flatten the loss landscape, thus improving both generalization and robustness. The experimental results show that our method significantly improves the generalization and robustness of LMs compared to other existing methods across 13 publicly available language datasets, achieving state-of-the-art (SOTA) performance.
