Yuan Yuan’s research while affiliated with Chinese Academy of Sciences and other places

Low-Rank Adaptation (LoRA) is a crucial method for efficiently fine-tuning pretrained large language models (LLMs), with its performance largely influenced by two key factors: rank and initialization strategy. Numerous LoRA variants have been proposed to enhance its performance by addressing these factors. However, these variants often compromise LoRA's usability or efficiency. In this paper, we analyze the fundamental limitations of existing methods and introduce a novel approach, GoRA (Gradient-driven Adaptive Low Rank Adaptation), which adaptively assigns ranks and initializes weights for low-rank adapters simultaneously based on gradient information. Extensive experimental results demonstrate that GoRA significantly improves performance while preserving the high usability and efficiency of LoRA. On the T5 model fine-tuned for the GLUE benchmark, GoRA achieves a 5.88-point improvement over LoRA and slightly surpasses full fine-tuning. Similarly, on the Llama3.1-8B-Base model fine-tuned for GSM8k tasks, GoRA outperforms LoRA with a 5.13-point improvement and exceeds full fine-tuning in high-rank settings by a margin of 2.05 points.

The emergence of the HotpotQA dataset addressed the lack of training datasets on multi-hop question answering. Based on the strengths of this dataset, we proposed a novel model applicable to multi-hop question answering, called it ELECTRA-based Graph Network model (EGN). First, the method was able to correlate questions with contextual paragraphs and external Wikipedia data to naturally obtain next-hop connected paragraph, initialized the text data with a pre-trained context encoder, Efficiently Learning an Encoder that Classifies Token Re-placements Accurately (ELECTRA). Second, it iterated and updated text features at different levels with the modified Graph Attention Network (GATv2) network. EGN was able to achieve a comparable result in less time with the iterative computation of GATv2 by linking more sensible clues and using ELECTRA to obtain a better representation of the data. In the experiments, EGN performed well with the FullWiki setting on the HotpotQA validation dataset, achieving a Joint EM/F1 score of 47.35/74.62 on the validation set.

Traditional search engines retrieve relevant web pages based on keywords in the entered questions, while sometimes the required information may not be included in these keyword-based retrieved web pages. Compared to the search engines, the question answering system can provide more accurate answers. However, traditional question answering systems can only provide answers to users based on matching the questions in a question answering pair. At the same time, the number of question answering pairs remain somewhat limited. As a result, the user’s requirements cannot be met well. In contrast, knowledge graphs can store information such as entities and their relationships in a structured pattern. Therefore, the knowledge graph is highly scalable as the data is stored in a structured form. Besides, the relationship between entities and the knowledge graph structure allows the desired answer to be found quickly. Moreover, the process of relation classification can also be regarded as an operation of text classification. Therefore, this study proposed a new approach to knowledge graph-based question answering systems that require a named entity recognition method and a multi-label text classification method to search for the answers. The results of entity name and question type are turned into a Cypher query that searches for the answer in the knowledge graph. In this paper, three models, i.e., TextCNN, bi-LSTM, and bi-LSTM + Att, are used to examine the effectiveness of multi-label text classification, demonstrating our method’s feasibility. Among these three models, TextCNN worked best, attaining an F1 score of 0.88.

Recurrent Neural Network (RNN) based abstractive text summarization models have made great progress over the past few years, largely triggered by the encoder-decoder architecture. However, there has been little work improving the generation of relatively long summaries. In this paper, we concentrate on two prominent problems in long summary generation. First, although significant efforts have been made to assist the encoder in handling long sequences, the decoder struggles with long sequences owing to the limited storage capacity of RNN. We propose a simple and effective approach called history reuse, which first mines critical information from the history summary sequence and then transmits the information to the decoder. Second, since encoder-decoder models are typically trained to produce exactly the same summary as the target summary, certain word order deviations between the predicted summary and target summary are excessively punished. Accordingly, we introduce a fully differentiable loss called bag-of-words (BoW) loss, which takes advantage of the feature of BoW discarding word order information in texts, and computes the difference between the two summaries at the BoW space. Experiments on two benchmark datasets, CNN/Daily Mail and Pubmed, demonstrate that our methods significantly improve the baseline.