Huiqun Yu’s research while affiliated with East China University of Science and Technology and other places

Safety is a crucial challenge in the application of reinforcement learning. Multi-agent safe reinforcement learning is an emerging field focused on learning control policies that maximize cumulative rewards while adhering to safety constraints. However, existing research is limited and faces challenges such as environmental non-stationarity and the curse of dimensionality in action spaces, making it difficult to balance performance and safety. To address these, this paper proposes a multi-agent safe reinforcement learning algorithm based on the Transformer (MAST). Our key contribution is the multiagent total advantage decomposition theorem, which establishes the connection between multi-agent safe reinforcement learning and sequence models. MAST employs a Transformer-based actor network that generates joint actions in parallel during training, and autoregressively during inference. Empirical evaluations on the Safe MAMuJoCo benchmark show that MAST achieves a 13.06% improvement over state-of-the-art algorithms. Our attention-based reward and safety critics achieve a 22.10% increase in rewards and an 83.58% reduction in safety costs. Additionally, the Transformer-based actor improves performance by 53.60% to 111.93% compared to RNN-based methods.

Large-scale sparse multiobjective optimization problems (SMOPs) exist widely in real-world applications, and solving them requires algorithms that can handle high-dimensional decision space while simultaneously discovering the sparse distribution of Pareto optimal solutions. However, it is difficult for most existing multiobjective evolutionary algorithms (MOEAs) to get satisfactory results. To address this problem, this article proposes a dynamic knowledge-guided coevolutionary algorithm, which employs a cooperative coevolutionary framework tailored for large-scale SMOPs. Specifically, variable selection is performed initially for the dimension reduction, and two populations are evolved in the original and reduced decision spaces, respectively. After offspring generation, variable replacement is performed to precisely identify the sparse distribution of Pareto optimal solutions. Furthermore, a dynamic score update mechanism is designed based on the discovered sparsity knowledge, which aims to adjust the direction of evolution dynamically. The superiority of the proposed algorithm is demonstrated by applying it to a variety of benchmark test instances and real-world test instances with the comparison of five other state-of-the-art MOEAs.

Edge computing extends computing resources from the data center to the edge of the network to better handle latency‐sensitive tasks. However, with the rise of the Internet of Things, edge devices with limited processing capabilities face difficulties in executing requests with fluctuating request peaks. In order to meet the deadline constraints of latency‐sensitive tasks, a feasible solution is to offload some latency‐sensitive tasks to other nearby edge devices. This article studies the problem of request migration in edge computing systems and minimizes the request deadline violation rate based on actual online arrival patterns, performance interference phenomena, and deadline constraints. Since a request contains multiple services and request migration will lead to changes in server resource competition pressure, we split the problem into three sub‐problems, dividing the request deadline to determine the maximum response time of the service, determining the performance of the service under different resource pressures and the request migration strategies. To this end, we propose two deadline splitting methods, a performance interference model under multi‐resource pressure, and two heuristic request migration strategies. Since this article considers online edge scenarios, the number and type of requests are black boxes. We conduct simulation experiments and find that our method has only one‐third the number of request violations of other methods.

Protein structures are important for understanding their functions and interactions. Currently, many protein structure prediction methods are enriching the structure database. Discriminating the origin of structures is crucial for distinguishing between experimentally resolved and computationally predicted structures, evaluating the reliability of prediction methods, and guiding downstream biological studies. Building on works in structure prediction, We developed a structure-sensitive supervised deep learning model, Crystal vs Predicted Evaluator for Protein Structure (CPE-Pro), to represent and discriminate the origin of protein structures. CPE-Pro learns the structural information of proteins and captures inter-structural differences to achieve accurate traceability on four data classes, and is expected to be extended to more. Simultaneously, we utilized Foldseek to encode protein structures into "structure-sequence" and trained a protein Structural Sequence Language Model, SSLM. Preliminary experiments demonstrated that, compared to large-scale protein language models pre-trained on vast amounts of amino acid sequences, the "structure-sequences" enable the language model to learn more informative protein features, enhancing and optimizing structural representations. We have provided the code, model weights, and all related materials on https://github.com/GouWenrui/CPE-Pro-main.git.

Code summarization aims to automatically generate natural language descriptions for code, and has become a rapidly expanding research area in the past decades. Unfortunately, existing approaches mainly focus on the “one-to-one” mapping from methods to short descriptions, which hinders them from becoming practical tools: 1) The program context is ignored, so they have difficulty in predicting keywords outside the target method; 2) They are typically trained to generate brief function descriptions with only one sentence in length, and therefore have difficulty in providing specific information. These drawbacks are partially due to the limitations of public code summarization datasets. In this paper, we first build a large code summarization dataset including different code contexts and summary content annotations, and then propose a deep learning framework that learns to generate structured code summaries from hybrid program context, named StructCodeSum. It provides both an LLM-based approach and a lightweight approach which are suitable for different scenarios. Given a target method, StructCodeSum predicts its function description, return description, parameter description, and usage description through hybrid code context, and ultimately builds a Javadoc-style code summary. The hybrid code context consists of path context, class context, documentation context and call context of the target method. Extensive experimental results demonstrate: 1) The hybrid context covers more than 70% of the summary tokens and significantly boosts the model performance; 2) When generating function descriptions, StructCodeSum outperforms the state-of-the-art approaches by a large margin; 3) According to human evaluation, the quality of the structured summaries generated by our approach is better than the documentation generated by Code Llama.

Protein language models (PLMs) play a dominant role in protein representation learning. Most existing PLMs regard proteins as sequences of 20 natural amino acids. The problem with this representation method is that it simply divides the protein sequence into sequences of individual amino acids, ignoring the fact that certain residues often occur together. Therefore, it is inappropriate to view amino acids as isolated tokens. Instead, the PLMs should recognize the frequently occurring combinations of amino acids as a single token. In this study, we use the byte-pair-encoding algorithm and unigram to construct advanced residue vocabularies for protein sequence tokenization, and we have shown that PLMs pre-trained using these advanced vocabularies exhibit superior performance on downstream tasks when compared to those trained with simple vocabularies. Furthermore, we introduce PETA, a comprehensive benchmark for systematically evaluating PLMs. We find that vocabularies comprising 50 and 200 elements achieve optimal performance. Our code, model weights, and datasets are available at https://github.com/ginnm/ProteinPretraining. Scientific contribution This study introduces advanced protein sequence tokenization analysis, leveraging the byte-pair-encoding algorithm and unigram. By recognizing frequently occurring combinations of amino acids as single tokens, our proposed method enhances the performance of PLMs on downstream tasks. Additionally, we present PETA, a new comprehensive benchmark for the systematic evaluation of PLMs, demonstrating that vocabularies of 50 and 200 elements offer optimal performance. Graphical Abstract

With growing demand for privacy-preserving reinforcement learning (RL) applications, federated RL (FRL) has emerged as a potential solution. However, existing FRL methods struggle with multiple sources of heterogeneity, while lacking robust privacy guarantees. In this study, we propose DPA-FedRL, the dynamic privacy-aware FRL framework, to simultaneously mitigate both issues. First, we innovatively put forward the concept of “multiheterogeneity” and embed the environmental heterogeneity into agents’ state representations. Next, to ensure privacy during model aggregation, we incorporate a differentially private mechanism in form of Gaussian noise and modify its global sensitivity, tailored to suit FRL’s unique characteristics. Encouragingly, our approach dynamically allocates privacy budget based on heterogeneity levels, which strikes a balance between privacy and utility. From the theoretical perspective, we give rigorous convergence, privacy, and sensitivity guarantees for our proposed method. Through extensive experiments on diverse datasets, we demonstrate that DPA-FedRL surpasses state-of-the-art approaches (PPO-DP-SGD, PAvg, and QAvg) in some highly heterogeneous environments. Notably, our novel privacy attack simulations enable quantitative privacy assessment, validating that DPA-FedRL offers over 1.359 $\times$ stronger protection than baselines.

Protein retrieval, which targets the deconstruction of the relationship between sequences, structures and functions, empowers the advancing of biology. Basic Local Alignment Search Tool (BLAST), a sequence-similarity-based algorithm, has proved the efficiency of this field. Despite the existing tools for protein retrieval, they prioritize sequence similarity and probably overlook proteins that are dissimilar but share homology or functionality. In order to tackle this problem, we propose a novel protein retrieval framework that mitigates the bias towards sequence similarity. Our framework initiatively harnesses protein language models (PLMs) to embed protein sequences within a high-dimensional feature space, thereby enhancing the representation capacity for subsequent analysis. Subsequently, an accelerated indexed vector database is constructed to facilitate expedited access and retrieval of dense vectors. Extensive experiments demonstrate that our framework can equally retrieve both similar and dissimilar proteins. Moreover, this approach enables the identification of proteins that conventional methods fail to uncover. This framework will effectively assist in protein mining and empower the development of biology.