Quanming Yao’s research while affiliated with Tsinghua University and other places

Drug-drug interaction (DDI) prediction is critical for treatment safety. While large language models (LLMs) show promise in pharmaceutical tasks, their effectiveness in DDI prediction remains challenging. Inspired by the well-established clinical practice where physicians routinely reference similar historical cases to guide their decisions through case-based reasoning (CBR), we propose CBR-DDI, a novel framework that distills pharmacological principles from historical cases to improve LLM reasoning for DDI tasks. CBR-DDI constructs a knowledge repository by leveraging LLMs to extract pharmacological insights and graph neural networks (GNNs) to model drug associations. A hybrid retrieval mechanism and dual-layer knowledge-enhanced prompting allow LLMs to effectively retrieve and reuse relevant cases. We further introduce a representative sampling strategy for dynamic case refinement. Extensive experiments demonstrate that CBR-DDI achieves state-of-the-art performance, with a significant 28.7% accuracy improvement over both popular LLMs and CBR baseline, while maintaining high interpretability and flexibility.

As higher-level intelligence emerges from the combination of modular components with lower-level intelligence, many works combines Large Language Models (LLMs) for collective intelligence. Such combination is achieved by building communications among LLMs. While current systems primarily facilitate such communication through natural language, this paper proposes a novel paradigm of direct dense vector communication between LLMs. Our approach eliminates the unnecessary embedding and de-embedding steps when LLM interact with another, enabling more efficient information transfer, fully differentiable optimization pathways, and exploration of capabilities beyond human heuristics. We use such stripped LLMs as vertexes and optimizable seq2seq modules as edges to construct LMNet, with similar structure as MLPs. By utilizing smaller pre-trained LLMs as vertexes, we train a LMNet that achieves comparable performance with LLMs in similar size with only less than 0.1% training cost. This offers a new perspective on scaling for general intelligence rather than training a monolithic LLM from scratch. Besides, the proposed method can be used for other applications, like customizing LLM with limited data, showing its versatility.

The binary relational knowledge base (KB, a.k.a. knowledge graph), representing real-world knowledge with binary relations and entities, has been an important research topic in artificial intelligence, while, considerable knowledge also involves beyond-binary relations. Recently, the area proposes to model n-ary relational KBs with both binary and beyond-binary relations included. However, most current models are extended from translational distance and neural network models in binary relational KBs, which suffer from weak expressiveness and high complexity respectively. To overcome such issues, in this work, we propose a novel two-step modeling framework, GETD, generalizing the powerful tensor decomposition technique from binary relational KBs to the n-ary case. For n-ary relational KBs with single-arity relations, the GETD framework introduces Tucker decomposition and Tensor Ring decomposition for expressive and efficient modeling. Furthermore, the framework is technically extended for the representation of n-ary relational KBs with mixed-arity relations. The existing negative sampling technique is also generalized to the n-ary case for GETD. In addition, we theoretically prove that the GETD framework is fully expressive to completely represent any KBs. Empirical results on two representative datasets show that the proposed framework significantly outperforms the state-of-the-art methods, achieving 11%-26% and 4%-7% improvements on Hits@10 for the single-arity and the mixed-arity cases respectively.

Model merging is a critical technique for combining the capabilities of multiple fine-tuned models without requiring additional training. While existing methods treat parameters as vectors, they overlook the intrinsic structure of linear transformation matrices - the core components that comprise the majority of model parameters. These matrices are fundamental to neural networks, mapping input representations to output features through linear combinations. Motivated by the linear representation hypothesis, we introduce task matrix and propose to Superpose Features from Task Matrix (SFTM), a novel approach that superposes features from individual task models into a merged model. SFTM employs singular value decomposition to identify feature bases of linear transformation matrices and solves a linear system to optimally combine them while preserving input-output mappings from individual task models. Extensive experiments on vision transformers and language models demonstrate that our method consistently outperforms existing methods, achieving superior performance and enhanced out-of-distribution generalization.

Heterogeneous graphs generally refer to graphs with different types of nodes and edges. A common approach for extracting useful information from heterogeneous graphs is to use meta‐graphs, which can be seen as a special kind of directed acyclic graph with same node and edge types as the heterogeneous graph. However, how to design proper meta‐graphs is challenging. Recently, there have been many works on learning suitable meta‐graphs from a heterogeneous graph. Existing methods generally introduce continuous weights for edges that are independent of each other, which ignores the topological structures of meta‐graphs and can be ineffective. To address this issue, the authors propose a new viewpoint from tensor on learning meta‐graphs. Such a viewpoint not only helps interpret the limitation of existing works by CANDECOMP/PARAFAC (CP) decomposition, but also inspires us to propose a topology‐aware tensor decomposition, called TENSUS, that reflects the structure of DAGs. The proposed topology‐aware tensor decomposition is easy to use and simple to implement, and it can be taken as a plug‐in part to upgrade many existing works, including node classification and recommendation on heterogeneous graphs. Experimental results on different tasks demonstrate that the proposed method can significantly improve the state‐of‐the‐arts for all these tasks.

The brute‐force scaleup of training datasets, learnable parameters and computation power, has become a prevalent strategy for developing more robust learning models. However, due to bottlenecks in data, computation, and trust, the sustainability of this strategy is a serious concern. In this paper, we attempt to address this issue in a parsimonious manner (i.e., achieving greater potential with simpler models). The key is to drive models using domain‐specific knowledge, such as symbols, logic, and formulas, instead of purely relying on scaleup. This approach allows us to build a framework that uses this knowledge as “building blocks” to achieve parsimony in model design, training, and interpretation. Empirical results show that our methods surpass those that typically follow the scaling law. We also demonstrate our framework in AI for science, specifically in the problem of drug‐drug interaction prediction. We hope our research can foster more diverse technical roadmaps in the era of foundation models.

Network Function Virtualization (NFV) improves the flexibility and scalability of network services and reduces operating costs. As the core of research on network virtualization, Virtual Network Embedding (VNE) aims to effectively deploy service requests on physical network components and allocate underlying physical resources. However, network services can be complex and diverse, which makes it difficult for existing embedding methods to effectively utilize the graph structure of services, tackle the complexity of dynamic networks, and provide effective embedding solutions. To this end, we propose GPRL, an online VNE method based on graph pointer network and Deep Reinforcement Learning (DRL). By combining the graph neural network and pointer network, we design a novel graph pointer network as the DRL agent. It employs the graph attention network to encode graph feature data and decodes to output the embedding policy via the pointer network architecture. Furthermore, the Proximal Policy Optimization (PPO) algorithm is used to effectively train the designed agent. The effectiveness and superiority of GPRL are verified by simulation experiments, and GPRL is shown to perform better than existing embedding methods.