Yu Meng’s research while affiliated with Aerospace Information Research Institute, Chinese Academy of Sciences and other places

This study addresses the technical bottlenecks in handling long text and the "hallucination" issue caused by insufficient short text information in remote sensing vision-language foundation models (VLFM). We propose a novel vision-language foundation model, LRSCLIP, and a multimodal dataset, LRS2M. The main contributions are as follows: (1) By integrating multi-source remote sensing data and adopting a large language model labeling strategy, we construct the LRS2M dataset, which contains 2 million image-text pairs, providing both short and long texts for the first time, thus solving the problem of semantic granularity limitations in existing datasets; (2) The design of the LRSCLIP architecture based on Long-CLIP's KPS module, which extends CLIP's text processing capacity and achieves fine-grained cross-modal feature alignment through a dual-text loss weighting mechanism. Experimental results show that LRSCLIP improves retrieval accuracy by 10\%-20\% over the Long-CLIP baseline in the zero-shot long-text cross-modal retrieval task. For the zero-shot short-text cross-modal retrieval task, LRSCLIP achieves improvements over the current best model, GeoRSCLIP, with increases of 0.17\%, 0.67\%, and 0.92\% in Text to Image R@1, Image to Text R@1, and mR on RSITMD, respectively, and 0.04\%, 2.93\%, and 1.28\% on RSICD. In the zero-shot image classification task (average accuracy=75.75\%) and semantic localization task (Rmi=0.7653), LRSCLIP achieves state-of-the-art performance. These results validate the dual advantages of fine-grained semantic understanding and global feature matching in LRSCLIP. This work provides a new benchmark model and data support for remote sensing multimodal learning. The related code has been open source and is available at https://github.com/MitsuiChen14/LRSCLIP.

Recent advances in diffusion models have led to significant progress in audio-driven lip synchronization. However, existing methods typically rely on constrained audio-visual alignment priors or multi-stage learning of intermediate representations to force lip motion synthesis. This leads to complex training pipelines and limited motion naturalness. In this paper, we present SayAnything, a conditional video diffusion framework that directly synthesizes lip movements from audio input while preserving speaker identity. Specifically, we propose three specialized modules including identity preservation module, audio guidance module, and editing control module. Our novel design effectively balances different condition signals in the latent space, enabling precise control over appearance, motion, and region-specific generation without requiring additional supervision signals or intermediate representations. Extensive experiments demonstrate that SayAnything generates highly realistic videos with improved lip-teeth coherence, enabling unseen characters to say anything, while effectively generalizing to animated characters.

In this study, we propose a one-branch IncepTAE network to extract local and global hybrid temporal attention simultaneously and congruously for fine-grained satellite image time series (SITS) classification. Transformer and the temporal self-attention mechanism have been the research focus of SITS classification in recent years. However, its effectiveness seems to diminish in the scenario of fine-grained classification among similar categories, for example, different crop types. Theoretically, most of the existing methods focus on only one type of temporal attention, either global attention or local attention, but actually, both of them are required to achieve fine-grained classification. Even though some works adopt two-branch architecture to extract hybrid attention, they usually lack congruity between different types of temporal attention and hinder the expected discriminating ability. Compared with the existing methods, IncepTAE exhibits multiple methodological novelties. Firstly, we insert average/maximum pooling layers into the calculation of multi-head attention to extract hybrid temporal attention. Secondly, IncepTAE adopts one-branch architecture, which reinforces the interaction and congruity of different temporal information. Thirdly, the proposed IncepTAE is more lightweight due to the use of group convolutions. IncepTAE achieves 95.65% and 97.84% overall accuracy on two challenging datasets, TimeSen2Crop and Ghana. The comparative results with existing state-of-the-art methods demonstrate that IncepTAE is able to achieve superior classification performance and faster inference speed, which is conducive to the large-area application of SITS classification.

Building Footprint Extraction (BFE) in off-nadir aerial images often relies on roof segmentation and roof-to-footprint offset prediction, then drugging roof-to-footprint via the offset. However, the results from this multi-stage inference are not applicable in data production, because of the low quality of masks given by prediction. To solve this problem, we proposed OBMv2 in this paper, which supports both end-to-end and promptable polygonal footprint prediction. Different from OBM, OBMv2 using a newly proposed Self Offset Attention (SOFA) to bridge the performance gap on bungalow and skyscraper, which realized a real end-to-end footprint polygon prediction without postprocessing. %, such as Non-Maximum Suppression (NMS) and Distance NMS (DNMS). % To fully use information contained in roof masks, building masks and offsets, we proposed a Multi-level Information SyStem (MISS) for footprint prediction, with which OBMv2 can predict footprints even with insufficient predictions. Additionally, to squeeze information from the same model, we were inspired by Retrieval-Augmented Generation (RAG) in Nature Language Processing and proposed "RAG in BFE" problem. To verify the effectiveness of the proposed method, experiments were conducted on open datasets BONAI and OmniCity-view3. A generalization test was also conducted on Huizhou test set. The code will be available at \url{https://github.com/likaiucas/OBM}.

The abundant details embedded in very-high-resolution remote sensing images establish a solid foundation for comprehending the land surface. Simultaneously, as spatial resolution advances, there is a corresponding escalation in the required granularity of land cover and land use (LCLU) categories. The coarse classes identified necessitate further refinement into more detailed categories. For instance, the ‘built-up’ class can be subdivided into specific categories such as squares, stadiums, and airports. These refined LCLU classifications are better equipped to support diverse domains. Nonetheless, most studies simply adopt methods initially designed for coarse LCLU when addressing the challenging refined LCLU segmentation. Few studies have considered the inherent relationships between coarse and refined LCLU, overlooking the potential exploitation of the numerous recently released LCLU products. To better leverage this prior knowledge, we propose the Global Heterogeneous Graph Convolutional Network (GHGCN). The GHGCN introduces a heterogeneous graph and excels in establishing relationships between coarse and refined LCLU, which can extract long-distance dependencies more effectively than convolutional neural networks. Furthermore, this model is performed end-to-end, eliminating the necessity for presegmentation and facilitating training acceleration. GHGCN exhibits competitive performance compared to state-of-the-art models, indicating its effective design in exploiting coarse LCLU data, especially for categories with limited samples. The source code is released at: https://github.com/Liuzhizhiooo/GHGCN.

More accurate extraction of invisible building footprints from very-high-resolution (VHR) aerial images relies on roof segmentation and roof-to-footprint offset extraction. Existing methods based on instance segmentation suffer from poor generalization when extended to large-scale data production and fail to achieve low-cost human interaction. This prompt paradigm inspires us to design a promptable framework for roof and offset extraction, and transforms end-to-end algorithms into promptable methods. Within this framework, we propose a novel Offset-Building Model (OBM). Based on prompt prediction, we first discover a common pattern of predicting offsets and tailored Distance-NMS (DNMS) algorithms for offset optimization. To rigorously evaluate the algorithm’s capabilities, we introduce a prompt-based evaluation method, where our model reduces offset errors by 16.6% and improves roof Intersection over Union (IoU) by 10.8% compared to other models. Leveraging the common patterns in predicting offsets, DNMS algorithms enable models to further reduce offset vector loss by 6.5%. To further validate the generalization of models, we tested them using a newly proposed test set, Huizhou test set, with over 7,000 manually annotated instance samples. Our algorithms and dataset will be available at https://github.com/likaiucas/OBM.

With the advancement of remote sensing satellite technology, the acquisition of Satellite Image Time Series (SITS) data has significantly increased, providing new opportunities and challenges for land cover analysis. Traditional unsupervised clustering methods often struggle with the complexity of these data due to limitations in scalability and generalization capabilities. In response, this paper proposes a new unsupervised learning approach called Deep Temporal Joint Clustering (DTJC) designed for efficient pixel-wise clustering of SITS data. DTJC optimizes the reconstruction of temporal information along with clustering objectives, which not only preserves the temporal dynamics of the original data but also creates a feature space conducive to clustering. Experimental results show that DTJC achieves optimal clustering performance across four publicly available multi-spectral SITS datasets, including TimeSen2Crop, Cerrado Biome, Reunion Island, and Imperial datasets. Compared to traditional K-means and projection algorithms, DTJC significantly improves clustering accuracy, especially in environments with complex geographical distributions. Leveraging the principles of the K-means clustering algorithm, DTJC showcases remarkable performance improvements over traditional optimized K-means and projection algorithms in land cover analysis, heralding a new era in the unsupervised learning landscape of SITS data. The DTJC method greatly enhances the efficiency of SITS data analysis without the need for labeled data, making it a powerful tool for automated land cover classification and environmental monitoring.