Licheng Jiao’s research while affiliated with Xidian University and other places

Automated water body (WB) extraction is one of the hot research topics in the field of remote sensing image processing. To address the challenges of over-extraction and incomplete extraction in complex water scenes, we propose an encoder–decoder architecture semantic segmentation network for high-precision extraction of WBs called EDWNet. We integrate the Cross-layer Feature Fusion (CFF) module to solve difficulties in segmentation of WB edges, utilizing the Global Attention Mechanism (GAM) module to reduce information diffusion, and combining with the Deep Attention Module (DAM) module to enhance the model’s global perception ability and refine WB features. Additionally, an auxiliary head is incorporated to optimize the model’s learning process. In addition, we analyze the feature importance of bands 2 to 7 in Landsat 8 OLI images, constructing a band combination (RGB 763) suitable for algorithm’s WB extraction. When we compare EDWNet with various other semantic segmentation networks, the results on the test dataset show that EDWNet has the highest accuracy. EDWNet is applied to accurately extract WBs in the Weihe River basin from 2013 to 2021, and we quantitatively analyzed the area changes of the WBs during this period and their causes. The results show that EDWNet is suitable for WB extraction in complex scenes and demonstrates great potential in long time-series and large-scale WB extraction.

Road++ Track3 proposes a multi-label atomic activity recognition task in traffic scenarios, which can be standardized as a 64-class multi-label video action recognition task. In the multi-label atomic activity recognition task, the robustness of visual feature extraction remains a key challenge, which directly affects the model performance and generalization ability. To cope with these issues, our team optimized three aspects: data processing, model and post-processing. Firstly, the appropriate resolution and video sampling strategy are selected, and a fixed sampling strategy is set on the validation and test sets. Secondly, in terms of model training, the team selects a variety of visual backbone networks for feature extraction, and then introduces the action-slot model, which is trained on the training and validation sets, and reasoned on the test set. Finally, for post-processing, the team combined the strengths and weaknesses of different models for weighted fusion, and the final mAP on the test set was 58%, which is 4% higher than the challenge baseline.

Solving the complex challenges of sophisticated terrain and multi-scale targets in remote sensing (RS) images requires a synergistic combination of Transformer and convolutional neural network (CNN). However, crafting effective CNN architectures remains a major challenge. To address these difficulties, this study introduces the knowledge guided evolutionary Transformer for RS scene classification (Evo RSFormer). It amalgamates adaptive evolutionary CNN (Evo CNN) with Transformers in a hybrid strategy synergistically, which combines fine-grained local feature extraction of CNNs with long-range contextual dependency modeling of Transformers. Furthermore, for the development of Evo CNN blocks, this paper presents a knowledge-guided adaptive efficient multi-objective evolutionary neural architecture search (MOE ² -NAS) strategy. This approach markedly diminishes the labor-intensive characteristics associated with traditional CNN design, striking a balance for both accuracy and compactness. Additionally, by leveraging domain knowledge from natural scene analysis into the RS field, MOE ² -NAS facilitates the efficiency of classical NAS. It utilizes a priori knowledge to generate promising initial solutions and constructs a surrogate model for efficient search. The effectiveness of the proposed Evo RSFormer has been rigorously tested on various benchmark RS datasets, including UC Merced, NWPU45, and AID. Empirical results strongly support the superiority of Evo RSFormer over existing methods. Furthermore, experiments on MOE ² -NAS have been studied to confirm the important role of knowledge guidance in improving the efficiency of NAS.

Recently, contrastive learning (CL) has exhibited considerable advantages for automatic modulation classification (AMC) with a scarcity of labeled samples. Nevertheless, the majority of the available CL-based AMC methods use the simple signal augmentation strategy and suffer from interference from false negatives. To explore the more generalizable global temporal semantics within signals, a novel masked contrastive learning with hard negatives (MCLHN) method is proposed in this paper. MCLHN first strategically incorporates semantic-preserving data augmentation, ensuring the diversity and semantic invariance of signals. Second, MCLHN adopts an encoder with temporal masking to enable robust temporal modeling. Moreover, a debiased hardness-weighted contrastive (DHWC) loss is designed to balance the adverse impact of the debiased strategy and the advantage of hard negatives. Extensive experiments are conducted on several benchmark datasets, and the experimental results demonstrate the superior performance and generalization capability of MCLHN to other methods. Significantly, the performance of MCLHN with only one labeled sample per modulation under each signal-to-noise ratio (SNR) rivals that of other methods with five to twenty times the number of labeled samples.

With the development of multimedia technology, events are usually presented in multimedia forms, thus multimedia event extraction (MEE) has become more and more important. Existing MEE works usually use simple strategies to align two modalities, making it difficult to precisely extract events and arguments in complex multimedia documents. To address this problem, we propose a novel Multi-grained Gradual Inference Model (MGIM) that focuses on inferring and interpreting events in complex multimedia structures in a coarse-to-fine manner. To efficiently integrate textual and visual modalities, we design a Coarse-grained Alignment (CA) module, which represents the two modalities in a graph structure and performs coarse-grained alignment. Based on the CA module, we further propose a Fine-grained Inference module (FI) that fine-grained aligns text and image by performing multiple rounds of gradual inference. MGIM provides a comprehensive interpretation of multimedia events at two information granularities (coarse and fine). Extensive experiments on the M2E2 dataset demonstrate the effectiveness of MGIM.

We propose an adaptive fine-tuning algorithm for multimodal large models. The core steps of this algorithm involve two stages of truncation. First, the vast amount of data is projected into a semantic vector space, and the MiniBatchKMeans algorithm is used for automated clustering. This classification ensures that the data within each cluster exhibit high semantic similarity. Next, we process the data in each cluster, calculating the translational difference between the original and perturbed data in the multimodal large model's vector space. This difference serves as a generalization metric for the data. Based on this metric, we select the data with high generalization potential for training. We applied this algorithm to train the InternLM-XComposer2-VL-7B model on two 3090 GPUs using one-third of the GeoChat multimodal remote sensing dataset. The results demonstrate that our algorithm outperforms the state-of-the-art baselines. various baselines. The model trained on our optimally chosen one-third dataset, based on experimental validation, exhibited only 1% reduction in performance across various remote sensing metrics compared to the model trained on the full dataset. This approach significantly preserved general-purpose capabilities while reducing training time by 68.2%. Furthermore, the model achieved scores of 89.86 and 77.19 on the UCMerced and AID evaluation datasets, respectively, surpassing the GeoChat dataset by 5.43 and 5.16 points. It only showed a 0.91-point average decrease on the LRBEN evaluation dataset.

Existing efforts are dedicated to designing many topologies and graph-aware strategies for the graph Transformer, which greatly improve the model’s representation capabilities. However, manually determining the suitable Transformer architecture for a specific graph dataset or task requires extensive expert knowledge and laborious trials. This article proposes an evolutionary graph Transformer architecture search (EGTAS) framework to automate the construction of strong graph Transformers. We build a comprehensive graph Transformer search space with the micro-level and macro-level designs. EGTAS evolves graph Transformer topologies at the macro level and graph-aware strategies at the micro level. Furthermore, a surrogate model based on generic architectural coding is proposed to directly predict the performance of graph Transformers, substantially reducing the evaluation cost of evolutionary search. We demonstrate the efficacy of EGTAS across a range of graph-level and node-level tasks, encompassing both small-scale and large-scale graph datasets. Experimental results and ablation studies show that EGTAS can construct high-performance architectures that rival state-of-the-art manual and automated baselines.