Wei Lu’s research while affiliated with Tianjin University and other places

Colorectal cancer is a multifaceted disease, but it can be effectively prevented through colonoscopy for the detection of polyps. In clinical practice, the development of automatic polyp segmentation techniques for colonoscopy images can significantly enhance the efficiency and accuracy of polyp detection, and help clinicians to precisely localize the polyps. However, existing segmentation methods have several obvious limitations: (1) inadequate utilization of multi-level features extracted by feature encoders, (2) ineffective aggregation of high-level and low-level features, and (3) unclear delineation of polyp boundaries. To address these challenges while enhancing the clarity of polyp boundaries in segmentation, we propose a novel Multi-layer Information Fusion and Optimization Network (MIFONet) consisting of the following components: (1) Contextual and Fine Feature Processing (CFFP) module, employed to effectively extract both local and global contextual information, (2) Hierarchical Feature Integration Module (HFIM), added to facilitate efficient aggregation of processed high-level and low-level features and strengthen the association between contextual features, (3) Multi-Scale Contextual Attention (MSCA) module, used to deeply integrate aggregated high-level features with low-level features, and (4) a novel refinement module composed of an Adaptive Channel Attention Pyramid (ACAP) part and a Skip-Reverse Attention (SRA) part, with the ability of capturing fine-grained information and refining feature representation. We conducted extensive experiments and comparative analysis of our proposed model with 19 popular or state-of-the-art (SOTA) methods on five renowned polyp benchmark datasets. To further validate the model’s generalization performance, we also designed three cross-dataset experiments. Experimental results demonstrate that MIFONet consistently achieves excellent segmentation performance across most datasets. Especially, we achieve 94.6% mean Dice on CVC-ClinicDB dataset which obtains the superior performance compared with SOTA methods.

The increasing commercial value of micro-videos has spurred a rising demand for grasping their contents. The abundant multimodal cues in micro-videos exhibit substantial potential in enhancing content comprehension. However, effectively harnessing the collaborative characteristics across different modalities remains a significant challenge, especially in multi-label scenarios due to inconsistent behaviors regarding label correlations. To better tackle this issue, in this paper, we first introduce a multimodal dual-graph collaborative network with serial attentive aggregation mechanism (MDGCN) for micro-video multi-label classification. In MDGCN, we exploit an asymmetric encoder-decoder framework, which incorporates multiple parallel encoders with complementary representations and a decoder to ensure the completeness of encoded results. Meanwhile, an adversarial constraint is used to ensure individual differences prominently featured within each modality. Furthermore, considering the inconsistency of label correlations across various modalities, we then construct a serial attentive graph convolutional network that employs an interactive dual-graph attention paradigm to sequentially integrate multimodal representations and dynamically explore label correlations. The experiments conducted on two datasets demonstrate that our proposed method outperforms state-of-the-art approaches.

Early detection of colonic polyps is crucial for the prevention and diagnosis of colorectal cancer. Currently, deep learning-based polyp segmentation methods have become mainstream and achieved remarkable results. Acquiring a large number of labeled data is time-consuming and labor-intensive, and meanwhile the presence of numerous similar wrinkles in polyp images also hampers model prediction performance. In this paper, we propose a novel approach called Phase- wise Feature Pyramid with Retention Network (PFPRNet), which leverages a pre-trained Transformer-based Encoder to obtain multi-scale feature maps. A Phase- wise Feature Pyramid with Retention Decoder is designed to gradually integrate global features into local features and guide the model's attention towards key regions. Additionally, our custom Enhance Perception module enables capturing image information from a broader perspective. Finally, we introduce an innovative Low-layer Retention module as an alternative to Transformer for more efficient global attention modeling. Evaluation results on several widely-used polyp segmentation datasets demonstrate that our proposed method has strong learning ability and generalization capability, and outperforms the state-of-the-art approaches.

Facial semantic guidance (including facial landmarks, facial heatmaps, and facial parsing maps) and facial generative adversarial networks (GAN) prior have been widely used in blind face restoration (BFR) in recent years. Although existing BFR methods have achieved good performance in ordinary cases, these solutions have limited resilience when applied to face images with serious degradation and pose-varied (e.g., looking right, looking left, laughing, etc.) in real-world scenarios. In this work, we propose a well-designed blind face restoration network with generative facial prior. The proposed network is mainly comprised of an asymmetric codec and a StyleGAN2 prior network. In the asymmetric codec, we adopt a mixed multi-path residual block (MMRB) to gradually extract weak texture features of input images, which can better preserve the original facial features and avoid excessive fantasy. The MMRB can also be plug-and-play in other networks. Furthermore, thanks to the affluent and diverse facial priors of the StyleGAN2 model, we adopt it as the primary generator network in our proposed method and specially design a novel self-supervised training strategy to fit the distribution closer to the target and flexibly restore natural and realistic facial details. Extensive experiments on synthetic and real-world datasets demonstrate that our model performs superior to the prior art for face restoration and face super-resolution tasks.

Semi-supervised learning aims to train a high-performance model with a minority of labeled data and a majority of unlabeled data. Existing methods mostly adopt the mechanism of prediction task to obtain precise segmentation maps with the constraints of consistency or pseudo-labels, whereas the mechanism usually fails to overcome confirmation bias. To address this issue, in this paper, we propose a novel Confidence-Guided Mask Learning (CGML) for semi-supervised medical image segmentation. Specifically, on the basis of the prediction task, we further introduce an auxiliary generation task with mask learning, which intends to reconstruct the masked images for extremely facilitating the model capability of learning feature representations. Moreover, a confidence-guided masking strategy is developed to enhance model discrimination in uncertain regions. Besides, we introduce a triple-consistency loss to enforce a consistent prediction of the masked unlabeled image, original unlabeled image and reconstructed unlabeled image for generating more reliable results. Extensive experiments on two datasets demonstrate that our proposed method achieves remarkable performance.

Automatic polyp segmentation is critical for early prevention and diagnosis of colorectal cancer. However, diverse foreground appearance and complicated background interference severely degrade the performance of pixel-level prediction. The excessive computational overheads further hinder the practical clinical applications of existing methods. In this paper, we propose a novel Lesion-Aware Contextual Interaction Network (LACINet), which aims to explore the long-range dependencies and global contexts with friendly computing resource consumption for polyp segmentation. Specifically, we present a Lesion-aware Pyramid Mechanism (LPM) to weaken the influence of background noise and refine lesion-related features. We also develop a robust Representation Enhancement Decoder (RED) to learn global feature representations and aggregate the multi-level contexts. In RED, we first build a Non-local Contextual Lesion Interaction Module (NCLIM) to integrate the cross-level contextual information for obtaining the intrinsic feature representations, and then design a Tri-branching Multi-scale Perceptual Self-attention Module (TMPSM) to sufficiently excavate the global features. Notably, we introduce an asymmetric multi-branch strategy to alleviate the computational burden. The experimental results on several widely-used benchmark datasets demonstrate the superior performance of our proposed LACINet in comparison with state-of-the-art methods.

Video memorability measures the degree to which a video is remembered by different viewers and has shown great potential in various contexts, including advertising, education, and health care. While extensive research has been conducted on image memorability, the study of video memorability is still in its early stages. Existing methods in this field primarily focus on coarse-grained spatial feature representation and decision fusion strategies, overlooking the crucial interactions between spatial and temporal domains. Therefore, we propose an end-to-end collaborative spatial-temporal network called VMemNet, which incorporates targeted attention mechanisms and intermediation fusion strategies. This enables VMemNet to capture the intricate relationships between spatial and temporal information and uncover more elements of memorability within video visual features. VMemNet integrates spatially and semantically guided attention modules into a dual-stream network architecture, allowing it to simultaneously capture static local cues and dynamic global cues in videos. Specifically, the spatial attention module is used to aggregate more memorable elements from spatial locations, and the semantically guided attention module is used to achieve semantic alignment and intermediate fusion of the local and global cues. In addition, two types of loss functions with complementary decision rules are associated with the corresponding attention modules to guide the training process of the proposed network. Experimental results obtained on a publicly available dataset verify that the proposed VMemNet approach outperforms all current single- and multi-modal methods in terms of video memorability prediction.

Currently, micro-videos have attracted increasing attention due to their unique properties and great commercial value. Considering that micro-videos naturally incorporate multimodal information, a powerful representation method for distinct joint multimodal representations is essential for real applications. Inspired by the potential of attention neural network architectures over various tasks, we propose a multimodal aggregation network (MANET) with a serial self-attention mechanism to perform tasks of micro-video multi-label classification. Specifically, we first propose a parallel content-dependent graph neural networks (CDGNN) module, which explores category-related embeddings of micro-videos by disentangling category relations into modality-specific and modality-shared category dependency patterns. Then we introduce a serial self-attention (SSA) module to transmit the multimodal information in sequential order, in which an aggregation bottleneck is incorporated to better collect and condense the significant information. Experiments conducted on a large-scale multi-label micro-video dataset demonstrate that our proposed method has achieved competitive results compared with several state-of-the-art methods.

Speech enhancement system is applied in many devices such as hearing aids. To improve speech quality retrieved from noisy observations, this paper proposes a two-stage network with the frequency-time dilated dense network (FTDDN). This improvement lies in 3 aspects. Firstly, both frequency modeling and temporal modeling are considered to optimize a time-frequency mask; Secondly, to acquire the large receptive field, dilated convolution is incorporated into 3 basic processing units: frequency-dilated convolutional unit (FDCU), time-dilated convolutional unit (TDCU), and frequency-time dilated convolutional unit (FTDCU); Thirdly, for any one of them, 12 units were densely connected to assemble a frequency-dilated dense block (FDDB), a time-dilated dense block (TDDB), or a frequency-time dilated dense block (FTDDB), all of which are combined with some feature mapping operators to build up an FTDDN. With the above considerations, high-quality speech can be retrieved via implementing information reuse and feature fusion operations on two FTDDNs in a two-stage model. Using Librispeech and VCTK data sets, we conducted several experimental comparisons between our method and the state-of-the-art speech enhancement methods, showing that our proposed model outperforms these baseline models.