Kaiming He’s research while affiliated with Meta and other places

... As these models are trained to learn visual representations aligned with the language representation, most MLLMs [1,41,46,47,86] adopt these vision encoders to process the image input. However, instancelevel contrastive learning suffers from feature suppression, where the model learns only the dominant features in the data while neglecting other valuable features [12,42,44,60,66,69,75,82]. In other words, the model creates so-called simple shortcut features and decision rules that do not consider all available distinguishing features. ...

Reference:

COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training

... The integration of augmented reality and computer vision holds promise for object detection [11]. A non-hierarchical Vision Transformer (ViT) serves as the primary object detection network [12], while an enhanced multiscale feature fusion method improves small object detection performance [13]. A Polar Transformer (Polar-Former) enhances the accuracy of 3D object detection in a bird'seye-view system [14]. ...

Reference:

Object Detection Using TensorFlow for Road Navigation: Enhancing Safety for the Visually Impaired

... The attention blocks within SAM, pre-trained with MAE [35], encapsulate a wealth of insights for token embedding analysis, making them pivotal components within the SAM framework. To harness these insights while facilitating memory-efficient training, we choose to preserve the parameters in attention blocks. ...

Reference:

Volumetric medical image segmentation via fully 3D adaptation of Segment Anything Model

... Among various neural networks that have been successfully applied in structural engineering [26][27][28][29][30], an auto-encoder is a specialized deep-learning architecture designed to learn a compact representation of data that encodes the most meaningful information [31]. The authors postulate that the learned compact data representation of an auto-encoder architecture will filter out noise, anomalies, redundant information, and other spurious artifacts. ...

Reference:

Development of Structural Damage Detection Method Working with Contaminated Vibration Data via Autoencoder and Gradient Boosting

... Final Vision Transformer representation z L consists of (N + 1) tokens of shape D. In high-level perception tasks such as image classification, the most common strategy is to use only the [cls] token output of the final ViT block (z L,0 ) as the representation of the entire image which serves as an input to the classifier [14,24,63]. The same approach is used in JEA pretraining, where the invariance objective is imposed on the [cls] representations (typically followed by a projector network [12,16]), while patch tokens are discarded [14,19]. An alternative strategy is to summarize the image representation as the average value of patch tokens, i.e. ...

Reference:

Beyond [cls]: Exploring the true potential of Masked Image Modeling representations

... For our object detection experiments, we used the Faster R-CNN implementation available from [46] based on the ResNet-50 backbone presented in [47]. To employ an anomaly detection task in the Faster R-CNN baseline, the architecture is supplemented by a one-class classification module based on the predicted object labels, as shown in Fig. 7. ...

Reference:

Toward phytoplankton parasite detection using autoencoders

... For both D25 and D50, 70% of the ima ges fr om the indistribution datasets were used for training, 20% for validation, and 10% for testing. An EfficientNet network, pr etr ained on Ima geNet [ 100 ], was trained using our method called MAPLE (MAhalanobis distance based uncertainty Prediction for reLiablE classification [ 101 ]) illustrated in Fig. 6 . To address high intraclass variances due to, for instance, different viewpoints from which the images were acquired, we use X-means clustering [ 102 ] to break down classes into m ultiple clusters, eac h of whic h contains ima ges clustering together in the feature space of representations learned by the network. ...

Reference:

“UDE DIATOMS in the Wild 2024”: a new image dataset of freshwater diatoms for training deep learning models

... The key idea of SSL is to design pretext tasks that leverage data itself or its augmentation as label information. Typical pretext tasks include reconstruction and comparison, which allow models to learn useful representations for downstream tasks [35,36]. A typical SSL workflow is to leverage vast unlabeled data for pre-training, followed by supervised fine-tuning [37]. ...

Reference:

Transferring self-supervised pre-trained models for SHM data anomaly detection with scarce labeled data

... To demonstrate this, we pretrain our model on K400 dataset, and pick checkpoints from various epochs. We then finetune these models on UCF101 dataset to evaluate on action recognition performance and report the corresponding accuracy results in Fig. 3. Given that MoCov2 [9] achieves 84.6% for UCF101 after 50 epochs with heavy training, our method outperforms by solely utilizing 5 epochs (85%). This observation implies that our proposed method can be a strong alternative with shorter pretraining requirements against large-scale video representation learning methods that rely on longer pretraining. ...

Reference:

Mitigating background bias in self-supervised video representation learning

... Similar to the slice-wise transition in medical images, adjacent frames in videos could carry useful continuity information. Feichtenhofer et al. [17] selected multiple video clips within a one-minute timespan as positive pairs and found improvements across various contrastive learning frameworks and downstream tasks. Since objects could have substantial difference across spatial and temporal dimensions, the spatial distance and time span may not be the best for positive pair selection. ...

Reference:

3D Segmentation of Necrotic Lung Lesions in CT Images Using Self-Supervised Contrastive Learning