No file available
To read the file of this research,
you can request a copy directly from the authors.
Exploring Plain Vision Transformer Backbones for Object Detection
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
We explore the plain, non-hierarchical Vision Transformer (ViT) as a backbone network for object detection. This design enables the original ViT architecture to be fine-tuned for object detection without needing to redesign a hierarchical backbone for pre-training. With minimal adaptations for fine-tuning, our plain-backbone detector can achieve competitive results. Surprisingly, we observe: (i) it is sufficient to build a simple feature pyramid from a single-scale feature map (without the common FPN design) and (ii) it is sufficient to use window attention (without shifting) aided with very few cross-window propagation blocks. With plain ViT backbones pre-trained as Masked Autoencoders (MAE), our detector, named ViTDet, can compete with the previous leading methods that were all based on hierarchical backbones, reaching up to 61.3 box AP on the COCO dataset using only ImageNet-1K pre-training. We hope our study will draw attention to research on plain-backbone detectors. Code will be made available.
Vision transformer has demonstrated great potential in abundant vision tasks. However, it also inevitably suffers from poor generalization capability when the distribution shift occurs in testing (
i.e
., out-of-distribution data). To mitigate this issue, we propose a novel method, Semantic-aware Message Broadcasting (SAMB), which enables more informative and flexible feature alignment for unsupervised domain adaptation (UDA). Particularly, we study the attention module in the vision transformer and notice that the alignment space using one global class token lacks enough flexibility, where it interacts information with all image tokens in the same manner but ignores the rich semantics of different regions. In this paper, we aim to improve the richness of the alignment features by enabling semantic-aware adaptive message broadcasting. Particularly, we introduce a group of learned group tokens as nodes to aggregate the global information from all image tokens, but encourage different group tokens to adaptively focus on the message broadcasting to different semantic regions. In this way, our message broadcasting encourages the group tokens to learn more informative and diverse information for effective domain alignment. Moreover, we systematically study the effects of adversarial-based feature alignment (ADA) and pseudo-label based self-training (PST) on UDA. We find that one simple two-stage training strategy with the cooperation of ADA and PST can further improve the adaptation capability of the vision transformer. Extensive experiments on DomainNet, OfficeHome, and VisDA-2017 demonstrate the effectiveness of our methods for UDA.
To transfer the representation capacity of large pre-trained models to lightweight models, knowledge distillation has been widely explored. However, conventional single-stage distillation methods are prone to getting stuck in the transfer of task-specific knowledge, making it difficult to retain task-agnostic knowledge which is crucial for model generalization. In this study, we propose generic-to-specific distillation (G2SD), to boost lightweight models under the assistance of large models pre-trained by masked image modeling. In generic distillation, the decoder of a small model is encouraged to align feature predictions with that of a large model, so that task-agnostic knowledge can be transferred. In specific distillation, predictions of the small model are encouraged to be consistent with those of the large model, to guarantee task performance. G2SD is also applicable for heterogeneous settings(
i.e
., distilling from ViT to CNN). With G2SD, the ViT-Small model respectively achieves 98.9%, 98.4%, 99.3% and 98.9% accuracies when compared with its teachers (ViT-Base) for image classification, object detection, semantic segmentation and video recognition tasks. The lightweight ResNet models are improved to a new height on image classification task. The code is available at github.com/pengzhiliang/G2SD.
Space flight, which we anticipate as SpaceX does, has the potential to address the challenge of human life in the near future. The first stage towards interstellar travel is to establish a satellite network, which could deliver worldwide internet access to every location on the planet. However, satellites are not as smart as everyone thinks! Because the satellite industry faces severe problems including: Cost, Speed, Vacuum environment, Energy, Computing power, Hardware stability, Software lightweight, Model optimization, Data set related etc., these are all the obstacles to upgrade the local calculation to orbital computation. And because this research category involves the scientific research secrets of various countries, many related materials are kept confidential and difficult to access publicly. Although AI researchers have enthusiasm with the application of deep learning in the field of remote sensing data, and the processing speed and scale of remote sensing image information have been improved. However, the above research work belongs to ground-oriented processing, and the orbital application is still blank! The visual models currently popular used on the ground are evaluated , and their respective computing power requirements are analyzed for possibilities on orbit. Experiments on genuine microsatellites are hampered by the high expense of deployment and unknown risks associated with space technology. Our Tiansuan constellation, fortunately, can facilitate experimentation on real satellite networks. We introduce our brand-new integrated hybrid Orbital Deep Neural Network Model WolongPro 1.0 system based on edge computing and 2.0 research concept scheme about Spintronics, which can have human attentions on disaster risk reduction. The attention of the orbital satellite can be focused on reporting early warning of perils, instead of transmitting high resolution pictures and then analyzed them by computer or manual on the ground, so as to improve the recognition accuracy and predict disasters in advance, such as tornadoes and floods etc. Our research: Intelligent software-defined microsatellite with orbital DNN attention for SDGs base on edge computing, can be applied on early warning of perils, facilitate the Sustainable Development Goals of United Nations Office for Disaster Risk Reduction (UNDRR).
We propose CornerNet, a new approach to object detection where we detect an object bounding box as a pair of keypoints, the top-left corner and the bottom-right corner, using a single convolution neural network. By detecting objects as paired keypoints, we eliminate the need for designing a set of anchor boxes commonly used in prior single-stage detectors. In addition to our novel formulation, we introduce corner pooling, a new type of pooling layer that helps the network better localize corners. Experiments show that CornerNet achieves a 42.2% AP on MS COCO, outperforming all existing one-stage detectors.
We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. SSD is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stages and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into systems that require a detection component. Experimental results on the PASCAL VOC, COCO, and ILSVRC datasets confirm that SSD has competitive accuracy to methods that utilize an additional object proposal step and is much faster, while providing a unified framework for both training and inference. For input, SSD achieves 74.3 % mAP on VOC2007 test at 59 FPS on a Nvidia Titan X and for input, SSD achieves 76.9 % mAP, outperforming a comparable state of the art Faster R-CNN model. Compared to other single stage methods, SSD has much better accuracy even with a smaller input image size. Code is available at https:// github. com/ weiliu89/ caffe/ tree/ ssd.
The explosion of image data on the Internet has the potential to foster more sophisticated and robust models and algorithms to index, retrieve, organize and interact with images and multimedia data. But exactly how such data can be harnessed and organized remains a critical problem. We introduce here a new database called ldquoImageNetrdquo, a large-scale ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500-1000 clean and full resolution images. This will result in tens of millions of annotated images organized by the semantic hierarchy of WordNet. This paper offers a detailed analysis of ImageNet in its current state: 12 subtrees with 5247 synsets and 3.2 million images in total. We show that ImageNet is much larger in scale and diversity and much more accurate than the current image datasets. Constructing such a large-scale database is a challenging task. We describe the data collection scheme with Amazon Mechanical Turk. Lastly, we illustrate the usefulness of ImageNet through three simple applications in object recognition, image classification and automatic object clustering. We hope that the scale, accuracy, diversity and hierarchical structure of ImageNet can offer unparalleled opportunities to researchers in the computer vision community and beyond.
Current object detectors are limited in vocabulary size due to the small scale of detection datasets. Image classifiers, on the other hand, reason about much larger vocabularies, as their datasets are larger and easier to collect. We propose Detic, which simply trains the classifiers of a detector on image classification data and thus expands the vocabulary of detectors to tens of thousands of concepts. Unlike prior work, Detic does not need complex assignment schemes to assign image labels to boxes based on model predictions, making it much easier to implement and compatible with a range of detection architectures and backbones. Our results show that Detic yields excellent detectors even for classes without box annotations. It outperforms prior work on both open-vocabulary and long-tail detection benchmarks. Detic provides a gain of 2.4 mAP for all classes and 8.3 mAP for novel classes on the open-vocabulary LVIS benchmark. On the standard LVIS benchmark, Detic obtains 41.7 mAP when evaluated on all classes, or only rare classes, hence closing the gap in performance for object categories with few samples. For the first time, we train a detector with all the twenty-one-thousand classes of the ImageNet dataset and show that it generalizes to new datasets without finetuning. Code is available at https://github.com/facebookresearch/Detic.
This work presents a simple vision transformer design as a strong baseline for object localization and instance segmentation tasks. Transformers recently demonstrate competitive performance in image classification. To adopt ViT to object detection and dense prediction tasks, many works inherit the multistage design from convolutional networks and highly customized ViT architectures. Behind this design, the goal is to pursue a better trade-off between computational cost and effective aggregation of multiscale global contexts. However, existing works adopt the multistage architectural design as a black-box solution without a clear understanding of its true benefits. In this paper, we comprehensively study three architecture design choices on ViT – spatial reduction, doubled channels, and multiscale features – and demonstrate that a vanilla ViT architecture can fulfill this goal without handcrafting multiscale features, maintaining the original ViT design philosophy. We further complete a scaling rule to optimize our model’s trade-off on accuracy and computation cost / model size. By leveraging a constant feature resolution and hidden size throughout the encoder blocks, we propose a simple and compact ViT architecture called Universal Vision Transformer (UViT) that achieves strong performance on COCO object detection and instance segmentation benchmark. Our code is available at https://github.com/tensorflow/models/tree/master/official/projects/uvit.
Modern top-performing object detectors depend heavily on backbone networks, whose advances bring consistent performance gains through exploring more effective network structures. In this paper, we propose a novel and flexible backbone framework, namely CBNet, to construct high-performance detectors using existing open-source pre-trained backbones under the pre-training fine-tuning paradigm. In particular, CBNet architecture groups multiple identical backbones, which are connected through composite connections. Specifically, it integrates the high- and low-level features of multiple identical backbone networks and gradually expands the receptive field to more effectively perform object detection. We also propose a better training strategy with auxiliary supervision for CBNet-based detectors. CBNet has strong generalization capabilities for different backbones and head designs of the detector architecture. Without additional pre-training of the composite backbone, CBNet can be adapted to various backbones (i.e., CNN-based vs. Transformer-based) and head designs of most mainstream detectors (i.e., one-stage vs. two-stage, anchor-based vs. anchor-free-based). Experiments provide strong evidence that, compared with simply increasing the depth and width of the network, CBNet introduces a more efficient, effective, and resource-friendly way to build high-performance backbone networks. Particularly, our CB-Swin-L achieves 59.4% box AP and 51.6% mask AP on COCO
test-dev
under the single-model and single-scale testing protocol, which are significantly better than the state-of-the-art results (i.e., 57.7% box AP and 50.2% mask AP) achieved by Swin-L, while reducing the training time by
. With multi-scale testing, we push the current best single model result to a new record of 60.1% box AP and 52.3% mask AP without using extra training data. Code is available at
https://github.com/VDIGPKU/CBNetV2
.
We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. It is a simple residual network that alternates (i) a linear layer in which image patches interact, independently and identically across channels, and (ii) a two-layer feed-forward network in which channels interact independently per patch. When trained with a modern training strategy using heavy data-augmentation and optionally distillation, it attains surprisingly good accuracy/complexity trade-offs on ImageNet. We also train ResMLP models in a self-supervised setup, to further remove priors from employing a labelled dataset. Finally, by adapting our model to machine translation we achieve surprisingly good results. We share pre-trained models and our code based on the Timm library.
We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster R-CNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and pretrained models are available at https://github.com/facebookresearch/detr.
In object detection, the intersection over union (IoU) threshold is frequently used to define positives/negatives. The threshold used to train a detector defines its
quality
. While the commonly used threshold of 0.5 leads to noisy (low-quality) detections, detection performance frequently degrades for larger thresholds. This paradox of high-quality detection has two causes: 1) overfitting, due to vanishing positive samples for large thresholds, and 2) inference-time quality mismatch between detector and test hypotheses. A multi-stage object detection architecture, the Cascade R-CNN, composed of a sequence of detectors trained with increasing IoU thresholds, is proposed to address these problems. The detectors are trained sequentially, using the output of a detector as training set for the next. This resampling progressively improves hypotheses quality, guaranteeing a positive training set of equivalent size for all detectors and minimizing overfitting. The same cascade is applied at inference, to eliminate quality mismatches between hypotheses and detectors. An implementation of the Cascade R-CNN without bells or whistles achieves state-of-the-art performance on the COCO dataset, and significantly improves high-quality detection on generic and specific object datasets, including VOC, KITTI, CityPerson, and WiderFace. Finally, the Cascade R-CNN is generalized to instance segmentation, with nontrivial improvements over the Mask R-CNN.
We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 dif- ferent classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implemen- tation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called dropout that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.0 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
Deeper neural networks are more difficult to train. We present a residual
learning framework to ease the training of networks that are substantially
deeper than those used previously. We explicitly reformulate the layers as
learning residual functions with reference to the layer inputs, instead of
learning unreferenced functions. We provide comprehensive empirical evidence
showing that these residual networks are easier to optimize, and can gain
accuracy from considerably increased depth. On the ImageNet dataset we evaluate
residual nets with a depth of up to 152 layers---8x deeper than VGG nets but
still having lower complexity. An ensemble of these residual nets achieves
3.57% error on the ImageNet test set. This result won the 1st place on the
ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100
and 1000 layers.
The depth of representations is of central importance for many visual
recognition tasks. Solely due to our extremely deep representations, we obtain
a 28% relative improvement on the COCO object detection dataset. Deep residual
nets are foundations of our submissions to ILSVRC & COCO 2015 competitions,
where we also won the 1st places on the tasks of ImageNet detection, ImageNet
localization, COCO detection, and COCO segmentation.
We present YOLO, a unified pipeline for object detection. Prior work on
object detection repurposes classifiers to perform detection. Instead, we frame
object detection as a regression problem to spatially separated bounding boxes
and associated class probabilities. A single neural network predicts bounding
boxes and class probabilities directly from full images in one evaluation.
Since the whole detection pipeline is a single network, it can be optimized
end-to-end directly on detection performance. Our unified architecture is also
extremely fast; YOLO processes images in real-time at 45 frames per second,
hundreds to thousands of times faster than existing detection systems. Our
system uses global image context to detect and localize objects, making it less
prone to background errors than top detection systems like R-CNN. By itself,
YOLO detects objects at unprecedented speeds with moderate accuracy. When
combined with state-of-the-art detectors, YOLO boosts performance by 2-3%
points mAP.
In this work we investigate the effect of the convolutional network depth on
its accuracy in the large-scale image recognition setting. Our main
contribution is a thorough evaluation of networks of increasing depth, which
shows that a significant improvement on the prior-art configurations can be
achieved by pushing the depth to 16-19 weight layers. These findings were the
basis of our ImageNet Challenge 2014 submission, where our team secured the
first and the second places in the localisation and classification tracks
respectively.
Existing deep convolutional neural networks (CNNs) require a fixed-size (e.g. 224×224) input image. This requirement is “artificial” and may hurt the recognition accuracy for the images or sub-images of an arbitrary size/scale. In this work, we equip the networks with a more principled pooling strategy, “spatial pyramid pooling”, to eliminate the above requirement. The new network structure, called SPP-net, can generate a fixed-length representation regardless of image size/scale. By removing the fixed-size limitation, we can improve all CNN-based image classification methods in general. Our SPP-net achieves state-of-the-art accuracy on the datasets of ImageNet 2012, Pascal VOC 2007, and Caltech101.
The power of SPP-net is more significant in object detection. Using SPP-net, we compute the feature maps from the entire image only once, and then pool features in arbitrary regions (sub-images) to generate fixed-length representations for training the detectors. This method avoids repeatedly computing the convolutional features. In processing test images, our method computes convolutional features 30-170× faster than the recent leading method R-CNN (and 24-64× faster overall), while achieving better or comparable accuracy on Pascal VOC 2007.
We present a new dataset with the goal of advancing the state-of-the-art in
object recognition by placing the question of object recognition in the context
of the broader question of scene understanding. This is achieved by gathering
images of complex everyday scenes containing common objects in their natural
context. Objects are labeled using per-instance segmentations to aid in
understanding an object's precise 2D location. Our dataset contains photos of
91 objects types that would be easily recognizable by a 4 year old along with
per-instance segmentation masks. With a total of 2.5 million labeled instances
in 328k images, the creation of our dataset drew upon extensive crowd worker
involvement via novel user interfaces for category detection, instance spotting
and instance segmentation. We present a detailed statistical analysis of the
dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide
baseline performance analysis for bounding box and segmentation detection
results using a Deformable Parts Model.
Can a large convolutional neural network trained for whole-image
classification on ImageNet be coaxed into detecting objects in PASCAL? We show
that the answer is yes, and that the resulting system is simple, scalable, and
boosts mean average precision, relative to the venerable deformable part model,
by more than 40% (achieving a final mAP of 48% on VOC 2007). Our framework
combines powerful computer vision techniques for generating bottom-up region
proposals with recent advances in learning high-capacity convolutional neural
networks. We call the resulting system R-CNN: Regions with CNN features. The
same framework is also competitive with state-of-the-art semantic segmentation
methods, demonstrating its flexibility. Beyond these results, we execute a
battery of experiments that provide insight into what the network learns to
represent, revealing a rich hierarchy of discriminative and often semantically
meaningful features.
The ability of learning networks to generalize can be greatly enhanced by providing constraints from the task domain. This paper demonstrates how such constraints can be integrated into a backpropagation network through the architecture of the network. This approach has been successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service. A single network learns the entire recognition operation, going from the normalized image of the character to the final classification.
- Jimmy Lei Ba
- Jamie Ryan Kiros
- Geoffrey E Hinton
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization.
arXiv:1607.06450, 2016.
- Hangbo Bao
- Li Dong
- Furu Wei
Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image Transformers. arXiv:2106.08254, 2021.
Training deep nets with sublinear memory cost
- Tianqi Chen
- Bing Xu
- Chiyuan Zhang
- Carlos Guestrin
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets
with sublinear memory cost. arXiv:1604.06174, 2016.
ELEC-TRA: Pre-training text encoders as discriminators rather than generators
- Kevin Clark
- Minh-Thang Luong
- V Quoc
- Christopher D Le
- Manning
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. ELEC-TRA: Pre-training text encoders as discriminators rather than generators. In
ICLR, 2020.
BERT: Pre-training of deep bidirectional Transformers for language understanding
- Jacob Devlin
- Ming-Wei Chang
- Kenton Lee
- Kristina Toutanova
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of deep bidirectional Transformers for language understanding. In
NAACL, 2019.
An image is worth 16x16 words: Transformers for image recognition at scale
- Alexey Dosovitskiy
- Lucas Beyer
- Alexander Kolesnikov
- Dirk Weissenborn
- Xiaohua Zhai
- Thomas Unterthiner
- Mostafa Dehghani
- Matthias Minderer
- Georg Heigold
- Sylvain Gelly
- Jakob Uszkoreit
- Neil Houlsby
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth
16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
XCiT: Cross-covariance image Transformers
- Alaaeldin El-Nouby
- Hugo Touvron
- Mathilde Caron
- Piotr Bojanowski
- Matthijs Douze
- Armand Joulin
- Ivan Laptev
- Natalia Neverova
- Gabriel Synnaeve
- Jakob Verbeek
- Herve Jegou
Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs
Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob
Verbeek, and Herve Jegou. XCiT: Cross-covariance image Transformers. In
NeurIPS, 2021.
LVIS challenge track technical report 1st place solution: Distribution balanced and boundary refinement for large vocabulary instance segmentation
- Weifu Fu
- Congchong Nie
- Ting Sun
- Jun Liu
- Tianliang Zhang
- Yong Liu
WeiFu Fu, CongChong Nie, Ting Sun, Jun Liu, TianLiang Zhang, and Yong
Liu. LVIS challenge track technical report 1st place solution: Distribution
balanced and boundary refinement for large vocabulary instance segmentation.
arXiv:2111.02668, 2021.
- Priya Goyal
- Piotr Dollár
- Ross Girshick
- Pieter Noordhuis
- Lukasz Wesolowski
- Aapo Kyrola
- Andrew Tulloch
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski,
Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large
minibatch SGD: Training ImageNet in 1 hour. arXiv:1706.02677, 2017.
- Kaiming He
- Georgia Gkioxari
- Piotr Dollár
- Ross Girshick
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In
ICCV, 2017.
- Dan Hendrycks
- Kevin Gimpel
Dan Hendrycks and Kevin Gimpel.
Gaussian error linear units (GeLUs).
arXiv:1606.08415, 2016.
MViTv2: Improved multiscale Vision Transformers for classification and detection
- Yanghao Li
- Chao-Yuan
- Haoqi Wu
- Karttikeya Fan
- Bo Mangalam
- Jitendra Xiong
- Christoph Malik
- Feichtenhofer
Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. MViTv2: Improved multiscale Vision
Transformers for classification and detection. arXiv:2112.01526, 2021.
Benchmarking detection transfer learning with Vision Transformers
- Yanghao Li
- Saining Xie
- Xinlei Chen
- Piotr Dollar
- Kaiming He
- Ross Girshick
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross
Girshick. Benchmarking detection transfer learning with Vision Transformers.
arXiv:2111.11429, 2021.
Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection
- Tsung-Yi Lin
- Piotr Dollár
- Ross Girshick
Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and
Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
- Ze Liu
- Han Hu
- Yutong Lin
- Zhuliang Yao
- Zhenda Xie
- Yixuan Wei
- Jia Ning
- Yue Cao
- Zheng Zhang
- Li Dong
- Furu Wei
- Baining Guo
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning,
Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin Transformer
V2: Scaling up capacity and resolution. arXiv:2111.09883, 2021.
Decoupled weight decay regularization
- Ilya Loshchilov
- Frank Hutter
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In
ICLR, 2019.
Learning transferable visual models from natural language supervision
- Alec Radford
- Jong Wook Kim
- Chris Hallacy
- Aditya Ramesh
- Gabriel Goh
- Sandhini Agarwal
- Girish Sastry
- Amanda Askell
- Pamela Mishkin
- Jack Clark
- Gretchen Krueger
- Ilya Sutskever
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen
Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. 2021.