No file available
To read the file of this research,
you can request a copy directly from the authors.
Sparsifying Transformer Models with Differentiable Representation Pooling
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
We propose a novel method to sparsify attention in the Transformer model by learning to select the most-informative token representations, thus leveraging the model's information bottleneck with twofold strength. A careful analysis shows that the contextualization of encoded representations in our model is significantly more effective than in the original Transformer. We achieve a notable reduction in memory usage due to an improved differentiable top-k operator, making the model suitable to process long documents, as shown on an example of a summarization task.
ResearchGate has not been able to resolve any citations for this publication.
We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities. After deriving its properties, we show how its Jacobian can be efficiently computed, enabling its use in a network trained with backpropagation. Then, we propose a new smooth and convex loss function which is the sparsemax analogue of the logistic loss. We reveal an unexpected connection between this new loss and the Huber classification loss. We obtain promising empirical results in multi-label classification problems and in attention-based neural networks for natural language inference. For the latter, we achieve a similar performance as the traditional softmax, but with a selective, more compact, attention focus.
Many machine learning tasks require sampling a subset of items from a collection based on a parameterized distribution. The Gumbel-softmax trick can be used to sample a single item, and allows for low-variance reparameterized gradients with respect to the parameters of the underlying distribution. However, stochastic optimization involving subset sampling is typically not reparameterizable. To overcome this limitation, we define a continuous relaxation of subset sampling that provides reparameterization gradients by generalizing the Gumbel-max trick. We use this approach to sample subsets of features in an instance-wise feature selection task for model interpretability, subsets of neighbors to implement a deep stochastic k-nearest neighbors model, and sub-sequences of neighbors to implement parametric t-SNE by directly comparing the identities of local neighbors. We improve performance in all these tasks by incorporating subset sampling in end-to-end training.
Automatic text summarization, the automated process of shortening a text while reserving the main ideas of the document(s), is a critical research area in natural language processing. The aim of this literature review is to survey the recent work on neural-based models in automatic text summarization. We examine in detail ten state-of-the-art neural-based summarizers: five abstractive models and five extractive models. In addition, we discuss the related techniques that can be applied to the summarization tasks and present promising paths for future research in neural-based summarization.
Beam search is a desirable choice of test-time decoding algorithm for neural sequence models because it potentially avoids search errors made by simpler greedy methods. However, typical cross entropy training procedures for these models do not directly consider the behaviour of the final decoding method. As a result, for cross-entropy trained models, beam decoding can sometimes yield reduced test performance when compared with greedy decoding. In order to train models that can more effectively make use of beam search, we propose a new training procedure that focuses on the final loss metric (e.g. Hamming loss) evaluated on the output of beam search. While well-defined, this "direct loss" objective is itself discontinuous and thus difficult to optimize. Hence, in our approach, we form a sub-differentiable surrogate objective by introducing a novel continuous approximation of the beam search decoding procedure. In experiments, we show that optimizing this new training objective yields substantially better results on two sequence tasks (Named Entity Recognition and CCG Supertagging) when compared with both cross entropy trained greedy decoding and cross entropy trained beam decoding baselines.
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.0 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
We present two novel and contrasting Recurrent Neural Network (RNN) based architectures for extractive summarization of documents. The Classifier based architecture sequentially accepts or rejects each sentence in the original document order for its membership in the final summary. The Selector architecture, on the other hand, is free to pick one sentence at a time in any arbitrary order to piece together the summary. Our models under both architectures jointly capture the notions of salience and redundancy of sentences. In addition, these models have the advantage of being very interpretable, since they allow visualization of their predictions broken up by abstract features such as information content, salience and redundancy. We show that our models reach or outperform state-of-the-art supervised models on two different corpora. We also recommend the conditions under which one architecture is superior to the other based on experimental evidence.
In this paper, we use belief-propagation techniques to develop fast algorithms for image inpainting. Unlike traditional gradient-based approaches, which may require many iterations to converge, our techniques achieve competitive results after only a few iterations. On the other hand, while belief-propagation techniques are often unable to deal with high-order models due to the explosion in the size of messages, we avoid this problem by approximating our high-order prior model using a Gaussian mixture. By using such an approximation, we are able to inpaint images quickly while at the same time retaining good visual results.
Generating long sequences with sparse transformers
- R Child
- S Gray
- A Radford
- I Sutskever
Child, R.; Gray, S.; Radford, A.; and Sutskever, I. 2019.
Generating long sequences with sparse transformers.
Differentiable ranking and sorting using optimal transport
- M Cuturi
- O Teboul
- J.-P Vert
Cuturi, M.; Teboul, O.; and Vert, J.-P. 2019. Differentiable
ranking and sorting using optimal transport. In Wallach, H.;
Larochelle, H.; Beygelzimer, A.; dÁlché-Buc, F.; Fox, E.;
and Garnett, R., eds., Advances in Neural Information Processing Systems 32. Curran Associates, Inc. 6861-6871.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- J Devlin
- M.-W Chang
- K Lee
- K Toutanova
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.
BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding. In NAACL-HLT.
Reformer: The Efficient Transformer
- N Kitaev
- L Kaiser
- A Levskaya
Kitaev, N.; Kaiser, L.; and Levskaya, A. 2020. Reformer:
The Efficient Transformer. ArXiv abs/2001.04451.
Neural nearest neighbors networks
- T Plötz
Plötz, T., and Roth, S. 2018. Neural nearest neighbors networks.
Improving language understanding by generative pre-training
- A Radford
Radford, A. 2018. Improving language understanding by
generative pre-training.
Linformer: Self-attention with linear complexity
- S Wang
- B Z Li
- M Khabsa
- H Fang
- H Ma
Wang, S.; Li, B. Z.; Khabsa, M.; Fang, H.; and Ma, H. 2020.
Linformer: Self-attention with linear complexity.
Big Bird: Transformers for Longer Sequences
- M Zaheer
- G Guruganesh
- A Dubey
- J Ainslie
- C Alberti
- S Ontanon
- P Pham
- A Ravula
- Q Wang
- L Yang
Zaheer, M.; Guruganesh, G.; Dubey, A.; Ainslie, J.; Alberti,
C.; Ontanon, S.; Pham, P.; Ravula, A.; Wang, Q.; Yang, L.;
and Ahmed, A. 2020. Big Bird: Transformers for Longer
Sequences.