Demystifying Efficient Self-Attention
A practical overview of efficient attention mechanisms that tackle the quadratic scaling problem.
Transformers
Overcoming Input Length Constraints of Transformers
Using extractive summarization to train Transformers on long documents efficiently.