---

Table of contents

• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• Contextual Position Encoding: Learning to Count What’s Important
• LieRE: Generalizing Rotary Position Encodings
• Gaussian Kernel-enhanced Rotary Position Embedding