The neural network architecture behind virtually all modern AI language models.
The neural network architecture behind virtually all modern AI language models. Introduced in the 2017 paper 'Attention Is All You Need,' it replaced recurrent networks by processing entire sequences in parallel using attention mechanisms. Powers GPT, Claude, Gemini, LLaMA, and almost every LLM you've heard of.
A mechanism that lets neural networks focus on the most relevant parts of their input when producing output.
An attention mechanism where a sequence attends to itself — each element looks at all other elements to understand relationships.
An AI model that understands and generates human language.
A mathematical function applied to a neuron's output that introduces non-linearity into the network.
An optimization algorithm that combines the best parts of two other methods — AdaGrad and RMSProp.
Artificial General Intelligence.
Browse our complete glossary or subscribe to our newsletter for the latest AI news and insights.