The component that converts raw text into tokens that a language model can process.
The component that converts raw text into tokens that a language model can process. Different models use different tokenizers — BPE (Byte Pair Encoding), SentencePiece, and tiktoken are common. The choice of tokenizer affects how efficiently the model handles different languages and types of text.
The basic unit of text that language models work with.
Byte Pair Encoding.
A mathematical function applied to a neuron's output that introduces non-linearity into the network.
An optimization algorithm that combines the best parts of two other methods — AdaGrad and RMSProp.
Artificial General Intelligence.
The research field focused on making sure AI systems do what humans actually want them to do.
Browse our complete glossary or subscribe to our newsletter for the latest AI news and insights.