A technique that simulates larger batch sizes by accumulating gradients over multiple forward passes before updating weights.
A technique that simulates larger batch sizes by accumulating gradients over multiple forward passes before updating weights. Lets you train with effectively larger batches when GPU memory is limited. You get the stability benefits of large batches without needing the VRAM to hold all examples at once.
The number of training examples processed together before the model updates its weights.
The fundamental optimization algorithm used to train neural networks.
The process of teaching an AI model by exposing it to data and adjusting its parameters to minimize errors.
A mathematical function applied to a neuron's output that introduces non-linearity into the network.
An optimization algorithm that combines the best parts of two other methods — AdaGrad and RMSProp.
Artificial General Intelligence.
Browse our complete glossary or subscribe to our newsletter for the latest AI news and insights.