Dictionary

Token

A unit of text or other input that a language model processes.

Last updated 2026-08-29 · Author

General AI term

A unit of text or other input that a language model processes.

Tokens are not always whole words: a word, part of a word, punctuation mark or other text unit may be represented by one or more tokens. Token counts influence context-window limits, cost and sometimes latency.

A Gaiish writer does not need to count every token to be clear. They do need to keep context relevant and know the model's limits when pasting long source material.

Receive the Gaiish key concepts and definitions guide for reference.