Token / Tokenization 是什么？
· One min read
- Online tools: https://platform.openai.com/tokenizer
- Key concepts: https://platform.openai.com/docs/concepts/tokens
tip
- In a sentence, the first token of each word typically starts with a space character.
- As a rough rule of thumb, 1 token is approximately 4 characters or 0.75 words for English text.