Tokenization is the process of dividing a stream of text into tokens — the smallest units a tool will count or analyse, usually words but sometimes characters, subwords, or sentences. It is the invisible first step behind nearly every text metric: before you can produce a word count, measure keyword density, or build n-grams, you must decide where one token ends and the next begins.
The hard part is the edge cases. Consider the string "don't email me at bob@example.com — it's $5.50, isn't it?". A naive "split on whitespace" tokenizer treats bob@example.com and $5.50 as single tokens and keeps don't intact. A more aggressive tokenizer might split don't into do + n't, break the email at the @, or strip the dollar sign. Each choice changes the resulting counts. This is the main reason two tools report different word counts for the same paragraph: they disagree on hyphens, contractions, URLs, decimals, and emoji, not on arithmetic.
Different domains need different rules. Search engines lowercase and strip punctuation so that "Cat" and "cat," match. NLP pipelines may use subword tokenization (splitting "unbelievable" into "un", "believe", "able") so a model can handle words it has never seen. Sentence tokenization — needed for readability formulas that divide by sentence count — has its own trap: a period is not always a sentence boundary ("Dr.", "3.14", "e.g."). Understanding tokenization explains why word and character counts vary, and it underpins lemmatisation and stop-word filtering downstream. Try our word frequency counter to see tokenization in action on your own text.