Writing about technology, systems, and whatever I’m trying to understand.
Understanding LLMs From Scratch: Tokenizers
You wake up, Rip Van Winkle-like, from a deep sleep. It’s 2026 and LLM’s are taking over the world. Time to learn something about how these work, you think (and write up your notes). Best to start at the very beginning of the LLM processing pipeline - what is a tokenizer and how does it work? What are Tokenizers? At its most basic, a tokenizer is a way to take a string of text and turn it into a list of numbers that a computer can understand. ...