Large Language Models: a mathematician's notes to self
By Prof. T. R. Ramadas from CMI
Madhava Hall, Main Building, 3rd Floor, Math Department
Abstract
I will give an outline of how large language models (such as the ones
powering Gemini or ChatGPT) work. The aim will be to furnish enough detail that a
mathematician can have an informed understanding of the "attention'' mechanism and transformer architecture, the "size'' of a model, how a model is trained and
then used for inference (i.e., text generation in response to a query), how queries
are processed and how to better frame them, why hallucinations happen, why early
models struggled to quote sources and how the interaction differs from search.