2. (Large) language models

The first probabilistic language models were created many years ago, and were inspired by work to model human and computer languages using Markov chains. Models have been used in automatic language processing and speech recognition since the 1980s-1990s, making it possible to estimate the probability of a word's appearance based on previous words. Using a short history (a single previous word for unigram models, two and three for bigram and trigram models), these models were unable to take into account dependencies beyond a few words, and had difficulty handling rare and new words.

Large Language Models (LLMs) emerged in the late 2010s. They have benefited both from theoretical advances in machine learning, with deep neural networks