Typewriters: Language Models
Transformers: Deep Neural Network Architectures for Natural Language Processing

Let’s start by considering a basic task in natural language processing (NLP): spam filtering. The probabilistic approach to this task involves three steps:

1. the collection of a representative sample of emails, containing a set ${\text{D}}_{\text{ok}}$ acceptable emails and a set ${\text{D}}_{\text{ko}}$ spam emails;

2. the creation of a digital representation for texts. A very simple representation converts each email...