Typewriters: language models
Transformer: neural networks for automatic language processing

Let's start with an elementary language processing task: spam filtering. Its probabilistic processing involves three steps:

1. collection of a representative set of e-mails, containing a set D _ok of acceptable e-mails and a set D _ko of undesirable e-mails ;

2. construction of a numerical representation for texts. A very simple representation transforms each e-mail d into a large binary vector h in ${\left\{\mathrm{0,}1\right\}}^{\left|V\right|}$...