Transformer: neural networks for automatic language processing

Linguistic technologies figure prominently among the applications of artificial intelligence (AI), and are now reaching the general public. They are essential for efficiently accessing textual information available on the Web or in large documentary databases; they enable new forms of interaction with the machine, by voice or through input or writing aids; they help us communicate with other humans, for example through machine translation systems; in a more underground way, these algorithms structure, organize, filter, select, transform and make it possible to manage the mounds of text and audio recordings that continually circulate on the web or on social networks.

This transition has accelerated as these technologies have become progressively more powerful for ever wider and more varied uses. This progress is the result of a combination of factors: on the one hand, the development of increasingly sophisticated machine learning algorithms, capable of taking advantage of improved computing hardware; on the other, the possibility of accessing very large masses of textual data, annotated or unannotated, to carry out this learning. Among algorithms, neural algorithms, and in particular the Transformer architecture, are at the forefront. This architecture has in fact become central to three types of processing that previously required dedicated architectures: firstly, text mining and information retrieval algorithms, which benefit from the richness of the internal representations computed by this model; secondly, linguistic analysis algorithms, which take advantage of Transformers' ability to take into account very long-distance dependencies; and finally, text generation algorithms, which use these models primarily for their predictive capacity. Add to this the fact that this same architecture also lends itself to the processing of oral and even multimodal data, and that it enables efficient calculations on a very large scale, and it's easy to see why this model has become the Swiss Army Knife of the linguistic engineer.