RAG for Optimizing Generative AI - Response generation from LLMs enhanced by information retrieval

Without generative AI and assisted response generation, information retrieval can be carried out using search engines which, based on unstructured documents, respond to queries only with lists of documents, or by querying databases, which require complex and costly organization of source data. RAG (Retrieval Augmented Generation) is a third approach, integrating information retrieval with the major uses of conversational agents.

The years 2010-2020 have seen the emergence of neural machine learning and automatic language processing approaches, which have been rapidly exploited for information search and retrieval and, even more broadly, for knowledge engineering. Large-scale language models help not only to obtain semantic representations of documents, but also, in their generative form, to create fluid, comprehensible answers to complex questions, expressed in the form of "prompts".

Unfortunately, the cost of training large language models limits this operation to a handful of AI players who, in addition to having outstanding computing infrastructures, exploit data that extends far beyond the public Web alone. Even if refining large, pre-trained, freely distributed language models is a less costly operation than full training, and enables adaptation to specialist fields or private data, it is not enough to deploy secure search engines. Indeed, refinement must be kept to a minimum, otherwise the model will be unable to generate comprehensible text. A model, even when refined, remains dependent on its original training data. If you try to answer specific questions using a generative model alone, you run a high risk of obtaining obsolete, erroneous (hallucinations) or confused answers, simply because of the presence of contradictory information in the training data. What's more, when exploited without care, these masses of data reflect major societal biases, and indiscriminately mix opinions and false information.

While not a universal miracle solution, RAG can help reduce the risks outlined above. The main idea is to force the large language model to generate responses whose information comes from a data set pre-selected on the fly according to the query. The general knowledge of the model should only be used to ensure the linguistic competence needed to generate a comprehensible response.

Most of this article is devoted to the description of a RAG system, so as to enable the creation of functional software prototypes, based on a good understanding of theoretical principles, and knowledge of freely available reusable solutions. The solutions described in this article focus on the use of RAG for text documents, but the availability of foundation models and multimodal models ensures transfer to data including oral data,...