5. Evaluation of a RAG solution

Automated evaluation of the RAG is all the more delicate because of the complexity of the architecture adopted. Firstly, each module can be evaluated independently. Secondly, the response itself, generated by the LLM, can be evaluated along several informational (relevance, accuracy, completeness) and linguistic (readability, fluency, complexity) dimensions. This involves checking that the answers provided are present in the context documents, or that they can be deduced from them, and that they are sufficiently concise, limiting themselves to what is sought, and which is useful for understanding, and validating, the answer.

It is also interesting to estimate the system's usefulness from the user's point of view: time needed to find a satisfactory answer, time saved compared to using a conventional search engine. A good evaluation will necessarily be carried out...