RAG: How to improve LLM relevance on your private data
In this article, we explain how RAGs make LLMs more reliable, efficient, trustworthy, and flexible by diving into the various components of their architecture.
Matthieu BOUSSARD
·
Nov 30, 2023
Introduction
Large Language Models (LLMs) are formidable tools, capable of providing convincing answers to any question asked. Unfortunately, this ability to persuade is a double-edged sword. Under the guise of seemingly logical reasoning, the stated facts can be completely false. This can be caused by the absence of documents in the training corpus, whether due to their novelty, rarity, or confidential nature.
Furthermore, in certain use cases where the LLM user is held accountable, it is necessary to be able to provide the sources used to answer the question. This is the case for contractual clauses, for example, where the answer is less important than the way it was reached.
These use cases are grouped under the term "knowledge-intensive applications." In this context, the primary goal is to access knowledge, as opposed to applications for translation or paraphrasing, for example. We therefore want to use the comprehension and synthesis power of LLMs while ensuring the use of a controlled knowledge base, all while citing sources. This best-of-both-worlds scenario exists, and it is based on RAGs: Retrieval-Augmented Generation.
A RAG is an LLM-based system designed to retrieve information from a user-controlled corpus and then synthesize an answer based on selected elements.
Main components
A RAG is an architecture involving several components. Before presenting the complete architecture, we will zoom in on the main parts.
Embedding
The first concept of a RAG is embedding. Computers cannot directly manipulate words or sentences. They must be transformed into numerical values that can be calculated. It is important that this transformation preserves semantic proximity, meaning that two concepts that are semantically close are also numerically close.
Fortunately, there are pre-trained models adapted for this task, such as BERT-derived sentence embeddings. This operation can be computationally intensive, even if the models are smaller than LLMs. Recently, models specifically designed to improve inference times have even been proposed.
Document Chunking
The corpus containing the knowledge to be used by the RAG cannot be used directly. It must be broken down into small pieces, or "chunks" (potentially with an overlap). As we have seen, these chunks cannot be used directly and must be transformed via embedding. Furthermore, it is important to keep metadata around the chunks, such as the source file from which it was taken, the chapter within the document, the last update date, etc.
The size of the corpus can be large, and storing and retrieving these chunks poses new problems; this is where vector databases come in.
Specialized information system: Vector databases
To answer a given question, a RAG calculates an embedding of the question and searches for relevant chunks. Since embeddings preserve the notion of semantic distance, finding relevant documents is essentially finding documents that are close in terms of distance within the embedding space. We can therefore formalize the problem of finding relevant documents as "find the k closest chunks in the embedding space." This operation must be inexpensive, even with a large corpus. This is where vector databases come in.
To solve these problems, it is no longer possible to use relational databases like MySQL or NoSQL databases like Redis. The way information is stored in them is not adapted to the types of queries made in a RAG.
Fortunately, there are databases designed specifically for certain tasks, such as TimescaleDB for time series or PostGIS for geographic data. Here, it is vector databases that solve our problem. They store embeddings in an optimized way so that it is possible to find the L vectors closest to a given vector. Key players here include ChromaDB, Qdrant, and PGVector.
So, at the end of this step, the RAG has retrieved the k most relevant chunks for the question asked from the database. They are then transferred to the LLM to provide the final answer.
LLM
The user's question and the chunks are assembled to form a prompt, which is then provided as input to an LLM, such as LLaMa 2, Falcon, etc., in the standard way. It should be noted that the task of creating an answer from elements provided to the LLM is simpler than having to generate everything from scratch. Thus, even with a "small" LLM (7b), very relevant results are already achieved.
RAG architecture
With the elements mentioned above, we can present the complete architecture of a RAG system. First, the VectorDB is populated with embeddings and metadata from chunks derived from the document base. A query arrives, and its embedding is calculated. The K most relevant chunks are extracted from the VectorDB. The question and the chunks are combined to create a prompt. This prompt is sent to an LLM, which provides the answer.
This answer can also be augmented with the sources used, thanks to the metadata associated with the chunks.
Technically, this general architecture is standard, and we have identified the various building blocks required for its construction. In practice, Python libraries such as Langchain or LlamaIndex allow for the efficient selection and combination of the LLM, embedding model, and Vector DB, along with their respective parameters.
Conclusion
The RAG architecture therefore offers numerous advantages over using an LLM alone. We can list the following points:
Reliability: By using a knowledge base controlled by the user, the probability of hallucinations is significantly reduced.
Trust: The sources used for generation are provided. If the user bears significant responsibility, they can refer directly to the sources.
Efficiency: The LLM used to generate the answer can be much smaller than a GPT-4 while still delivering results. This architecture avoids the need to fine-tune a model on your own corpus. Furthermore, it is possible to have a relevant RAG system even with only a few documents.
Flexibility: Modifying the knowledge base is simple, involving only the addition of documents to the VectorDB.
This architecture is already highly effective with generic components. It is possible to further improve performance through fine-tuning. You can re-train the LLM on a question-document-answer database, which could, for example, be generated by a larger model (like GPT-4), thereby improving the quality of the generated response.
Furthermore, this architecture can be modified to incorporate the ability to use tools. This is referred to as "agents." In this framework, before answering a question directly, an LLM can choose to call upon a tool, such as an online search, an API, a calculator, etc. Thus, the LLM can itself choose the query to perform in the Vector DB. In this context, it can combine RAG with other tools, such as online search.
While RAGs offer many advantages, they remain machine learning systems. Their performance must be carefully measured, and their usage must be monitored continuously, particularly due to the dynamic nature of the knowledge base. The metrics to observe are the subject of in-depth study, but they include performance metrics regarding result quality as well as security metrics, such as those related to toxicity.
Want to learn more about how to deploy a domain-specific, LLM and test a RAG system with your own data? Book an appointment here.
Subscribe to our Newsletter
Stay up to date with the latest news, articles, and updates. Subscribe now!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.