Boost LLM Accuracy with Retrieval Augmented Generation (RAG) and Reranking

Imagine a Large Language Model (LLM) where the answers you get aren’t just relevant; they’re finely curated, prioritized, and refined to match your exact needs. While LLMs have revolutionized the realm of AI, they are not without limitations. Issues such as hallucinations and data obsolescence can compromise the accuracy and relevance of their outputs. This is where Retrieval-Augmented Generation (RAG) and reranking come into play, offering a way to enhance LLMs by integrating them with dynamic, up-to-date information retrieval processes. Curious about the steps of this recipe? Keep reading.

Why RAG For Enhancing LLMs?

LLMs have revolutionized the realm of AI and pushed the boundaries of what we can achieve by using AI. They have become the go-to tool for anyone seeking versatile NLP solutions in virtually any domain by modeling a range of natural language understanding and generation tasks, as shown below.

Despite their capabilities, LLMs also have limitations in certain scenarios. Depending on the specific use case and the knowledge they have learned from the vast dataset they are trained on, LLMs sometimes fail to generate coherent, relevant, or contextually appropriate text. Sometimes, in the absence of relevant truthful data to create a response to user queries, they may even generate incorrect or nonsensical information as if it were true. This phenomenon is known as hallucination.

Consider, for instance, the question: “What are the common flu symptoms?”

A standard LLM might generate a response based on general knowledge, listing common symptoms like fever, cough, and body aches.

However, unless it had been trained on very domain-specific flu virus data, the LLM might not consider variations in symptom severity or distinguish between flu strains, thereby providing rather generic and even somewhat “automated” replies to different users regardless of their circumstances or needs.

What’s more, if, for instance, the model had been trained on clinical flu data collected up to December 2023, and a new flu strain appears and quickly spreads across the population in January 2024, a standalone LLM will be unable to provide accurate responses due to the lack of up-to-date knowledge about the problem domain.

This “data obsolescence” problem is known as knowledge cut-off.

In some cases, the solution to the above problem could be frequently retraining and fine-tuning the LLM on fresh, up-to-date information. But is this necessarily the best course of action?

LLMs (Large Language Models) are known to be challenging and computationally expensive to train. They require millions to billions of text data instances and often thousands of domain-specific texts for fine-tuning (see diagram below).

Most LLMs, even domain-specific ones, are typically fine-tuned to work across wide-scope domains, such as health. Therefore, introducing new fine-grained data to accommodate all possible contextual nuances across the domain may not be the most efficient solution.

This is where RAG (Retrieval Augmented Generation) comes to help!

RAG is an information retrieval process whereby the outputs produced by an LLM are optimized. LLMs rely on the knowledge gained from the data they have been generated upon to generate responses.

Meanwhile, RAG points to an external knowledge base.

By combining both solutions, RAG can be used to enhance the quality, user relevance, coherence, and truthfulness of the “raw” output generated by the LLM by retrieving knowledge from the knowledge base mentioned earlier.

As a result, the necessity of continuously retraining the LLM to adapt it to ever-emerging new contexts and situations is largely removed.

The general workflow of an RAG system is described in a simplified way below:

Query: The user query or prompt is collected by the LLM.

Retrieve: During runtime, the LLM accesses a retrieval component that indexes a corpus of text documents and identifies the top-K matching components with the user query.

Generate: The LLM combines two pieces of information -the original user input and the retrieved documents- to generate the output response, employing the maximum likelihood principle as usual to generate every next token in the output sequence.

Back to the flu example, we can see that with RAG, the model could retrieve up-to-date, relevant information from medical databases or recent articles, enabling it to generate a more nuanced and accurate answer to the patient or practitioner’s inquiry.

It could incorporate insights about current flu strains, regional variations in symptoms, or emerging patterns, thereby providing a more relevant and coherent response to the user’s query.

What is Reranking?

Reranking is an information retrieval process where an initial set of retrieved results is reordered to improve relevance to the user query, needs, and context, thereby enhancing the overall output quality. Here’s how it works:

First, an initial document retrieval pass is conducted, in which the retriever fetches K relevant documents. Commonplace approaches like TF-IDF, vector space model, and so on, are considered in this stage.

Second, a more sophisticated or domain-specific mechanism than the initial retriever, called a ranker, comes into play. The ranker re-evaluates retrieved results based on additional criteria, e.g., learned user preferences, context, and needs, or more complex algorithms. This secondary sorting aims to enhance the relevance of the results presented to the particular user.

The following diagram illustrates the reranking process:

Let’s clarify an important point here: reranking reorders retrieved documents based on various criteria, such as user preferences. However, reranking is different from recommender engines, like the ones that suggest relevant products to buy on e-commerce sites.

Reranking is used in search use cases where a user provides an input query in real time.

In contrast, recommender engines proactively create personalized suggestions for users based on their interactions and preferences over time.

Let’s revisit the flu example.

Consider a situation where a healthcare professional searches for “best treatments for flu symptoms.” An initial retrieval system might return a list of documents, including general flu information, treatment guidelines, and research articles.

But a reranking model, possibly using additional patient-specific data and contextual information, can then reorder these documents to prioritize the most relevant and recent treatment protocols, patient care tips, and peer-reviewed research studies directly addressing flu symptoms and their management, thereby prioritizing results that go “straight to the point.”

To sum up, reranking reorganizes a list of retrieved documents based on additional relevance criteria to display the most relevant to the specific user first.

Why is Reranking Useful For LLMs With RAG?

Reranking is particularly useful in Large Language Models (LLMs) equipped with Retrieval-Augmented Generation (RAG). RAG combines LLMs with external document retrieval to provide more informed and accurate responses.

After the initial retrieval of documents based on a query, a reranking process can refine the selection, ensuring the LLM works with the most relevant and high-quality information.

This process enhances the overall performance of the LLM, improving response accuracy and relevance, particularly in specialized domains where precise information is critical.

Types of Rerankers

There is no single recipe for implementing a reranker. Several approaches have been established, some of which are:

Multi-vector rerankers: This type of reranker assigns multiple vector representations to documents and user queries, using vector similarity to rerank results.

Learning to Rank (LTR): Widely popular in recommender algorithms, LTR employs machine learning principles and training data to learn predicting an optimal ranking of results. Given a list of documents as input, the output to predict is the “best” possible ranking of them.

BERT-based rerankers: BERT is a transformer-based model specialized in language understanding tasks like text classification. In the context of reranking, BERT models can be leveraged to understand the semantic nuances of a set of retrieved documents and fine-tune their ranking.

Reinforcement learning rerankers: based on continued user interaction data, optimize the ranking guided by a long-term reward function e.g., user satisfaction that is gradually learned by trial and error, just like in a reinforcement learning algorithm.

Hybrid rerankers: they combine multiple reranking strategies, for instance, LTR with multiple combined machine learning or deep learning models.