Augmenting Human Judgment to Enhance Retrieval
Dropbox is leveraging 'large language models' (LLMs) to scale the process of 'labeling documents' for its 'Retrieval-Augmented Generation' (RAG) systems. The core problem is that existing search systems, like Dropbox's "Dash," face a bottleneck in retrieving the most pertinent information from vast repositories of documents before feeding it to an LLM. This retrieval step is crucial because, in large enterprises, indices can contain billions of documents, meaning only a small fraction can be passed on for analysis.
The strategy involves using LLMs to generate relevance ratings for query-document pairs, which are then compared against human judgments. This comparison focuses intensely on discrepancies where human behavior (like clicking a document an LLM rated low, or skipping one it rated high) provides the most valuable feedback for refining the system. This 'supervised learning' approach requires a significant volume of high-quality relevance labels, a challenge addressed by the LLM augmentation.
Read More: AI development faster than rules, experts worry about control
Evaluating the AI's Insight
Dropbox's evaluation process meticulously compares the LLM-generated relevance scores with human evaluations. This isn't just about matching scores; it involves 'penalties for disagreement'. The LLM's performance is benchmarked against a test subset of query-document pairs that were deliberately excluded from the training data. This method ensures the system's ability to generalize and perform reliably on unseen data.
The goal is to improve the 'quality of document retrieval', which is identified as a primary constraint in RAG applications. By using LLMs to produce relevance judgments at scale, Dropbox aims to overcome the limitations of manual labeling, which can be both time-consuming and expensive, particularly when dealing with the sheer volume of data in enterprise environments.
Technical Underpinnings and Future Directions
While exploring various sophisticated techniques, including 'knowledge graphs' and advanced context engines, Dropbox ultimately leaned on 'index-based retrieval' methods for its Dash platform. This choice was influenced by the need for broad access to company-wide connectors and shared content, a facet where the complexity of knowledge graphs presented significant hurdles. The company is also employing 'DSPy', a framework designed for systematically optimizing prompts through iterative evaluation.
Read More: Amazon and Coca-Cola Long-Term Stock Picks Questioned by Analysts in 2024
The broader discussion around RAG systems touches upon the fundamentals of how these systems function: retrieving relevant data to enhance LLM accuracy, especially on topics outside their original training scope. Challenges and advanced techniques are continually being explored, with an emphasis on optimizing the 'retrieval component' of the RAG pipeline. This often involves fine-tuning 'embeddings' specific to a domain or utilizing models like 'Cohere' for reranking documents. The conversation also includes methodologies for evaluating LLM performance in RAG, with frameworks like 'RAGAS' and concepts such as 'Faithfulness' and 'Context Adherence' being key considerations. The ongoing development in this space suggests a push towards more robust evaluation APIs and the integration of 'LLM-as-a-judge' capabilities.