RAGs to Riches: How Our Content Affects Retrieval Augmented Generation

 

Manny Silva, Head of Documentation at Skyflow, started by saying there is a knowledge problem. AI was around long before generative AI, and gen AI tools have significant challenges. As technical communicators, we are uniquely qualified to solve this problem, and RAG is part of that solution.

RAG is retrieval augmented generation.

 Today's language models are incredibly powerful--but also have limitations. LLMs are trained on vast troves of data. This approach has limitations. The training data is inherently outdated, which leads to a knowledge cutoff. When asked about topics outside their training data, LLMs hallucinate. The inherent boas is to provide an answer, any answer, even if they have no data on which to base an answer. It is a horrendous problem. 

Hallucinations are more insidious when they are plausible sounding. They undermine user trust. 

RAG: a technique aiming to solve this problem by providing LLMs with relevant information at runtime. RAG works when a user submits a query, it goes to the retrieval system, which gets relevant documents, sends that to the LLM, which generates a response.  The response includes the relevant data that is provided at runtime. 

A couple of different ways to implement RAG: vector database and knowledge graph. 

Vector-based RAG is most common and easiest to implement.  Allow us to find relevant information from similarity searches. Includes document processing pipeline, vector database, retrieval mechanism, and LLM integration. Similarity search is really fast and low cost. Scales well for large doc sets. But doesn't explicitly understand relationship between contexts and struggles with longform content. Quality of vector-based RAG depends on how well you process your content. 

Disadvantage of a knowledge graph is that they take a lot of time and effort to create. There is additional computational overhead because it is computational AI. This means additional cost, especially at enterprise scale.  

So content best practices. Think about how content will be chunked and retrieved. 

Clear headings are important. Logical organization, whether more or less structured. Summary paragraphs are particularly useful. Consistent formatting across documents. 

 Metadata and tagging are important. May capture information not in actual content. Can often include metadata with each chunk. Think of this as SEO for RAG.

Human and machine needs. Where appropriate, content should be machine readable, such as DITA, XML, etc.  Maintain (and ship) a glossary.