Technical Process

The technical process generally follows this sequence:

  • Load the CSV file.
  • Embed the selected data into a vector database.
  • Create a compression_retriever that I’ll introduce in details
  • Perform a similarity_search.
  • Generate a prompt based on previous conversations.
  • Define a chain with source data using RetrievalQAWithSourcesChain
  • Send the output to the LLM for semantic optimization.

The following diagram illustrates the main work flow of the data processing.

    
file CSVLoader
file embedding
database vectorstore
file base_retriever
file compression_retriever
file llm
file result

CSVLoader -> embedding
embedding -> vectorstore
vectorstore --> base_retriever
compression_retriever <- base_retriever
compression_retriever --> llm: prompt
llm -> result

Figure 5.1 Work Flow