How to maintain conversations with chat history, document retrieval, prompt and images ?

[Originn]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

const CONDENSE_PROMPT = `Given the history of the conversation and a follow up question, rephrase the follow up question to be a standalone question.
If the follow up question does not need context like when the follow up question is a remark like: excellent, thanks, thank you etc., return the exact same text back.

Chat History:
{chat_history}
Follow Up Input: {question}
Standalone question:`;

const QA_PROMPT = `You are a multilingual helpful and friendly assistant. 

- If the are images in the Images section, look carefully at the image before answering the question.
- If you do not have the information in the context to answer a question, admit it openly without fabricating responses.
- If the user's questions is valid and there is no documentation or context about it, let him know that he can leave a comment and we will do our best to include it at a later stage.
Your responses should be tailored to the question's intent, using text formatting (bold with **, italic with __, strikethrough with ~~) to enhance clarity, and organized with headings, paragraphs, or lists as appropriate.
=========
context: {context}
Images: {images}
=========
Question: {question}
Answer in the {language} language:`;

export const makeChain = (vectorstore: PineconeStore, onTokenStream: (token: string) => void, userEmail: string) => {
  const streamingModel = new ChatOpenAI({
    streaming: true,
    modelName: MODEL_NAME,
    temperature: TEMPRATURE,
    modelKwargs: {
      seed: 1,
    },
    callbacks: [
      {
        handleLLMNewToken: (token) => {
          onTokenStream(token); // Forward the streamed token to the front-end
        },
      },
    ],
  });

  const nonStreamingModel = new OpenAI({
    modelName: 'gpt-4o',
    temperature: TEMPRATURE
  });

  const translationModel = new OpenAI({
    modelName: 'gpt-4o',
    temperature: TEMPRATURE
  });

  function generateUniqueId(): string {
    return uuidv4();
  }

  return {
    call: async (input: string, Documents: MyDocument[], roomId: string, userEmail: string) => {
      const qaId = generateUniqueId();

      const chat_memory = MemoryService.getChatMemory(roomId);
      const language = await detectLanguageWithOpenAI(input, nonStreamingModel);

      if (language !== 'English') {
        input = await translateToEnglish(input, translationModel);
      }

      const imageUrls = await extractImageUrlsFromMemory(chat_memory);
      const formattedImages = imageUrls.join(', ');
      const formattedContext = `{context}`;

      const formattedPrompt = QA_PROMPT.replace('{language}', language)
        .replace('{context}', formattedContext)
        .replace('{images}', formattedImages);

      if ((chat_memory.chatHistory as any).messages.length === 0) {
        const initialHumanMessage = new HumanMessage({
          content: "Hi",
          name: "Human",
        });
        chat_memory.chatHistory.addMessage(initialHumanMessage);
      }

      const customRetriever = new CustomRetriever(vectorstore);
      const chain = ConversationalRetrievalQAChain.fromLLM(
        streamingModel,
        customRetriever,
        {
          memory: chat_memory,
          questionGeneratorChainOptions: {
            llm: nonStreamingModel,
          },
          qaTemplate: formattedPrompt,
          questionGeneratorTemplate: CONDENSE_PROMPT,
          returnSourceDocuments: true,
          verbose: true,
        }
      );

      const response = await chain.invoke({
        question: input,
        chat_history: chat_memory.chatHistory,
      });

      await chat_memory.saveContext(
        { question: input },
        { text: response.text }
      );

      const minScoreSourcesThreshold = process.env.MINSCORESOURCESTHRESHOLD !== undefined ? parseFloat(process.env.MINSCORESOURCESTHRESHOLD) : 0.78;
      let embeddingsStore;
      if (language !== 'English') {
        embeddingsStore = await customRetriever.storeEmbeddings(response.text, minScoreSourcesThreshold);
        Documents = [...response.sourceDocuments];
        
        // Apply filtering after combining sources
        Documents = Documents.filter(doc => doc.metadata.type !== 'other' && doc.metadata.type !== "txt" && doc.metadata.type !== "user_input");
      
        for (const [doc, score] of embeddingsStore) {
          if (doc.metadata.type !== "txt" && doc.metadata.type !== "user_input") {
            const myDoc = new MyDocument({
              pageContent: doc.pageContent,
              metadata: {
                source: doc.metadata.source,
                type: doc.metadata.type,
                videoLink: doc.metadata.videoLink,
                file: doc.metadata.file,
                score: score,
                image: doc.metadata.image
              }
            });
      
            Documents.push(myDoc);
          }
        }
        Documents.sort((a, b) => (b.metadata.score ? 1 : 0) - (a.metadata.score ? 1 : 0));
      }
      
      if (language === 'English') {
        embeddingsStore = await customRetriever.storeEmbeddings(response.text, minScoreSourcesThreshold);
        for (const [doc, score] of embeddingsStore) {
          const myDoc = new MyDocument({
            pageContent: doc.pageContent,
            metadata: {
              source: doc.metadata.source,
              type: doc.metadata.type,
              videoLink: doc.metadata.videoLink,
              file: doc.metadata.file,
              score: score,
              image: doc.metadata.image
            }
          });
      
          Documents.push(myDoc);
        }
        
        // Apply filtering after combining sources
        Documents = Documents.filter(doc => doc.metadata.type !== 'other');
      }
      
      if (roomId) {
        io.to(roomId).emit(`fullResponse-${roomId}`, {
          roomId: roomId,
          sourceDocs: Documents,
          qaId: qaId
        });
      } else {
        io.emit("fullResponse", {
          sourceDocs: Documents,
          qaId: qaId
        });
      }
     
     
      MemoryService.updateChatMemory(roomId, chat_memory);

      return {
        text: response.text,
        sourceDocuments: response.sourceDocuments
      };
    },
    vectorstore,
  };
};

Description

Although I use ConversationalRetrievalQAChain, and because it is now depreciated,

I would like to know if it is possible to set up a createRetrievalChain that will be able to maintain conversations with chat history, document retrieval, prompt and images.

for example, I upload an image and ask a question about the image. The createRetrievalChain will see the image, will also gather embeddings with context to the question and then give the answer based on the image + embedding context.

Then I ask another question maybe related to the image or maybe not, and if there was an image url in chat_history on in other location, it will read the image and embeddings context and also answer the question and the process repeats.

in the above code I have tried to implement that but he is not able to read the image URL as is.

System Info

[email protected] | MIT | deps: 16 | versions: 282

dependencies:
@langchain/core: >=0.2.11 <0.3.0 js-tiktoken: ^1.0.12 langsmith: ~0.1.30 uuid: ^10.0.0
@langchain/openai: >=0.1.0 <0.3.0 js-yaml: ^4.1.0 ml-distance: ^4.0.0 yaml: ^2.2.1
@langchain/textsplitters: ~0.0.0 jsonpointer: ^5.0.1 openapi-types: ^12.1.3 zod-to-json-schema: ^3.22.3
binary-extensions: ^2.2.0 langchainhub: ~0.0.8 p-retry: 4 zod: ^3.22.4

dist-tags:
latest: 0.2.10 next: 0.2.3-rc.0

[dosubot[bot]]

To set up a createRetrievalChain that maintains conversations with chat history, document retrieval, prompts, and images, you can follow these steps:

1. Initialize the Language Model and Retriever:

   import { ChatOpenAI, OpenAIEmbeddings } from "@langchain/openai";
   import { CheerioWebBaseLoader } from "@langchain/community/document_loaders/web/cheerio";
   
   const chatModel = new ChatOpenAI({});
   const embeddings = new OpenAIEmbeddings({});
   const loader = new CheerioWebBaseLoader("https://docs.smith.langchain.com/user_guide");
   
   const docs = await loader.load();

2. Split Documents:

   import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
   
   const splitter = new RecursiveCharacterTextSplitter();
   const splitDocs = await splitter.splitDocuments(docs);

3. Create Vector Store:

   import { MemoryVectorStore } from "langchain/vectorstores/memory";
   
   const vectorstore = await MemoryVectorStore.fromDocuments(splitDocs, embeddings);

4. Create the Contextualize Question Prompt:

   import { ChatPromptTemplate } from "@langchain/core/prompts";
   
   const QUESTION_GEN_TEMPLATE = `Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question.
   
   Chat History:
   {chat_history}
   Follow Up Input: {input}
   Standalone question:`;
   
   const contextualizeQPrompt = ChatPromptTemplate.fromTemplate(QUESTION_GEN_TEMPLATE);

5. Create the History Aware Retriever:

   import { createHistoryAwareRetriever } from "langchain/chains/history_aware_retriever";
   
   const historyAwareRetriever = await createHistoryAwareRetriever({
     llm: chatModel,
     retriever: vectorstore.asRetriever(),
     rephrasePrompt: contextualizeQPrompt,
   });

6. Create the Question Answering Prompt:

   const COMBINE_DOCS_PROMPT = `Based on the following context:
   
   {context}
   
   And chat history:
   {chat_history}
   
   Answer the following question:
   {input}`;
   
   const qaPrompt = ChatPromptTemplate.fromTemplate(COMBINE_DOCS_PROMPT);

7. Create the Question Answer Chain:

   import { RunnablePassthrough, RunnableSequence } from "@langchain/core/runnables";
   import { StringOutputParser } from "@langchain/core/output_parsers";
   
   const combineDocsChain = RunnableSequence.from([
     RunnablePassthrough.assign({
       context: (input: { context: Document[] }) =>
         input.context.map((doc) => doc.pageContent).join("\n\n"),
     }),
     qaPrompt,
     chatModel,
     new StringOutputParser(),
   ]);

8. Create the Retrieval Chain:

   import { createRetrievalChain } from "langchain/chains/retrieval";
   
   const ragChain = await createRetrievalChain({
     retriever: historyAwareRetriever,
     combineDocsChain: combineDocsChain,
   });

9. Invoke the Chain with Chat History and Input:

   import { HumanMessage, AIMessage } from "@langchain/core/messages";
   
   const chat_history = [
     new HumanMessage("Can LangSmith help test my LLM applications?"),
     new AIMessage("Yes!"),
   ];
   const response = await ragChain.invoke({
     chat_history,
     input: "Tell me how!",
   });
   
   console.log(response.answer);

This setup ensures that the createRetrievalChain maintains conversations with chat history, document retrieval, prompts, and images. Additionally, you can refer to the test example in the langchain/src/chains/tests/retrieval_chain.test.ts file for further insights on how to structure and test your retrieval chain ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[Originn]

@dosu I don't see how this suggestion can handle image URLs correctly as described in my initial question

[dosubot[bot]]

To handle image URLs correctly in your createRetrievalChain setup, you need to ensure that the image URLs are processed and included in the context for answering questions. Here's an updated version of your setup that includes processing image URLs:

1. Initialize the Language Model and Retriever:

   import { ChatOpenAI, OpenAIEmbeddings } from "@langchain/openai";
   import { CheerioWebBaseLoader } from "@langchain/community/document_loaders/web/cheerio";
   
   const chatModel = new ChatOpenAI({});
   const embeddings = new OpenAIEmbeddings({});
   const loader = new CheerioWebBaseLoader("https://docs.smith.langchain.com/user_guide");
   
   const docs = await loader.load();

2. Split Documents:

   import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
   
   const splitter = new RecursiveCharacterTextSplitter();
   const splitDocs = await splitter.splitDocuments(docs);

3. Create Vector Store:

   import { MemoryVectorStore } from "langchain/vectorstores/memory";
   
   const vectorstore = await MemoryVectorStore.fromDocuments(splitDocs, embeddings);

4. Extract Image URLs from Documents:

   function extractImageUrls(docs) {
     return docs.map(doc => {
       const imageUrls = doc.pageContent.match(/(https?:\/\/.*\.(?:png|jpg|jpeg|gif))/g);
       return {
         ...doc,
         imageUrls: imageUrls || []
       };
     });
   }
   
   const docsWithImages = extractImageUrls(docs);

5. Create the Contextualize Question Prompt:

   import { ChatPromptTemplate } from "@langchain/core/prompts";
   
   const QUESTION_GEN_TEMPLATE = `Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question.
   
   Chat History:
   {chat_history}
   Follow Up Input: {input}
   Standalone question:`;
   
   const contextualizeQPrompt = ChatPromptTemplate.fromTemplate(QUESTION_GEN_TEMPLATE);

6. Create the History Aware Retriever:

   import { createHistoryAwareRetriever } from "langchain/chains/history_aware_retriever";
   
   const historyAwareRetriever = await createHistoryAwareRetriever({
     llm: chatModel,
     retriever: vectorstore.asRetriever(),
     rephrasePrompt: contextualizeQPrompt,
   });

7. Create the Question Answering Prompt:

   const COMBINE_DOCS_PROMPT = `Based on the following context:
   
   {context}
   
   And chat history:
   {chat_history}
   
   Answer the following question:
   {input}`;
   
   const qaPrompt = ChatPromptTemplate.fromTemplate(COMBINE_DOCS_PROMPT);

8. Create the Question Answer Chain:

   import { RunnablePassthrough, RunnableSequence } from "@langchain/core/runnables";
   import { StringOutputParser } from "@langchain/core/output_parsers";
   
   const combineDocsChain = RunnableSequence.from([
     RunnablePassthrough.assign({
       context: (input: { context: Document[] }) =>
         input.context.map((doc) => doc.pageContent).join("\n\n"),
     }),
     qaPrompt,
     chatModel,
     new StringOutputParser(),
   ]);

9. Create the Retrieval Chain:

   import { createRetrievalChain } from "langchain/chains/retrieval";
   
   const ragChain = await createRetrievalChain({
     retriever: historyAwareRetriever,
     combineDocsChain: combineDocsChain,
   });

10. Invoke the Chain with Chat History and Input:

    import { HumanMessage, AIMessage } from "@langchain/core/messages";
    
    const chat_history = [
      new HumanMessage("Can LangSmith help test my LLM applications?"),
      new AIMessage("Yes!"),
    ];
    const response = await ragChain.invoke({
      chat_history,
      input: "Tell me how!",
    });
    
    console.log(response.answer);

This setup ensures that the createRetrievalChain maintains conversations with chat history, document retrieval, prompts, and images. The extractImageUrls function processes image URLs from the documents and includes them in the context for answering questions ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.