LangchainJs Streaming with OpenAIFunctions Agent

[saifu19]

So I am trying to make this AdonisJS app with langchain and OpenAI. I have got everything right but the streaming thing is giving me very tough time. I did everything exactly as in the documentation but it doesn't work and I have no idea why. I have this executor provider where I have functions to set memory, prompts, tools, initialize executor and invoke. I am sharing init and invoke methods below. Please help me

// Agent Intializer
public initExecutor() {
    const tools = this.tools

    const model = new ChatOpenAI({ modelName: "gpt-4", temperature: 0, streaming: true });

    const modelWithTools = model.bind({ functions: [...tools.map((tool) => formatToOpenAIFunction(tool))], });

    const formatAgentSteps = (steps: AgentStep[]): BaseMessage[] =>
        steps.flatMap(({ action, observation }) => {
            if ("messageLog" in action && action.messageLog !== undefined) {
                const log = action.messageLog as BaseMessage[];
                return log.concat(new FunctionMessage(observation, action.tool));
            } else {
                return [new AIMessage(action.log)];
            }
        });

    let runnableAgent;
    if (tools.length != 0) {
        runnableAgent = RunnableSequence.from([
            {
                input: (i: { input: string; steps: AgentStep[] }) => i.input,
                agent_scratchpad: (i: { input: string; steps: AgentStep[] }) =>
                    formatAgentSteps(i.steps),

                // Load memory here
                chat_history: async (_: { input: string; steps: AgentStep[] }) => {
                    const { history } = await this.memory.loadMemoryVariables({});
                    return history;
                },
            },
            this.prompt,
            modelWithTools,
            new OpenAIFunctionsAgentOutputParser(),
            // outputParser,
        ])
    } else {
        runnableAgent = RunnableSequence.from([
            {
                input: (i: { input: string; steps: AgentStep[] }) => i.input,
                agent_scratchpad: (i: { input: string; steps: AgentStep[] }) =>
                    formatAgentSteps(i.steps),
                // Load memory here
                chat_history: async (_: { input: string; steps: AgentStep[] }) => {
                    const { history } = await this.memory.loadMemoryVariables({});
                    return history;
                },
            },
            this.prompt,
            new OpenAIFunctionsAgentOutputParser(),
        ]);
    }
    this.executor = AgentExecutor.fromAgentAndTools({
        agent: runnableAgent,
        tools,
    });
    return this.executor;
}

// Agent Invoke
public async invoke({ input }: { input: string }) {
    try {
        const stream = await this.executor.stream({
            input: input,
        });
        for await (const chunk of stream) {
            console.log(JSON.stringify(chunk, null, 2));
        }
        console.log(stream)
        // return stream;
    } catch (error) {
        console.error(error);
        return error;
    }
}

[dosubot[bot]]

🤖

Hi @saifu19! I'm Dosu, a friendly bot here to lend a helping hand while we wait for a human maintainer. I'm here to assist you in troubleshooting bugs, answering your queries, and guiding you on how to contribute better. Let's get your issue sorted, shall we?

Based on the code you've shared, it seems like you're correctly setting up the AgentExecutor and invoking it with the stream method. However, there are a few things that might be causing issues.

Firstly, it's important to note that the stream method is designed to provide real-time updates as the language model generates new tokens, not to stream the complete response all at once. If you're expecting the latter, you might need to adjust your expectations or consider a different approach. You can refer to the chat_streaming.ts example in the langchainjs repository for more details on how streaming is expected to work.

Secondly, there was a similar issue reported in the langchainjs repository where streaming was not working when run behind a proxy server. The issue was resolved in version 0.0.142 of langchainjs. If your application is running behind a proxy, you might want to check if this is causing the issue. Here's the relevant code from the solution:

import { ProxyAgent } from 'proxy-agent';

const model = new ChatOpenAI(
  {
    streaming: true,
  },
  {
    httpAgent: new ProxyAgent(),
  },
);

You can refer to the issue Streaming does not work if run behind a proxy server for more details.

Lastly, if none of the above solutions work, it would be helpful to have more information about the error you're encountering. Are there any error messages being logged to the console? If so, could you share them? This could provide more insight into what might be going wrong.

Sources

• Streaming does not work if run behind a proxy server
• examples/src/models/chat/chat_streaming.ts
• examples/src/agents/stream.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.