Has anyone managed to get a tool_calls working when stream and append the tools chunk to message?

[josuasimatupang]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import { LangChainAdapter, Message as VercelChatMessage } from 'ai';
import { NextRequest, NextResponse } from 'next/server';

import { formatContext } from '@/app/(dashboard)/autobot/usecase/format-context';
import { extractInstanceName } from '@/app/(dashboard)/autobot/utils/extract-instance-name';
import { getAssistantPrompt } from '@/app/(dashboard)/autobot/utils/get-assistant-prompt';
import {
  SYSTEM_PROMPT_REWRITTEN,
  TEMPLATE_PREPROMPT,
} from '@/app/(dashboard)/autobot/view/presentation/AutobotSidebar/constants';
import { getContentFromDataset } from '@/app/action/server/autobotAction';
import {
  AIMessage,
  BaseMessage,
  HumanMessage,
  SystemMessage,
} from '@langchain/core/messages';
import { StringOutputParser } from '@langchain/core/output_parsers';
import { RunnableSequence } from '@langchain/core/runnables';
import { ChatOpenAI } from '@langchain/openai';

const formatMessage = (message: VercelChatMessage): BaseMessage => {
  switch (message.role) {
    case 'user':
      return new HumanMessage(message.content);
    case 'assistant':
      return new AIMessage(message.content);
    case 'system':
      return new SystemMessage(message.content);
    default:
      throw new Error(`Unknown message role: ${message.role}`);
  }
};

/**
 * This handler initializes and calls a simple chain with a prompt,
 * chat model, and output parser. See the docs for more information:
 *
 * https://js.langchain.com/docs/guides/expression_language/cookbook#prompttemplate--llm--outputparser
 */
export async function POST(req: NextRequest) {
  try {
    const startTime = performance.now(); // Start measuring total processing time
    const body = await req.json();
    const messages = body.messages ?? [];
    const autobotParameter = body.autobotParameter ?? {};
    const credentials = body.credentials ?? {};

    const formattedPreviousMessages = messages.map(formatMessage);
    const instanceName = extractInstanceName(credentials?.llm?.apiEndpoint);

    const model = new ChatOpenAI({
      azureOpenAIApiKey: credentials?.llm?.apiKey,
      azureOpenAIApiInstanceName: instanceName,
      model: credentials?.llm?.baseModel,
      azureOpenAIApiDeploymentName: credentials?.llm?.baseModel,
      azureOpenAIApiVersion: '2024-02-15-preview',
      temperature: autobotParameter.llm?.modelParameter?.temperature ?? 0.5,
      topP: autobotParameter.llm?.modelParameter?.topP ?? 0.75,
      presencePenalty:
        autobotParameter.llm?.modelParameter?.presencePenalty ?? 0.5,
      frequencyPenalty:
        autobotParameter.llm?.modelParameter?.frequencyPenalty ?? 0.5,
      maxTokens: autobotParameter.llm?.modelParameter?.maxTokens ?? 512,
      streaming: true,
      streamUsage: true,
      cache: true,
      verbose: true,
    }).bind({
      tools: [
        {
          type: 'function',
          function: {
            name: 'contact_preview_tools',
            description:
              'For previewing and managing professional contact information for employees and partners. This tool provides detailed information about personnel including their roles, responsibilities, and organizational relationships.',
            parameters: {
              type: 'object',
              required: [
                'name',
                'email',
                'phone1',
                'phone2',
                'portfolio',
                'provider',
                'label',
              ],
              properties: {
                name: {
                  type: 'string',
                  description:
                    'Full name of the contact person (First Name and Last Name)',
                },
                email: {
                  type: 'string',
                  description:
                    'Primary professional email address for communication',
                },
                phone1: {
                  type: 'string',
                  description:
                    'Primary contact number including country code and area code',
                },
                phone2: {
                  type: 'string',
                  description:
                    'Alternative contact number for backup communication',
                },
                portfolio: {
                  type: 'string',
                  description:
                    "Detailed information about the person's responsibilities and role in the company",
                },
                provider: {
                  type: 'string',
                  description:
                    'Business unit, division, or partner company affiliation',
                },
                label: {
                  type: 'string',
                  description:
                    "Primary classification of the contact's relationship with the organization",
                },
              },
              additionalProperties: false,
            },
          },
        },
        {
          type: 'function',
          function: {
            name: 'resource_centre_preview_tools',
            description:
              'For previewing article that acts as knowledge base on company',
            parameters: {
              type: 'object',
              required: ['id', 'name'],
              properties: {
                id: {
                  type: 'string',
                  description: 'The identifier of the resource centre/article',
                },
                name: {
                  type: 'string',
                  description: 'The title of the resource centre/article',
                },
              },
              additionalProperties: false,
            },
          },
        },
        {
          type: 'function',
          function: {
            name: 'clinic_preview_tools',
            description: 'For previewing registered clinics.',
            parameters: {
              type: 'object',
              required: [
                'name',
                'email',
                'telephone',
                'fax',
                'address',
                'femaleDoctorAvailability',
                'availableTest',
                'openingHour',
                'insurer',
              ],
              properties: {
                name: {
                  type: 'string',
                  description: 'The name of the clinic',
                },
                email: {
                  type: 'string',
                  description: 'Primary email address for communication',
                },
                telephone: {
                  type: 'array',
                  items: {
                    type: 'string',
                  },
                  description:
                    'Contact numbers including country code and area code',
                },
                fax: {
                  type: 'string',
                  description: 'Fax number for the clinic',
                },
                address: {
                  type: 'string',
                  description: 'Physical address of the clinic',
                },
                femaleDoctorAvailability: {
                  type: 'string',
                  description:
                    "Availability status of female doctors (e.g., 'Available', 'Not Available')",
                },
                availableTest: {
                  type: 'array',
                  items: {
                    type: 'string',
                  },
                  description: 'List of available tests at the clinic',
                },
                openingHour: {
                  type: 'string',
                  description: 'Opening hours of the clinic',
                },
                insurer: {
                  type: 'array',
                  items: {
                    type: 'string',
                  },
                  description: 'List of accepted insurers',
                },
              },
              additionalProperties: false,
            },
          },
        },
        {
          type: 'function',
          function: {
            name: 'product_commission_preview_tools',
            description: 'For previewing product commission',
            parameters: {
              type: 'object',
              required: [
                'id',
                'name',
                'provider',
                'productType',
                'benefit',
                'limitation',
                'subplan',
              ],
              properties: {
                id: {
                  type: 'string',
                  description: 'The identifier of the product',
                },
                name: {
                  type: 'string',
                  description: "The product's name",
                },
                provider: {
                  type: 'string',
                  description: 'The insurance provider who create the product',
                },
                productType: {
                  type: 'string',
                  description: "Product's type / category",
                },
                benefit: {
                  type: 'string',
                  description: 'The benefit of the product',
                },
                limitation: {
                  type: 'string',
                  description: 'The limitation of the product',
                },
                subplan: {
                  type: 'array',
                  items: {
                    type: 'object',
                    properties: {
                      subplanName: {
                        type: 'string',
                        description: 'The name of subplan',
                      },
                      commission: {
                        type: 'string',
                        description:
                          'The commission percentage or remark of suplan commission',
                      },
                    },
                    additionalProperties: false,
                    required: ['subplanName', 'commission'],
                  },
                  description: 'List of subplan under products',
                },
              },
              additionalProperties: false,
            },
          },
        },
      ],
    });

    // Rewritten Intent
    const rewrittenIntent = await model.invoke([
      ['system', SYSTEM_PROMPT_REWRITTEN],
      ...messages.map((item: { role: string; content: string }) => [
        item.role,
        item.content,
      ]),
    ]);
    // Get context
    const getContext = await getContentFromDataset(
      rewrittenIntent?.lc_kwargs.content,
    );

    if (getContext.success) {
      const modelStartTime = performance.now();
      const systemPrompt = TEMPLATE_PREPROMPT(autobotParameter?.prePrompt);
      const prompt = getAssistantPrompt(systemPrompt);
      const contextDocumentId =
        getContext.data.results[0].tags.__document_id[0];

      /**
       * Can also initialize as:
       *
       * import { RunnableSequence } from "@langchain/core/runnables";
       * const chain = RunnableSequence.from([prompt, model, outputParser]);
       */
      const chain = RunnableSequence.from([
        prompt,
        model,
        new StringOutputParser(),
      ]);

      // Create a context object with all required template variables
      const requiredTemplate = {
        chat_history: formattedPreviousMessages,
        input: rewrittenIntent?.lc_kwargs.content,
        context: formatContext(getContext.data),
      };

      // Then get the streaming response
      const stream = await chain.stream(requiredTemplate);

      return LangChainAdapter.toDataStreamResponse(stream, {
        init: {
          headers: {
            'x-processing-time': encodeURIComponent(
              JSON.stringify({
                total: performance.now() - startTime,
                model: performance.now() - modelStartTime,
              }),
            ),
            'x-prompt': encodeURIComponent(JSON.stringify(prompt)),
            'x-context-data-id': encodeURIComponent(contextDocumentId),
          },
        },
      });
    } else {
      console.error(getContext.message);
      return NextResponse.json({ error: getContext.message }, { status: 500 });
    }
  } catch (error) {
    console.error(error);
    return NextResponse.json(
      { error: error.message },
      { status: error.status },
    );
  }
}

Description

Hello! I'm new here, so I apologize in advance for any mistakes. I've been experimenting with LangChain and Vercel AI and was curious if anyone has a solution for appending tools to the stream. I'll continue researching on my end, but any guidance would be greatly appreciated. Thanks so much!

System Info

Node v22.13.0
"@langchain/core": "^0.2.10",
"@langchain/openai": "^0.2.0",
"ai": "^3.4.33",
"@ai-sdk/openai": "^1.0.11",

[dosubot[bot]]

Hey @josuasimatupang! I'm here to help you out with any bugs, questions, or contributions you have. Let me know how I can assist you today!

To get tool_calls working when streaming and appending the tools chunk to a message using LangChain and Vercel AI, you need to ensure that each tool call is followed by a corresponding tool message response. Here's a general approach you can follow:

1. Check for Tool Call Chunks: As you process the response stream, look for chunks that contain tool call information.

2. Invoke the Tool: When a tool call chunk is detected, invoke the corresponding tool and process its response.

3. Create a Tool Message: After invoking the tool, create a ToolMessage with the tool's response. This message should include the tool's name, the tool call ID, and the content of the response.

4. Append the Tool Message: Append the ToolMessage to the message history or handle it as needed in your application.

Here's a simplified example of how you might implement this in your code:

for (const response of responses) {
  const responseStream = await chainWithHistory.stream(
    {
      input: response,
      name: "Stacy",
      company: "Tech Unicorn",
      questions: questions.join("\n"),
      companyListings: companyListings,
    },
    {
      configurable: {
        sessionId: "123",
      },
    }
  );
  process.stdout.write("AI Response: ");
  for await (const chunk of responseStream) {
    if (chunk?.tool_call_chunks && chunk?.tool_call_chunks?.length > 0) {
      // Handle tool call chunks
      for (const toolCallChunk of chunk.tool_call_chunks) {
        const toolResult = await analysisTool.invoke(toolCallChunk);
        process.stdout.write(String(toolResult));

        // Ensure each tool call is followed by a ToolMessage
        const toolMessage = new ToolMessage({
          name: toolCallChunk.name,
          tool_call_id: toolCallChunk.id,
          content: String(toolResult)
        });
        // Append the tool message to the message history or handle it as needed
      }
    } else if (chunk.content) {
      process.stdout.write(String(chunk.content));
    }
  }
  console.log("\n");
}

This code snippet demonstrates how to handle tool call chunks in a streaming context, ensuring that each tool call is followed by a ToolMessage with the appropriate structure, as required by LangChain ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}