Function calling not working in ChatGoogleGenerativeAI with gemini-1.5-pro or gemini-1.5-flash

[DevDeepakBhattarai]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

// This is the model selecting function
export function modelPicker(
  model: z.infer<typeof AvailableModels>,
  stream?: boolean,
  modelName = "gpt-3.5-turbo",
) {
  let modelObject: Model;
  switch (model) {
    case "gpt": {
      modelObject = new ChatOpenAI({
        model: modelName,
        apiKey: env.OPENAI_API_KEY,
        streaming: stream,
        modelKwargs: stream
          ? {
              parallel_tool_calls: false,
            }
          : undefined,
      });

      break;
    }

    case "claude": {
      modelObject = new ChatAnthropic({
        model: "claude-3-sonnet-20240229",
        modelName: "claude-3-sonnet-20240229",
        apiKey: env.ANTHROPIC_API_KEY,
        streaming: true,
      });
      break;
    }

    case "gemini": {
      modelObject = new ChatGoogleGenerativeAI({
        model: "gemini-1.5-pro",
        apiKey: env.GOOGLE_AI_API_KEY,
        streaming: stream,
        apiVersion: "v1beta",
        temperature: 0,
      });
      break;
    }
    case "groq": {
      modelObject = new ChatGroq({
        apiKey: env.GROQ_API_KEY,
        streaming: true,
        model: "llama3-70b-8192",
        temperature: 0.7,
      });
    }
  }
  return modelObject;
}

 const initialPrompt =
    state.model !== "groq"
      ? ChatPromptTemplate.fromMessages([
          [
            "system",
            `You are a ALLWEONE, a helpful assistant that is capable of doing anything. You are provided with a list of functions, and an input from the user.\nYour job is to determine whether to chat with the user with plain text in markdown format, or call a function help the user.\nWhen using markdown format, you will use it wisely, only using it when absolutely necessary\n\n\nBelow is the summary of the conversation that has happened between you(AI) and the user, written by a third person\n----------\n{existingSummary}\n----------\n`,
          ],
          ["placeholder", "{input}"],
          new MessagesPlaceholder({
            variableName: "chat_history",
            optional: true,
          }),
        ])
      : ChatPromptTemplate.fromMessages([
          [
            "system",
            `You are a ALLWEONE, a helpful assistant that is capable of doing anything. You are provided with a list of functions, and an input from the user.\nYour job is to determine whether to chat with the user with plain text in markdown format, or call a function help the user.\nWhen using markdown format, you will use it wisely, only using it when absolutely necessary\n\n\nBelow is the summary of the conversation that has happened between you(AI) and the user, written by a third person\n----------\n{existingSummary}\n----------\n`,
          ],
          ["user", "{objective}"],
          new MessagesPlaceholder({
            variableName: "chat_history",
            optional: true,
          }),
        ]);

const llm = modelPicker(state.model, true).bindTools(tools, {
    tool_choice: "auto",
  });

Description

I am trying to build a LLM app with multiple model support in Next JS with app router and Vercel AI sdk. I currently have gpt, claude , groq(llama) and gemini. The app works as intended with other models like gpt, claude and groq. But when ever I try to use gemini in the app, it does not work.

When I ask gemini some random question like , "Hello" etc. It works and gives me answer. But as soon as I ask the model anything related to the tool (i.e tavily search tool and get weather at the momemt) it gives me completely empty response.

This is the output that I got from gemini in Langsmith

{
  "llmOutput": {},
  "generations": [
    {
      "text": "",
      "generationInfo": {
        "finishReason": "STOP",
        "index": 0,
        "safetyRatings": [
          {
            "category": "HARM_CATEGORY_HATE_SPEECH",
            "probability": "NEGLIGIBLE"
          },
          {
            "category": "HARM_CATEGORY_HARASSMENT",
            "probability": "NEGLIGIBLE"
          },
          {
            "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
            "probability": "NEGLIGIBLE"
          },
          {
            "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
            "probability": "NEGLIGIBLE"
          }
        ]
      },
      "message": {
        "lc": 1,
        "type": "constructor",
        "id": [
          "langchain_core",
          "messages",
          "AIMessageChunk"
        ],
        "kwargs": {
          "content": "",
          "name": "model",
          "additional_kwargs": {},
          "tool_calls": [],
          "invalid_tool_calls": [],
          "tool_call_chunks": [],
          "response_metadata": {}
        }
      }
    }
  ]
}

The response does not have content nor does it have tool_calls.

I saw that tool calling is a beta feature in Gemini , so I even changed the apiVersion of ChatGoogleGenerativeAI to v1beta after looking up the docs here

Is this a problem with langchain or is it problem with Google ?

System Info

pnpm version 9.4.0
platform Windows 10
Node version 18.17.0
Next Js 14.2.5

[dosubot[bot]]

Hey there, @DevDeepakBhattarai! I'm here to help you out with any bugs, questions, or contributions you have in mind. Let's tackle this issue together! 😊

The issue you're experiencing with the Gemini model returning an empty response when asked to use a tool could be due to several factors:

1. Safety Checks: The message might be violating the safety checks of the LLM. If a message violates these checks, the model will return an empty response ^[1].

2. Tool Call Arguments: Ensure that the tool call arguments are properly formatted and that the message content is correctly structured. The Gemini model expects specific formats for tool declarations and arguments. For example, the tool must have a functionDeclarations array with properly defined name, description, and parameters ^[1].

3. First Message Constraint: Tool messages cannot be the first message passed to the model. If a tool message is the first message, an error will be thrown ^[2].

4. Configuration Options: There are additional configuration options for the ChatGoogleGenerativeAI class that could influence the behavior of tool calls. Ensure that you are using the correct parameters and configurations, such as specifying tools in the form of StructuredToolInterface[] or GoogleGenerativeAIFunctionDeclarationsTool[] ^[3].

Here is an example of how to properly configure the tool declarations:

const tools: GeminiTool[] = [
  {
    functionDeclarations: [
      {
        name: "test",
        description: "Run a test with a specific name and get if it passed or failed",
        parameters: {
          type: "object",
          properties: {
            testName: {
              type: "string",
              description: "The name of the test that should be run.",
            },
          },
          required: ["testName"],
        },
      },
    ],
  },
];

Ensure that your tool calls and message content adhere to these specifications to avoid issues that might lead to empty responses from the Gemini model.

To continue talking to Dosu, mention @dosu.

[DevDeepakBhattarai]

@dosu I know , everything is setup correctly because when ever I use other models, It works. And from the response that the model gives, it does not seem to be a problem with safety setting ?

[dosubot[bot]]

To resolve the issue with the Gemini model returning an empty response when attempting to use tools, ensure the following:

1. Tool Definition and Binding: Verify that the tool is correctly defined and bound to the model. Here is an example setup:

import { ChatVertexAI } from "@langchain/google-vertexai";
import { type GeminiTool } from "@langchain/google-vertexai/types";
import { zodToGeminiParameters } from "@langchain/google-vertexai/utils";
import { z } from "zod";

const calculatorSchema = z.object({
  operation: z.enum(["add", "subtract", "multiply", "divide"]).describe("The type of operation to execute"),
  number1: z.number().describe("The first number to operate on."),
  number2: z.number().describe("The second number to operate on."),
});

const geminiCalculatorTool: GeminiTool = {
  functionDeclarations: [
    {
      name: "calculator",
      description: "A simple calculator tool",
      parameters: zodToGeminiParameters(calculatorSchema),
    },
  ],
};

const model = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.5-flash-001",
}).bind({
  tools: [geminiCalculatorTool],
});

const response = await model.invoke("What is 1628253239 times 81623836?");
console.log(JSON.stringify(response.additional_kwargs, null, 2));

2. Response Handling: Ensure the response is correctly parsed and handled. Use the responseToParts and responseToString functions to process the response data:

import { responseToParts, responseToString } from "@langchain/google-vertexai/utils";

const parts = responseToParts(response);
const result = responseToString(parts);
console.log(result);

3. Configuration and Runtime Issues: Check for any additional configuration or runtime issues specific to your Next JS app or the Vercel AI SDK environment. Ensure that the GoogleConnection and GoogleAIConnection classes are correctly implemented and configured.

4. Error Handling: The ChatGoogleGenerativeAI class generates specific error messages when tool calls fail. For example, if tool_calls are not present in the response, it throws an error with the message "tool_calls not in response" ^[1].

If the setup is correct and you are still receiving an empty response, it might be worth checking for any updates or known issues with the Gemini model or the LangChain framework. Additionally, you can refer to the similar discussion for more insights ^[2].

To continue talking to Dosu, mention @dosu.

[DevDeepakBhattarai]

@dosu
It seem that ChatVertexAI does not give the tool call provided by the model in the result.tool_calls array. Is this a know bug ?

This is the response from gemini model in ChatVertex AI

{
  "llmOutput": {},
  "generations": [
    {
      "text": "",
      "generationInfo": {
        "finishReason": "stop"
      },
      "message": {
        "lc": 1,
        "type": "constructor",
        "id": [
          "langchain_core",
          "messages",
          "AIMessageChunk"
        ],
        "kwargs": {
          "content": "",
          "additional_kwargs": {
            "tool_calls": [
              {
                "id": "c1b8fb3040e74d10857948eb8f3bfad0",
                "type": "function",
                "function": {
                  "name": "get_weather",
                  "arguments": "{\"country\":\"us\",\"state\":\"ca\",\"city\":\"oakland\"}"
                }
              }
            ]
          },
          "response_metadata": {},
          "tool_call_chunks": [],
          "usage_metadata": {
            "input_tokens": 250,
            "output_tokens": 10,
            "total_tokens": 260
          },
          "tool_calls": [],
          "invalid_tool_calls": []
        }
      }
    }
  ]
}

Even tho there is a tool_call in the additional_kwargs provided by the model. Langchain does not provide that tool call in the tool_calls array. What is the work around for this ?

[dosubot[bot]]

It seems like the issue you're experiencing with the Gemini model in LangChain, where the tool call provided by the model is not being populated in the result.tool_calls array, might be related to how the tool calls are processed and converted within the code.

Here are a few steps to ensure that the tool calls are correctly extracted and populated:

1. Tool Definition and Binding: Verify that the tool is correctly defined and bound to the model. Here is an example setup:

import { ChatVertexAI } from "@langchain/google-vertexai";
import { type GeminiTool } from "@langchain/google-vertexai/types";
import { zodToGeminiParameters } from "@langchain/google-vertexai/utils";
import { z } from "zod";

const calculatorSchema = z.object({
  operation: z.enum(["add", "subtract", "multiply", "divide"]).describe("The type of operation to execute"),
  number1: z.number().describe("The first number to operate on."),
  number2: z.number().describe("The second number to operate on."),
});

const geminiCalculatorTool: GeminiTool = {
  functionDeclarations: [
    {
      name: "calculator",
      description: "A simple calculator tool",
      parameters: zodToGeminiParameters(calculatorSchema),
    },
  ],
};

const model = new ChatVertexAI({
  temperature: 0.7,
  model: "gemini-1.5-flash-001",
}).bind({
  tools: [geminiCalculatorTool],
});

const response = await model.invoke("What is 1628253239 times 81623836?");
console.log(JSON.stringify(response.additional_kwargs, null, 2));

2. Response Handling: Ensure the response is correctly parsed and handled. Use the responseToParts and responseToString functions to process the response data:

import { responseToParts, responseToString } from "@langchain/google-vertexai/utils";

const parts = responseToParts(response);
const result = responseToString(parts);
console.log(result);

3. Configuration and Runtime Issues: Check for any additional configuration or runtime issues specific to your Next JS app or the Vercel AI SDK environment. Ensure that the GoogleConnection and GoogleAIConnection classes are correctly implemented and configured.

4. Error Handling: The ChatGoogleGenerativeAI class generates specific error messages when tool calls fail. For example, if tool_calls are not present in the response, it throws an error with the message "tool_calls not in response".

If the setup is correct and you are still receiving an empty response, it might be worth checking for any updates or known issues with the Gemini model or the LangChain framework. Additionally, you can refer to the similar discussion for more insights.

If the issue persists, it might be a good idea to loop in a maintainer for further assistance. @jacoblee93, could you step in and provide your guidance on this issue?

To continue talking to Dosu, mention @dosu.

[DevDeepakBhattarai]

const invokeModel = async (
  state: AgentExecutorState,
  config?: RunnableConfig,
): Promise<Partial<AgentExecutorState>> => {
  const initialPrompt =
    state.model !== "groq"
      ? ChatPromptTemplate.fromMessages([
          [
            "system",
            `You are a ALLWEONE, a helpful assistant that is capable of doing anything. You are provided with a list of functions, and an input from the user.\nYour job is to determine whether to chat with the user with plain text in markdown format, or call a function help the user.\nWhen using markdown format, you will use it wisely, only using it when absolutely necessary\n\n\nBelow is the summary of the conversation that has happened between you(AI) and the user, written by a third person\n----------\n{existingSummary}\n----------\n`,
          ],
          ["placeholder", "{input}"],
          new MessagesPlaceholder({
            variableName: "chat_history",
            optional: true,
          }),
        ])
      : ChatPromptTemplate.fromMessages([
          [
            "system",
            `You are a ALLWEONE, a helpful assistant that is capable of doing anything. You are provided with a list of functions, and an input from the user.\nYour job is to determine whether to chat with the user with plain text in markdown format, or call a function help the user.\nWhen using markdown format, you will use it wisely, only using it when absolutely necessary\n\n\nBelow is the summary of the conversation that has happened between you(AI) and the user, written by a third person\n----------\n{existingSummary}\n----------\n`,
          ],
          ["user", "{objective}"],
          new MessagesPlaceholder({
            variableName: "chat_history",
            optional: true,
          }),
        ]);

  const MessageHistoryStore = new UpstashRedisChatMessageHistory({
    sessionId: `${state.userId}-chat-${state.chatId}`, // Or some other unique identifier for the conversation
    client: redis,
  });

  const memory = new ConversationSummaryBufferMemory({
    memoryKey: "history",
    maxTokenLimit: 300,
    llm: new OpenAI({
      model: "gpt-3.5-turbo",
      temperature: 0,
      apiKey: env.OPENAI_API_KEY,
    }),
  });

  const tools = [search_tool, weatherTool];
  const llm = modelPicker(state.model, true).bindTools(tools, {
    tool_choice: "auto",
  });
  const chain = initialPrompt.pipe(llm);
  const result = await chain.invoke(state, config);
  console.log("Result", result);
  if (result.tool_calls && result.tool_calls.length > 0) {
    const toolCall = {
      toolCall: {
        name: result.tool_calls[0]!.name,
        parameters: result?.tool_calls[0]!.args,
        id: result?.tool_calls[0]!.id ?? "",
      },
    };
    state.chat_history.push(result);
    console.log("ToolCall", toolCall);
    return toolCall;
  }

  if (
    state.model === "gemini" &&
    result.additional_kwargs.tool_calls &&
    result.additional_kwargs.tool_calls.length > 0
  ) {
    const toolCall = {
      toolCall: {
        name: result.additional_kwargs.tool_calls[0]!.function.name,
        parameters: safeJsonParse(
          result.additional_kwargs.tool_calls[0]!.function.arguments,
        )!,
        id: result.additional_kwargs?.tool_calls[0]!.id ?? "",
      },
    };
    state.chat_history.push(result);
    console.log("ToolCall", toolCall);
    return toolCall;
  }

  await MessageHistoryStore.addUserMessage(state.objective);
  await MessageHistoryStore.addAIMessage(result.content as string);
  const newSummary = await memory.predictNewSummary(
    [
      new HumanMessage(state.objective),
      new AIMessage(result.content as string),
    ],
    state.existingSummary,
  );

  await redis.set(`${state.userId}-summary-${state.chatId}`, newSummary);
  return {
    result: result.content as string,
    toolCall: undefined,
  };
};

This is how I am handling the inconsistency right now. I am just reading the tool call directly from the result. But this is prone to error when model outputs invalid tool call.