How to fix a function parsing problem before it becomes a function parsing error

[scottsuhy]

I am using an OpenAI functions agent

When I create the function definition for my function the 'name' and 'description' look like this:

const GETFROMVECTORDATABASE = new DynamicTool({
    name: "GETFROMVECTORDATABASE",
    description: "Retrieves the answer to a question from the users vector database. The user may mention their 'vector database' and if they do, then use " +
        "this to query it. " +
        "The input must be a JSON string using the same format as this example: 'input = {\"question\": \"This is my question\", \"user_email\": \"[email protected]\"}'. " +
        "This input format includes an 'input' object containing two fields: 'question' (the users question for the vector database), 'user_email' (the users email). " +
        "The function returns the answer.",

    func: async (input) => {
        const args = JSON.parse(input);

        const question = args.question;
        const user_email = args.user_email;

        const result = await myFunctions.getFromVectorDatabase(question, user_email);
        if(result.error){
            capturedLogs.push('***GETFROMVECTORDATABASE ERROR***: ' + result.error);
           }
        else{
            return result.answer;
        }
    },
});

The problem is that when this function definition gets 'input' and tries to parse it with JSON.parse(input) it fails periodically. It doesn't fail all the time, but it does fail one out of five times. It fails because 'input' comes to the function as 'undefined'.

When it works, 'input' comes in looking like this:

{"input":"{"question":"What is MCO doing with GenAI","user_email":"[email protected]"}"}.

I know this because when I turn on verbose mode, I see this:

[agent/action] [1:chain:AgentExecutor] Agent selected action: {
  \"tool\": \"GETFROMVECTORDATABASE\",
  \"toolInput\": {
    \"input\": \"{\\\"question\\\":\\\"What is MCO doing with GenAI?\\\",\\\"user_email\\\":\\\"[email protected]\\\"}\"
  },
  \"log\": \"Invoking \\\"GETFROMVECTORDATABASE\\\" with {\\\"input\\\":\\\"{\\\\\\\"question\\\\\\\":\\\\\\\"What is MCO doing with GenAI?\\\\\\\",\\\\\\\"user_email\\\\\\\":\\\\\\\"[email protected]\\\\\\\"}\\\"}\\n\",
  \"messageLog\": [
    {
      \"lc\": 1,
      \"type\": \"constructor\",
      \"id\": [
        \"langchain_core\",
        \"messages\",
        \"AIMessage\"
      ],
      \"kwargs\": {
        \"content\": \"\",
        \"additional_kwargs\": {
          \"function_call\": {
            \"name\": \"GETFROMVECTORDATABASE\",
            \"arguments\": \"{\\\"input\\\":\\\"{\\\\\\\"question\\\\\\\":\\\\\\\"What is MCO doing with GenAI?\\\\\\\",\\\\\\\"user_email\\\\\\\":\\\\\\\"[email protected]\\\\\\\"}\\\"}\"
          }
        }
      }
    }
  ]
}
[tool/start] [1:chain:AgentExecutor > 10:tool:GETFROMVECTORDATABASE] Entering Tool run with input: \"{\"question\":\"What is MCO doing with GenAI?\",\"user_email\":\"[email protected]\"}\"

When it fails, 'input' looks like this:

{"question":"What is MCO doing with GenAI?","user_email":"[email protected]"}.

I know this because when I turn on verbose mode, I see this:

[agent/action] [1:chain:AgentExecutor] Agent selected action: {
  \"tool\": \"GETFROMVECTORDATABASE\",
  \"toolInput\": {
    \"question\": \"What is MCO doing with GenAI?\",
    \"user_email\": \"[email protected]\"
  },
  \"log\": \"Invoking \\\"GETFROMVECTORDATABASE\\\" with {\\\"question\\\":\\\"What is MCO doing with GenAI?\\\",\\\"user_email\\\":\\\"[email protected]\\\"}\\n\",
  \"messageLog\": [
    {
      \"lc\": 1,
      \"type\": \"constructor\",
      \"id\": [
        \"langchain_core\",
        \"messages\",
        \"AIMessage\"
      ],
      \"kwargs\": {
        \"content\": \"\",
        \"additional_kwargs\": {
          \"function_call\": {
            \"name\": \"GETFROMVECTORDATABASE\",
            \"arguments\": \"{\\\"question\\\":\\\"What is MCO doing with GenAI?\\\",\\\"user_email\\\":\\\"[email protected]\\\"}\"
          }
        }
      }
    }
  ]
}
[tool/start] [1:chain:AgentExecutor > 10:tool:GETFROMVECTORDATABASE] Entering Tool run with input: \"undefined\"

what is the appropriate way to get 'input' into the right format? or try/catch it to reformat it? I can't try/catch it in the function definition because it's already 'undefined' by the time it gets there. I have read about handle_parsing_errors but I can't see how this would be a fix to this problem other than handling the issue more gracefully. What am i missing?

[dosubot[bot]]

🤖

Hey @scottsuhy! Great to see you again in the LangChainJS repo. Hope all is well on your end.

It seems like the issue you're encountering is due to the inconsistent format of the 'input' that is being passed to your function. Sometimes it's a stringified JSON object, and other times it's a plain JSON object. This inconsistency is likely causing the JSON.parse(input) to fail when the input is already a JSON object.

One way to handle this is to check if the input is a string before trying to parse it. If it's not a string, you can assume it's already a JSON object and use it directly. Here's how you can modify your function to handle this:

const GETFROMVECTORDATABASE = new DynamicTool({
    name: "GETFROMVECTORDATABASE",
    description: "Retrieves the answer to a question from the users vector database. The user may mention their 'vector database' and if they do, then use " +
        "this to query it. " +
        "The input must be a JSON string using the same format as this example: 'input = {\"question\": \"This is my question\", \"user_email\": \"[email protected]\"}'. " +
        "This input format includes an 'input' object containing two fields: 'question' (the users question for the vector database), 'user_email' (the users email). " +
        "The function returns the answer.",

    func: async (input) => {
        const args = typeof input === 'string' ? JSON.parse(input) : input;

        const question = args.question;
        const user_email = args.user_email;

        const result = await myFunctions.getFromVectorDatabase(question, user_email);
        if(result.error){
            capturedLogs.push('***GETFROMVECTORDATABASE ERROR***: ' + result.error);
           }
        else{
            return result.answer;
        }
    },
});

This should prevent the JSON.parse(input) from throwing an error when the input is already a JSON object.

As for the handle_parsing_errors you mentioned, it's more about handling errors that occur during the parsing of the function definitions themselves, not the inputs to the functions. So you're correct that it wouldn't be a fix for this particular issue.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• langchain-core/src/utils/stream.ts
• libs/langchain-openai/src/utils/openai-format-fndef.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot[bot]]

🤖

Based on the code snippet you provided, it seems like you're correctly using console.log to print the steps variable. However, if the steps variable is not appearing in the console, it could be due to a few reasons:

1. The plan method is not being called at all. You can verify this by adding a console.log statement at the beginning of the plan method, like so:

plan(steps, inputs, runManager) {
  console.log('plan method called'); // Add this line
  console.log(steps); // Your existing log statement
  // Rest of the code...
}

2. The steps variable is undefined or empty when the plan method is called. You can check this by logging the type of steps:

plan(steps, inputs, runManager) {
  console.log(typeof steps); // Add this line
  console.log(steps); // Your existing log statement
  // Rest of the code...
}

3. Your environment is suppressing console output. This is less likely, as most environments do not suppress console output by default. However, if you're using a test runner or a similar tool that might suppress console output, you might need to check its configuration.

Regarding your issue with the 'input' parameter being 'undefined' when passed to your GETFROMVECTORDATABASE function, it's hard to diagnose the problem without more context. Could you provide more details about how you're calling this function and what you're passing as the 'input' parameter?

For more information about the RunnableAgent class and the plan method, you can refer to the executor.js file in the LangChain JavaScript framework repository.

Sources

• langchain/src/agents/executor.ts
• langchain/src/agents/agent.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[scottsuhy]

@dosu-bot i figured out why the variables were not printing--that was my fault.

The issue may be in the way the toolInput is being created in the OpenAIFunctionsAgentOutputParser class (agents/openai_functions/output_parser.js), specifically within the parseAIMessage method.

    parseAIMessage(message) {
        if (message.content && typeof message.content !== "string") {
            throw new Error("This agent cannot parse non-string model responses.");
        }
        if (message.additional_kwargs.function_call) {
            // eslint-disable-next-line prefer-destructuring
            const function_call = message.additional_kwargs.function_call;
            try {
                const toolInput = function_call.arguments
                    ? JSON.parse(function_call.arguments)
                    : {};
                return {
                    tool: function_call.name,
                    toolInput,
                    log: `Invoking "${function_call.name}" with ${function_call.arguments ?? "{}"}\n${message.content}`,
                    messageLog: [message],
                };
            }
            catch (error) {
                throw new OutputParserException(`Failed to parse function arguments from chat model response. Text: "${function_call.arguments}". ${error}`);
            }
        }
        else {
            return {
                returnValues: { output: message.content },
                log: message.content,
            };
        }
    }

'function_call.arguments' comes in sometimes with this format:

{"input":"{"question":"What is MCO doing with GenAI","user_email":"[email protected]"}"} //THIS WORKS

and other times with this format:

{"question":"What is MCO doing with GenAI?","user_email":"[email protected]"} //THIS FAILS

I need to know what is the best method to ensure the model returns the right format for this parser to use to provide the function with the format it needs versus sending it an 'undefined' input.

[dosubot[bot]]

🤖

Based on the issue you're facing, it seems like the function_call.arguments is sometimes not a stringified JSON object, which is causing the JSON.parse to fail and return undefined. This is why the function is sometimes receiving undefined as input.

To handle this, you can modify the parseAIMessage method to check if function_call.arguments is a stringified JSON object before attempting to parse it. If it's not, you can assume it's already a parsed object and use it directly. Here's how you can do it:

parseAIMessage(message) {
    if (message.content && typeof message.content !== "string") {
        throw new Error("This agent cannot parse non-string model responses.");
    }
    if (message.additional_kwargs.function_call) {
        const function_call = message.additional_kwargs.function_call;
        try {
            let toolInput;
            if (typeof function_call.arguments === "string") {
                toolInput = JSON.parse(function_call.arguments);
            } else {
                toolInput = function_call.arguments || {};
            }
            return {
                tool: function_call.name,
                toolInput,
                log: `Invoking "${function_call.name}" with ${JSON.stringify(toolInput) ?? "{}"}\n${message.content}`,
                messageLog: [message],
            };
        }
        catch (error) {
            throw new OutputParserException(`Failed to parse function arguments from chat model response. Text: "${function_call.arguments}". ${error}`);
        }
    }
    else {
        return {
            returnValues: { output: message.content },
            log: message.content,
        };
    }
}

In this modification, we first check if function_call.arguments is a string. If it is, we parse it as before. If it's not, we assume it's already a parsed object and use it directly. We also use JSON.stringify(toolInput) when logging the function call to ensure we always log a stringified JSON object, regardless of the original format of function_call.arguments.

This modification should allow the parseAIMessage method to handle function_call.arguments in both stringified and parsed formats, preventing it from sending undefined as input.

Please try this modification and let me know if it resolves your issue. If you have any further questions or need more help, feel free to ask.

Sources

• langchain/src/agents/openai_functions/output_parser.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[scottsuhy]

@dosu-bot I only want to change the Langchain source code as a desperate last resort.

Is there any way to ensure the agent formats the data correctly before it parses it and prepares it to be sent to the function? does describing the data format the function needs in its description help with this problem? does it help to put a note in the system message to ensure consistency in format of function data? What else helps ensure the data is prepared properly? Provide me with everything you know about this topic.

[dosubot[bot]]

🤖

Based on the information you've provided, it seems like the inconsistency in the data format is causing the issue. The GETFROMVECTORDATABASE function expects a stringified JSON object as input, but sometimes it receives a plain JSON object. This inconsistency is causing the function to fail when it tries to parse the input with JSON.parse(input), as the input is 'undefined'.

To ensure consistent data formatting in the toolInput of the OpenAIFunctionsAgentOutputParser class, you can create a wrapper function that validates and formats the input before passing it to the GETFROMVECTORDATABASE function. This can be done without modifying the LangChain source code.

Here's an example of how you can do this:

function formatInputForGetFromVectorDatabase(input) {
  // Check if input is already a stringified JSON object
  if (typeof input === 'string') {
    try {
      JSON.parse(input);
      return input;
    } catch (e) {
      throw new Error('Invalid stringified JSON input for GETFROMVECTORDATABASE function');
    }
  }

  // If input is a plain JSON object, stringify it
  if (typeof input === 'object') {
    return JSON.stringify(input);
  }

  throw new Error('Invalid input for GETFROMVECTORDATABASE function');
}

// Then, when calling the GETFROMVECTORDATABASE function:
const formattedInput = formatInputForGetFromVectorDatabase(rawInput);
const result = await GETFROMVECTORDATABASE.func(formattedInput);

This way, you can ensure that the input to the GETFROMVECTORDATABASE function is always a stringified JSON object, regardless of the format of the original input.

As for the toolInput creation in the OpenAIFunctionsAgentOutputParser class, it is indeed created by parsing the arguments property of the function_call object. If the arguments property is not present, an empty object is used as the toolInput. This is done in the parseAIMessage method of the OpenAIFunctionsAgentOutputParser class.

However, modifying the OpenAIFunctionsAgentOutputParser class would require changes to the LangChain source code, which you mentioned you want to avoid. Therefore, the best approach would be to ensure the data is correctly formatted before it is passed to the GETFROMVECTORDATABASE function, as shown in the example above.

I hope this helps! If you have any further questions, feel free to ask.

Sources

• langchain/src/agents/openai_functions/output_parser.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[scottsuhy]

there is a bug in the langchain code related to JSON. The workaround in this bug report #1115 worked for me.