Video Input for Bedrock

[jswietek]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

const {ChatBedrockConverse} = require('@langchain/aws');
const { ChatPromptTemplate } = require("@langchain/core/prompts");
const {HumanMessage} = require('@langchain/core/messages')

const model = new ChatBedrockConverse({
  region: 'us-east-1',
  model: 'amazon.nova-lite-v1:0',
});

const text = `This is my prompt`

const performTest = async () => {
  const response = await model.invoke([new HumanMessage({
    content: [
      {
        type: 'video',
        video: {
          format: "mp4",
          source: {
            s3Location: {
              uri: `s3://test_bucket/sample.mp4`
            }
          }
        }
      },
      {type: 'text', text},
    ]
    })]);
  console.log(response);
}

performTest();

Description

I'm trying to use langchainjs package with Nova Lite model in Bedrock. I wanted to attach a video to a prompt so I can ask the model questions about the attached video. AWS models in bedrock allow attaching the file directly from an S3 bucket. Is that supported by langchainjs or langchain/aws?

If attaching video directly from s3 s not supported - is there a support for attaching video at all? If yes, could yo share an example?

System Info

Node v20.12.2
mac

npm info langchain:
[email protected] | MIT | deps: 12 | versions: 306
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/langchain/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-0.3.11.tgz
.shasum: dbfadf47aaffc23fc0ac80c953c41fe8176614c9
.integrity: sha512-PgAG4ZLeuSRkKsyf98cmWGdwKv3I1hOFC8a4fr7e+bm7E+F6Fx6xUkgbuC78ff0N/Cjs5BBryZIFMrqoKPqsvg==
.unpackedSize: 2.9 MB

dependencies:
@langchain/openai: >=0.1.0 <0.4.0 openapi-types: ^12.1.3
@langchain/textsplitters: >=0.0.0 <0.2.0 p-retry: 4
js-tiktoken: ^1.0.12 uuid: ^10.0.0
js-yaml: ^4.1.0 yaml: ^2.2.1
jsonpointer: ^5.0.1 zod-to-json-schema: ^3.22.3
langsmith: ^0.2.8 zod: ^3.22.4

maintainers:

• nfcampos [email protected]
• jacoblee93 [email protected]
• andrewnguonly [email protected]
• davidduong [email protected]
• hwchase17 [email protected]
• basproul [email protected]

dist-tags:
latest: 0.3.11 tag-for-publishing-older-releases: 0.2.20
next: 0.3.2-rc.0

[dosubot[bot]]

Hello @jswietek! I'm here to assist you with any bugs, questions, or even becoming a contributor. Feel free to ask for help!

The ChatBedrockConverse class in langchainjs does not explicitly support attaching video files directly from an S3 bucket. However, you can attach media files, such as videos, to prompts by converting them to a base64 string and including them in the message content. Here's an example of how you can do this:

const fs = require('fs');
const { ChatPromptTemplate } = require("@langchain/core/prompts");
const { HumanMessage } = require('@langchain/core/messages');

function fileToBase64(filePath) {
  return fs.readFileSync(filePath, "base64");
}

const videoFilePath = "path/to/your/video.mp4";
const videoInBase64 = fileToBase64(videoFilePath);

const prompt = ChatPromptTemplate.fromMessages([
  new HumanMessage({
    content: [
      {
        type: "media",
        mimeType: "video/mp4",
        data: videoInBase64,
      },
      {
        type: "text",
        text: `The following video is an overview...`,
      },
    ],
  }),
]);

// Use the prompt with your model

This example demonstrates how to encode a video file in base64 and include it as a media type in the message content ^[1]. If you need to use S3 URIs directly, you might need to handle the file retrieval and conversion to base64 yourself before attaching it to the prompt.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[jswietek]

@dosu
This code throws a following exception:

Error: Unsupported content block type: media