Upsert docs to Pinecone using a key

[bdcorps]

I'm uploading documents to Pinecone using:

import { PineconeClient } from "@pinecone-database/pinecone";
import { Document } from "langchain/document";
import { OpenAIEmbeddings } from "langchain/embeddings/openai";
import { PineconeStore } from "langchain/vectorstores/pinecone";

const docs:any = [
  new Document({
    metadata: { baz: "qux"},
    pageContent: "pinecones are the woody fruiting body and of a pine tree",
  }),
];
const index = client.Index("my-index");
const res = await PineconeStore.fromDocuments(docs, new OpenAIEmbeddings(), {
  pineconeIndex: index, 
  textKey: "key2"
});

But every time I upload the same documents, it creates additional vectors in the store. I know there's a way to specify a key to only upsert docs.

I've tried specifying the textKey property in the fromDocuments but that isn't working.

Should I be doing this a different way?

[dbotha]

You'll need to use the addDocuments method as it allows you to pass an array of string ids to use. The static fromDocuments function has no means of passing the ids.

Something like this will work:

const pineconeStore = new PineconeStore(new OpenAIEmbeddings(), {
    pineconeIndex: index
});

await pineconeStore.addDocuments(
    docs,
    ids
);

[sugarforever]

What are the ids in this case? Does it mean I have to fetch all the record IDs in every upsert operation? It sounds very inefficient.

[billkabanga]

How do you even get the ids? 😄 similaritySearch doesn't return the id?

[siquick]

I'm uploading documents to Pinecone using:

import { PineconeClient } from "@pinecone-database/pinecone";
import { Document } from "langchain/document";
import { OpenAIEmbeddings } from "langchain/embeddings/openai";
import { PineconeStore } from "langchain/vectorstores/pinecone";

const docs:any = [
  new Document({
    metadata: { baz: "qux"},
    pageContent: "pinecones are the woody fruiting body and of a pine tree",
  }),
];
const index = client.Index("my-index");
const res = await PineconeStore.fromDocuments(docs, new OpenAIEmbeddings(), {
  pineconeIndex: index, 
  textKey: "key2"
});

But every time I upload the same documents, it creates additional vectors in the store. I know there's a way to specify a key to only upsert docs.

I've tried specifying the textKey property in the fromDocuments but that isn't working.

Should I be doing this a different way?

Did you manage to find a solution?

[sugarforever]

The only way I can think of so far, is to use namespace to identify a document or relevant documents (say chunks of a document). When same file is to be indexed, and the namespace already exists, delete the namespace and rebuild the index. It's brutal.

[siquick]

Yeah thats tricky for web content which can change all the time - only way I can think of is to generate a unique ID based upon the web page URL + n

[ahmedsomaa]

You can get the added documents Ids from the addDocuments method. When you insert the documents initially, the addDocuments method returns a list of Ids for the inserted docs. So you should use a simple database table to store the id for each document with any other unique metadata for the doc, and when you try to do an update for a document, just fetch the document id with that unique metadata detail from the db.