Custom Id for document creation

[KeenanFernandes2000]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

if (response.status === 200) {
      const $ = cheerio.load(response.data);
      const scrapedData: string[] = [];

      // Example: scraping all paragraph texts
      $("p").each((index, element) => {
        const paragraph = $(element).text().trim();
        if (paragraph) {
          scrapedData.push(paragraph);
        }
      });
      // Send scraped data as response
      const scrape = scrapedData.join("\n");

      //@ts-ignore
      allResults["Link " + i] = {
        title: title,
        url: link,
        description: desc,
        content: scrape,
      };

      const output = await splitter.createDocuments([scrape]);

      console.log(output);

      // await pineconeStore.addDocuments(output);

      await new Promise((resolve) => setTimeout(resolve, 3000));
    }

Description

I am scraping data and dynamically creating a document to push to pinecone, however I already have data on pinecone, so i can't use 'await pineconeStore.delete({ deleteAll: true });' to delete the scraped data on pinecone. Is there a way to set a custom Id so that i can only delete the scraped data or is there another solution?

System Info

langchain 0.2.0

[dosubot[bot]]

You can set custom IDs for the documents you add to Pinecone, which will allow you to delete specific scraped data later. When adding documents using the addDocuments method, you can provide an array of custom IDs. Here's how you can modify your example code to include custom IDs:

Modified Example Code

if (response.status === 200) {
  const $ = cheerio.load(response.data);
  const scrapedData: string[] = [];

  // Example: scraping all paragraph texts
  $("p").each((index, element) => {
    const paragraph = $(element).text().trim();
    if (paragraph) {
      scrapedData.push(paragraph);
    }
  });
  // Send scraped data as response
  const scrape = scrapedData.join("\n");

  //@ts-ignore
  allResults["Link " + i] = {
    title: title,
    url: link,
    description: desc,
    content: scrape,
  };

  const output = await splitter.createDocuments([scrape]);

  console.log(output);

  // Generate custom IDs for the documents
  const customIds = output.map((_, index) => `scraped-doc-${i}-${index}`);

  // Add documents with custom IDs to Pinecone
  await pineconeStore.addDocuments(output, { ids: customIds });

  await new Promise((resolve) => setTimeout(resolve, 3000));
}

Deleting Specific Scraped Data

Later, you can delete these documents by specifying the custom IDs in the delete method:

// Example: Deleting the documents with custom IDs
const customIdsToDelete = output.map((_, index) => `scraped-doc-${i}-${index}`);
await pineconeStore.delete({ ids: customIdsToDelete });

This approach ensures that you can manage and delete specific documents based on the custom IDs you set.

Details

Sources

• libs/langchain-pinecone/src/vectorstores.ts
• libs/langchain-community/src/vectorstores/pinecone.ts
• libs/langchain-pinecone/src/tests/vectorstores.int.test.ts
• examples/src/indexes/vector_stores/pinecone/delete_docs.ts
• libs/langchain-textsplitters/src/text_splitter.ts
• libs/langchain-community/src/stores/doc/in_memory.ts
• langchain-core/src/indexing/base.ts

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.