Module not found: Error: Can't resolve 'fs' when using langchain document loader

[limyenkai]

Example Code

import { WebPDFLoader } from "@langchain/community/document_loaders/web/pdf";

const blob = new Blob();
const loader = new WebPDFLoader(blob);
const docs = await loader.load();
console.log({ docs });

Description

I'm trying to implement langchain in my React webapp. How can I resolve this ERROR in ./node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js 17187:13-26 Module not found: Error: Can't resolve 'fs' in <LOCAL_PATH>. 'fs' is no where to be found in my code. Thanks in advance.

Tested solution but not useful:

 "browser": {
    "fs": false
  },

System Info

langchain version: 0.2.5
OS: MacOS Sonoma
npm version: 10.7.0
node version: 20.14.0
yarn version: 1.22.22

[dosubot[bot]]

To resolve the issue with the WebPDFLoader from @langchain/community/document_loaders/web/pdf in your React web app, you need to ensure that the pdf-parse library is properly installed and configured. The pdf-parse library relies on Node.js modules like fs, which are not available in the browser environment.

Here is a step-by-step solution:

1. Install pdf-parse:
   Ensure that you have pdf-parse installed in your project. You can install it using npm:

   npm install pdf-parse

2. Use a Browser-Compatible PDF Library:
   Since pdf-parse is not browser-compatible due to its dependency on Node.js modules, you should use a library like pdfjs-dist which is designed to work in the browser.

3. Modify the WebPDFLoader to Use pdfjs-dist:
   Update the WebPDFLoader to use pdfjs-dist instead of pdf-parse. Here is an example of how you can modify the loader:

   import { Document } from "@langchain/core/documents";
   import { BaseDocumentLoader } from "../base.js";
   import { formatDocumentsAsString } from "../../util/document.js";
   import { logVersion020MigrationWarning } from "../../util/entrypoint_deprecation.js";
   import * as pdfjsLib from "pdfjs-dist";
   
   /* #__PURE__ */ logVersion020MigrationWarning({
     oldEntrypointName: "document_loaders/web/pdf",
     newPackageName: "@langchain/community",
   });
   
   export class WebPDFLoader extends BaseDocumentLoader {
     protected blob: Blob;
   
     protected splitPages = true;
   
     protected parsedItemSeparator: string;
   
     constructor(
       blob: Blob,
       {
         splitPages = true,
         parsedItemSeparator = "",
       } = {}
     ) {
       super();
       this.blob = blob;
       this.splitPages = splitPages ?? this.splitPages;
       this.parsedItemSeparator = parsedItemSeparator;
     }
   
     async load(): Promise<Document[]> {
       const loadingTask = pdfjsLib.getDocument({
         data: new Uint8Array(await this.blob.arrayBuffer()),
         useWorkerFetch: false,
         isEvalSupported: false,
         useSystemFonts: true,
       });
       const parsedPdf = await loadingTask.promise;
       const meta = await parsedPdf.getMetadata().catch(() => null);
   
       const documents: Document[] = [];
   
       for (let i = 1; i <= parsedPdf.numPages; i += 1) {
         const page = await parsedPdf.getPage(i);
         const content = await page.getTextContent();
   
         if (content.items.length === 0) {
           continue;
         }
   
         let lastY;
         const textItems = [];
         for (const item of content.items) {
           if ("str" in item) {
             if (lastY === item.transform[5] || !lastY) {
               textItems.push(item.str);
             } else {
               textItems.push(`\n${item.str}`);
             }
             lastY = item.transform[5];
           }
         }
         const text = textItems.join(this.parsedItemSeparator);
   
         documents.push(
           new Document({
             pageContent: text,
             metadata: {
               pdf: {
                 version: pdfjsLib.version,
                 info: meta?.info,
                 metadata: meta?.metadata,
                 totalPages: parsedPdf.numPages,
               },
               loc: {
                 pageNumber: i,
               },
             },
           })
         );
       }
   
       if (this.splitPages) {
         return documents;
       }
   
       if (documents.length === 0) {
         return [];
       }
   
       return [
         new Document({
           pageContent: formatDocumentsAsString(documents),
           metadata: {
             pdf: {
               version: pdfjsLib.version,
               info: meta?.info,
               metadata: meta?.metadata,
               totalPages: parsedPdf.numPages,
             },
           },
         }),
       ];
     }
   }

By using pdfjs-dist, which is designed to work in the browser, you can avoid the issue with the fs module not being found ^[1][2].

To continue talking to Dosu, mention @dosu.

[rossanodr]

any updates on this?
I have the same error when using turbopack