Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[🐛 Bug]: pg does not work #605

Open
1 task
janpio opened this issue Dec 20, 2023 · 24 comments
Open
1 task

[🐛 Bug]: pg does not work #605

janpio opened this issue Dec 20, 2023 · 24 comments
Labels
bug Something isn't working

Comments

@janpio
Copy link

janpio commented Dec 20, 2023

next-on-pages environment related information

System:
        Platform: linux
        Arch: x64
        Version: #202312122204 SMP PREEMPT_DYNAMIC Tue Dec 12 22:43:56 UTC 2023
        CPU: (16) x64 AMD EPYC 7B13
        Memory: 63 GB
        Shell: Unknown
Package Manager Used: npm (10.2.3)

Relevant Packages:
        @cloudflare/next-on-pages: 1.8.2
        vercel: 33.0.0
        next: 14.0.4

Description

pg has support for Cloudflare Workers and Pages.
This project makes it possible to run Next.js projects on Cloudflare Pages.
Unfortunately something gets lost along the way, so that pg triggers webpack errors during build and dev:

$ npm run build

> [email protected] build
> next build

   ▲ Next.js 14.0.4

Failed to compile.

./node_modules/pg-connection-string/index.js:76:69
Module not found: Can't resolve 'fs'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/pg/lib/connection-parameters.js
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./app/api/pg/route.js
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=private-next-app-dir%2Fapi%2Fpg%2Froute.js&page=%2Fapi%2Fpg%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcGclMkZyb3V0ZSZwYWdlPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZwZyUyRnJvdXRlLmpzJmFwcERpcj0lMkZ3b3Jrc3BhY2UlMkZuZXh0LXBnJTJGYXBwJmFwcFBhdGhzPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZiYXNlUGF0aD0mYXNzZXRQcmVmaXg9Jm5leHRDb25maWdPdXRwdXQ9JnByZWZlcnJlZFJlZ2lvbj0mbWlkZGxld2FyZUNvbmZpZz1lMzAlM0Qh&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!

./node_modules/pgpass/lib/helper.js:3:11
Module not found: Can't resolve 'path'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/pgpass/lib/index.js
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./app/api/pg/route.js
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=private-next-app-dir%2Fapi%2Fpg%2Froute.js&page=%2Fapi%2Fpg%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcGclMkZyb3V0ZSZwYWdlPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZwZyUyRnJvdXRlLmpzJmFwcERpcj0lMkZ3b3Jrc3BhY2UlMkZuZXh0LXBnJTJGYXBwJmFwcFBhdGhzPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZiYXNlUGF0aD0mYXNzZXRQcmVmaXg9Jm5leHRDb25maWdPdXRwdXQ9JnByZWZlcnJlZFJlZ2lvbj0mbWlkZGxld2FyZUNvbmZpZz1lMzAlM0Qh&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!

./node_modules/pgpass/lib/helper.js:4:13
Module not found: Can't resolve 'stream'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/pgpass/lib/index.js
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./app/api/pg/route.js
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=private-next-app-dir%2Fapi%2Fpg%2Froute.js&page=%2Fapi%2Fpg%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcGclMkZyb3V0ZSZwYWdlPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZwZyUyRnJvdXRlLmpzJmFwcERpcj0lMkZ3b3Jrc3BhY2UlMkZuZXh0LXBnJTJGYXBwJmFwcFBhdGhzPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZiYXNlUGF0aD0mYXNzZXRQcmVmaXg9Jm5leHRDb25maWdPdXRwdXQ9JnByZWZlcnJlZFJlZ2lvbj0mbWlkZGxld2FyZUNvbmZpZz1lMzAlM0Qh&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!

./node_modules/pgpass/lib/index.js:3:11
Module not found: Can't resolve 'path'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./app/api/pg/route.js
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=private-next-app-dir%2Fapi%2Fpg%2Froute.js&page=%2Fapi%2Fpg%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcGclMkZyb3V0ZSZwYWdlPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZwZyUyRnJvdXRlLmpzJmFwcERpcj0lMkZ3b3Jrc3BhY2UlMkZuZXh0LXBnJTJGYXBwJmFwcFBhdGhzPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZiYXNlUGF0aD0mYXNzZXRQcmVmaXg9Jm5leHRDb25maWdPdXRwdXQ9JnByZWZlcnJlZFJlZ2lvbj0mbWlkZGxld2FyZUNvbmZpZz1lMzAlM0Qh&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!

./node_modules/pgpass/lib/index.js:4:9
Module not found: Can't resolve 'fs'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./app/api/pg/route.js
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=private-next-app-dir%2Fapi%2Fpg%2Froute.js&page=%2Fapi%2Fpg%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcGclMkZyb3V0ZSZwYWdlPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZwZyUyRnJvdXRlLmpzJmFwcERpcj0lMkZ3b3Jrc3BhY2UlMkZuZXh0LXBnJTJGYXBwJmFwcFBhdGhzPSUyRmFwaSUyRnBnJTJGcm91dGUmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZiYXNlUGF0aD0mYXNzZXRQcmVmaXg9Jm5leHRDb25maWdPdXRwdXQ9JnByZWZlcnJlZFJlZ2lvbj0mbWlkZGxld2FyZUNvbmZpZz1lMzAlM0Qh&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!


> Build failed because of webpack errors

I theorize that the problem is because pg was made to use a adapter version of some code when running in workerd. But during the Next.js build this is not set, so it thinks it is running in a normal Node environment and includes the wrong code - which then would not work in edge-light which the Next.js bundler assumes for edge functions - and hence fails.

Reproduction

https://github.com/janpio/cloudflare-next-prisma/tree/plain-pg

Pages Deployment Method

Direct Upload (wrangler pages publish or the @cloudflare/pages-action GitHub Action)

Pages Deployment ID

No response

Additional Information

No response

Would you like to help?

  • Would you like to help fixing this bug?
@janpio janpio added the bug Something isn't working label Dec 20, 2023
@VanThanh12195
Copy link

VanThanh12195 commented Mar 29, 2024

Hello is there any workaround for this issue? I try to use pg and prisma on cloudflare next-on-pages with nextjs application. Observe similar issue.

@james-elicx
Copy link
Contributor

Hello is there any workaround for this issue? I try to use pg and prisma on cloudflare next-on-pages with nextjs application. Observe similar issue.

This build error comes from Next.js and is something we have no control over.

@webdeb
Copy link

webdeb commented May 8, 2024

@janpio / @VanThanh12195 have you found a workaround?

@webdeb
Copy link

webdeb commented May 8, 2024

@james-elicx can you provide tips what a developer can/should do fix it?
next-on-pages allows to develop next inside wranger context, if its ok to call it like this.
pg is supported by cloudflare, and those errors are coming from loading pg dependencies inside the wrangler context.
I mean, you guys are probably dealing with it on the cloudflare side, let it make work on the dev side, or show us how, please.

@james-elicx
Copy link
Contributor

@james-elicx can you provide tips what a developer can/should do fix it? next-on-pages allows to develop next inside wranger context, if its ok to call it like this. pg is supported by cloudflare, and those errors are coming from loading pg dependencies inside the wrangler context. I mean, you guys are probably dealing with it on the cloudflare side, let it make work on the dev side, or show us how, please.

Module not found: Can't resolve is a Next.js build error because they throw errors when libraries import Node.js builtins that aren't supported according to their edge runtime implementation. It has nothing to do with us or how our dev mode works, and we can't stop it from happening.

@webdeb
Copy link

webdeb commented May 9, 2024

Module not found: Can't resolve is a Next.js build error because they throw errors when libraries import Node.js builtins that aren't supported according to their edge runtime implementation. It has nothing to do with us or how our dev mode works, and we can't stop it from happening.

@james-elicx I understand that the build error is coming from nextjs. However, lets focus on a possible solution or a workaround to use pg in a next-on-pages project.

Or just declare that next-on-pages does not support pg and call it a day..

@huhan-123
Copy link

Has anyone solved this problem?

@janpio
Copy link
Author

janpio commented Jun 7, 2024

There might be a way to make it work by configuring Next.js and patching pg. Super hacky, but it shows there is a way forward: brianc/node-postgres#3206 (comment)

@protonys
Copy link

protonys commented Jun 26, 2024

Not working pg connection, MariaDB connection and MySQL connection in to middleware.ts
How then use middleware.ts with db connections? May be use pg connection in layout dashboard for auth?

@lancerace
Copy link

lancerace commented Sep 23, 2024

facing the same exact problem due to wanting to use cloudflare and vercel deployment with edge runtime.

Below are the summary i am facing,

  1. The following code could work with next build and @cloudflare/next-on-pages deploy
import { Prisma, PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();
export const runtime = 'edge';

//schema.prisma
generator client {
  provider        = "prisma-client-js"
  previewFeatures = []
}

in a cloudflare preview deployed environment. If a page call an api that requires prisma, the following error occurs
image

Next, i replace it with prisma/adapter-pg in attempt to resolve Next edge runtime

  1. The following code fails at next build.
import { Pool } from 'pg'
import { PrismaPg } from '@prisma/adapter-pg'
import { Prisma, PrismaClient } from '@prisma/client';
const connectionString = `${process.env.DATABASE_URL}`
const pool = new Pool({ connectionString })
const adapter = new PrismaPg(pool)
const prisma = new PrismaClient({ adapter } as any)
export const runtime = 'edge';


//schema.prisma
generator client {
  provider        = "prisma-client-js"
  previewFeatures = ["driverAdapters"]
}
image

wonder if anyone resolved this

@irvinebroque
Copy link

If you're trying to get pg working — it's worth experimenting with @opennextjs/cloudflare, which lets you use a much wider set of Node.js APIs, which DB drivers like pg typically depend on. (The edge "runtime" in the Next.js compiler intentionally constrains which Node.js APIs your app can use to a much narrower set than what Cloudflare Workers supports)

https://opennext.js.org/cloudflare

@maltekuehl
Copy link

maltekuehl commented Sep 29, 2024

If you're trying to get pg working — it's worth experimenting with @opennextjs/cloudflare, which lets you use a much wider set of Node.js APIs, which DB drivers like pg typically depend on. (The edge "runtime" in the Next.js compiler intentionally constrains which Node.js APIs your app can use to a much narrower set than what Cloudflare Workers supports)

https://opennext.js.org/cloudflare

Has anybody yet been able to get the Cloudflare version of OpenNext to work with prisma?

The following prisma configuration works fine with next dev, builds without errors with npx cloudflare but fails when using wrangler dev or wrangler deploy:

import { Pool } from "pg";
import { PrismaPg } from "@prisma/adapter-pg";
import { PrismaClient } from "@prisma/client";

const pool = new Pool({ connectionString: connectionString, max: 1 })
const adapter = new PrismaPg(pool)
const prisma = new PrismaClient({ adapter })

Error messages:

Could not resolve "#main-entry-point"

    .worker-next/.next/standalone/node_modules/.prisma/client/default.js:1:30:
      1 │ module.exports = { ...require('#main-entry-point') }
        ╵                               ~~~~~~~~~~~~~~~~~~~

  The module "./wasm.js" was not found on the file system:

    .worker-next/.next/standalone/node_modules/.prisma/client/package.json:108:19:
      108 │         "workerd": "./wasm.js",
          ╵                    ~~~~~~~~~~~

and

ould not resolve "pg-cloudflare"

    .worker-next/.next/standalone/node_modules/pg/lib/stream.js:10:41:
      10 │     const { CloudflareSocket } = require('pg-cloudflare')
         ╵                                          ~~~~~~~~~~~~~~~

  The module "./dist/index.js" was not found on the file system:

    .worker-next/.next/standalone/node_modules/pg-cloudflare/package.json:13:15:
      13 │     "workerd": "./dist/index.js",
         ╵                ~~~~~~~~~~~~~~~~~

  You can mark the path "pg-cloudflare" as external to exclude it from the bundle, which will remove this error. You can also surround this "require" call with a try/catch block to handle this failure at run-time instead of bundle-time.

Adding pg-cloudflare as a dependency does also not seem to help.
Wondering if anybody has managed to set it up successfully.

@ianthpun
Copy link

having same issue as above ^

@ianthpun
Copy link

@maltekuehl were you able to figure this out?

@irvinebroque do you have an example of using @opennextjs/cloudflare with prisma to get it working?

@karthikjn01
Copy link

Btw - not an expert by any means, but I had managed to get supabase (postgres) working with the neon driver from prisma, not sure if that's helpful, but might be worth a shot until pg is working?

@ianthpun
Copy link

@karthikjn01 could you give me an example of how you set it up?

@karthikjn01
Copy link

Unfortunately don't have the code on hand. Have a look here: https://www.prisma.io/docs/orm/overview/databases/neon

@ianthpun
Copy link

@karthikjn01 i just tried it using the neon drivers and got this error

image

same issue as above. What does your package.json look like?

@karthikjn01
Copy link

Ah yeah, I was running into that on opennext - try next-on-pages, I believe that's where it was working.

@ianthpun
Copy link

hi yes thank you @karthikjn01 i got it working with next-on-pages, thank you. It looks like it sstill bugged on opennext

@ianthpun
Copy link

I seem to have gotten it compiling and running with prisma pool, but ther seems to be approx 1 second overhead for each function call made, as I think the setting up of the PrismaClient with the adapter takes around a second each time to instantiate. @karthikjn01 have you had any of those issues? The moment i moved it back to a singleton (to test this theory) it worked fine

@yzqzy
Copy link

yzqzy commented Nov 16, 2024

If you're trying to get pg working — it's worth experimenting with @opennextjs/cloudflare, which lets you use a much wider set of Node.js APIs, which DB drivers like pg typically depend on. (The edge "runtime" in the Next.js compiler intentionally constrains which Node.js APIs your app can use to a much narrower set than what Cloudflare Workers supports)
https://opennext.js.org/cloudflare

Has anybody yet been able to get the Cloudflare version of OpenNext to work with prisma?有人能够让 OpenNext 的 Cloudflare 版本与 prisma 一起使用吗?

The following prisma configuration works fine with next dev, builds without errors with npx cloudflare but fails when using wrangler dev or wrangler deploy:以下 prisma 配置在next dev中运行良好,在npx cloudflare中构建时没有错误,但在使用wrangler devwrangler deploy时失败:

import { Pool } from "pg";
import { PrismaPg } from "@prisma/adapter-pg";
import { PrismaClient } from "@prisma/client";

const pool = new Pool({ connectionString: connectionString, max: 1 })
const adapter = new PrismaPg(pool)
const prisma = new PrismaClient({ adapter })

Error messages:

Could not resolve "#main-entry-point"

    .worker-next/.next/standalone/node_modules/.prisma/client/default.js:1:30:
      1 │ module.exports = { ...require('#main-entry-point') }
        ╵                               ~~~~~~~~~~~~~~~~~~~

  The module "./wasm.js" was not found on the file system:

    .worker-next/.next/standalone/node_modules/.prisma/client/package.json:108:19:
      108 │         "workerd": "./wasm.js",
          ╵                    ~~~~~~~~~~~

and

ould not resolve "pg-cloudflare"

    .worker-next/.next/standalone/node_modules/pg/lib/stream.js:10:41:
      10 │     const { CloudflareSocket } = require('pg-cloudflare')
         ╵                                          ~~~~~~~~~~~~~~~

  The module "./dist/index.js" was not found on the file system:

    .worker-next/.next/standalone/node_modules/pg-cloudflare/package.json:13:15:
      13 │     "workerd": "./dist/index.js",
         ╵                ~~~~~~~~~~~~~~~~~

  You can mark the path "pg-cloudflare" as external to exclude it from the bundle, which will remove this error. You can also surround this "require" call with a try/catch block to handle this failure at run-time instead of bundle-time.

Adding pg-cloudflare as a dependency does also not seem to help.添加 pg-cloudflare 作为依赖项似乎也没有帮助。 Wondering if anybody has managed to set it up successfully.想知道是否有人成功设置过。

I also encountered this problem and couldn't work together.

@yzqzy
Copy link

yzqzy commented Nov 16, 2024

I seem to have gotten it compiling and running with prisma pool, but ther seems to be approx 1 second overhead for each function call made, as I think the setting up of the PrismaClient with the adapter takes around a second each time to instantiate. @karthikjn01 have you had any of those issues? The moment i moved it back to a singleton (to test this theory) it worked fine我似乎已经使用 prisma 池编译和运行它,但每次函数调用似乎大约需要 1 秒的开销,因为我认为使用适配器设置 PrismaClient 每次实例化大约需要一秒钟。 @karthikjn01你有遇到过这些问题吗?当我将它移回单例(以测试这个理论)时,它工作得很好

Could not resolve "#main-entry-point" Have you solved this issue?

@foreleven
Copy link

foreleven commented Dec 4, 2024

@ianthpun @yzqzy I have solved this issue. The reason for this error is that the cloudflare command forces the standalone mode of the next build to be enabled when it is executed. Primsa runtime related files are not copied to the standalone directory.
Change the command to cloudflare && node ./scripts/prisma-runtime.js && wrangler deploy

@prisma/client/runtime Not all files below need to be copied, but I haven't tested which ones are required, so it works fine for now

const fs = require("fs");
const path = require("path");

// Base destination directory
const destNodeModules = path.join(
  ".worker-next",
  ".next",
  "standalone",
  "node_modules"
);

const sourceNodeModules = path.join("node_modules");

const files = [
  ".prisma/client/wasm.js",
  ".prisma/client/wasm-worker-loader.mjs",
  ".prisma/client/query_engine_bg.wasm",
  ".prisma/client/query_engine_bg.js",
  "@prisma/client/runtime",
];

// Copy runtime directory recursively
function copyRecursive(src, dest) {
  const exists = fs.existsSync(src);
  const stats = exists && fs.statSync(src);
  const isDirectory = exists && stats.isDirectory();

  if (isDirectory) {
    fs.mkdirSync(dest, { recursive: true });
    fs.readdirSync(src).forEach(function (childItemName) {
      copyRecursive(
        path.join(src, childItemName),
        path.join(dest, childItemName)
      );
    });
  } else {
    try {
      fs.copyFileSync(src, dest);
    } catch (error) {
      console.warn(error);
    }
  }
}

try {
  // Copy additional files
  for (const file of files) {
    copyRecursive(
      path.join(sourceNodeModules, file),
      path.join(destNodeModules, file)
    );
  }

  console.log(
    "Successfully copied Prisma runtime files and additional WASM files"
  );
} catch (err) {
  console.error("Error copying Prisma files:", err);
  process.exit(1);
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests