nestjs externaldependencies: none and distributing node_modules ?

[iangregsondev]

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I'm reporting the issue to the correct repository (not related to Angular, AngularCLI or any dependency)

Expected Behavior

I have my nx workspace now with quite a few apps. 2 x angular and 3 x nestjs (1 api and 2 x micrososervices that are used by the api)

I am using angular console to create all of these.

Right now, we have just 1 package.json in the root of the project - this seems to be shared amongst all applications.

Now I know its possible to create an angular library and specify for it to create its own package.json for publishing.

With the apps there seems to be no feature or am i missing something?

I mean there is no way to pass anything to the cli for it to create a package.json.

Is the correct thing here to just create a package.json manually ?

There was a way of bundling externaldependencies for NESTJS but it had issues see #1174 . and #1212

So I assume this is a no go, so my only option is to distribute node_modules - is this correct ?

If this is in case correct then that currently means distributing node_modules in the root which has a lot of dependencies that are mixed from various applications - this means increased size :-(

Any ideas - whats the correct flow now?

Context

Please provide any relevant information about your setup:

• version of Nx used latest
• version of Angular CLI 8

[FrozenPandaz]

You can ship all the node_modules but that isn't ideal. You can create a package.json and distribute that as well but you have to keep the versioning the same there. You can use externalDependencies as I mentioned here nestjs/nest#1706 (comment) but you were saying you were having issues with it. If you can provide a reproduction, I can take a look.

[iangregsondev]

Ok, thanks for the input. I thought that this wasn't working ? I will try and give it a try now - i remember having major issues before - and then a PR was rolled back and we just distributed the node_modules directory.

So if I am understanding you correctly, basically i would set the externalDependencies to a list that are not compatible with webpack bundling - which seems to be nestjs stuff..

These would be installed externally and distributed with the application.

So means I would need to keep 2 x copies of the package.json ? The real package.json for developing a some kind of RELEASE package.json which would just include the nestjs packages so on the CI server it could do a "npm install" and it would only install those dependencies ?

Do we have anything to support the creating of this separate package.json - or am i am missing something ?

Thanks in advance

[iangregsondev]

ng-packagr has something for libraries that strip the package.json of its scripts etc for security issues - maybe i should write something the same that basically goes through it and removes all scripts - it would leave "start" which is how you launch the app. It would also go through and only construct a "dependencies" section of the package.json that appears in the externalDependencies list.

I didn't want to re-invent the wheel :-)

Is this something i should do or is there something there that already does this - anybody using a similar method OR am i am completely misguided here :-)

What do you think @FrozenPandaz

[iangregsondev]

@FrozenPandaz Do you know where we are with this?

With regards to only using certain things as external dependencies. It failed!

I would be very interested to find out from others with NESTJS if they are using for externalDependencies

I tried inserting all NESTJS libraries I could think of and eventually node crashed :-(

I just couldn't get it to work. On a plus side, if anybody is interested, I wrote a small script to create a package.json for production which in effect just extracts items from the externalDependencies from angular.json and creates its own package.json. If its helpful to anybody

const argv = require("yargs").argv
const fs = require("fs")
const path = require("path")

console.log("Opening angular.json")
const angularJson = require(path.join(process.cwd(), "angular.json"))

console.log("Opening original package.json")
const packageJson = require(path.join(process.cwd(), "package.json"))

const productionJsonDirectory = path.join(process.cwd(), `dist/apps/${argv.project}`)
const productionJsonFilename = `${productionJsonDirectory}/prod_package.json`

console.log(`Opening temporary production package.json (${productionJsonFilename})`)
const productionPackageJson = require(productionJsonFilename)

console.log("\nParsing external dependency keys from angular.json that CANNOT be included in the bundle")
const externalDependencies = angularJson.projects[argv.project].architect.build.configurations.production.externalDependencies

console.log("Parsing dependencies from original package.json")
const packageDependencies = packageJson.dependencies

console.log("Constructing dependencies for new package.json (production)")
const externalDepsForPackage = Object.entries(packageDependencies)
  .filter(([key, value]) => {
    return externalDependencies.includes(key)
  })
  .reduce((accum, [k, v]) => {
    accum[k] = v
    return accum
  }, {})

console.log("The following dependencies will be merged into the production package.json\n")
console.log(externalDepsForPackage)

productionPackageJson.dependencies = { ...productionPackageJson.dependencies, ...externalDepsForPackage }

const newProductionPackageJson = path.join(productionJsonDirectory, "/package.json")
console.log(`\nWriting new package.json (${newProductionPackageJson})`)
const json = JSON.stringify(productionPackageJson, null, 2)
fs.writeFileSync(newProductionPackageJson, json, "utf8")

console.log(`Deleting temporary production package.json (${productionJsonFilename})`)
fs.unlinkSync(productionJsonFilename)

console.log("\nDone!!")

I used a standard package.json called prod_package.json - which I use for creating the production packagejson - seee above script.

{
  "name": "@myapp/my-api",
  "version": "0.0.0-development",
  "private": true,
  "license": "MIT",
  "scripts": {
    "start": "node --no-warnings dist/apps/my-api/main.js"
  },
  "dependencies": {
  },
  "devDependencies": {}
}

and added this to the angular.json to ensure it gets copied over

"assets": ["apps/my-api/src/prod_package.json"],

[iangregsondev]

I think the script above would help, it would for sure help me :-) BUT considering that i can't even get the application to run - its a no go.

Right now - I am STILL deploying the FULL contents of node_modules which includes a lot of stuff - even stuff from the UI project - not just the api.

Although the package.json (prod) version getss built and I am able to npm install using this new package.json that "JUST" contains the externalDependencies - node always crashes - will no help at all.

It would be nice to know if there was a strict rule of what to include in externalDependencies.

[godinc0]

Do you have any progress on this issue? We are facing the same problem right now.

We have two nest + one node applications.

We've tryed to use "generate-package-json-webpack-plugin" to generate the package.json, but it only detect half the dependencies. We have to hardcode some in the script. It's not viable.

[kaushiksamanta]

Any progress ??
I'm also facing the same issue.

[vsavkin]

@FrozenPandaz can you provide an update?

[thomas-jakemeyn]

I am facing the exact same problem and the following approach seems to work. It is made for yarn but you can easily make it compatible with npm as well.

1. Install the following development dependencies.

yarn add copy-webpack-plugin generate-package-json-webpack-plugin --dev

2. Create a file webpack.config.js in the root folder of your project with the following content.

const CopyPlugin = require('copy-webpack-plugin');
const GeneratePackageJsonPlugin = require('generate-package-json-webpack-plugin');
const path = require('path');
const packageJson = require('./package.json');

/**
 * Extend the default Webpack configuration from nx / ng.
 */
module.exports = (config, context) => {
  // Extract output path from context
  const {
    options: { outputPath },
  } = context;

  // Install additional plugins
  config.plugins = config.plugins || [];
  config.plugins.push(...extractRelevantNodeModules(outputPath));

  return config;
};

/**
 * This repository only contains one single package.json file that lists the dependencies
 * of all its frontend and backend applications. When a frontend application is built,
 * its external dependencies (aka Node modules) are bundled in the resulting artifact.
 * However, it is not the case for a backend application (for various valid reasons).
 * Installing all the production dependencies would dramatically increase the size of the
 * artifact. Instead, we need to extract the dependencies which are actually used by the
 * backend application. We have implemented this behavior by complementing the default
 * Webpack configuration with additional plugins.
 *
 * @param {String} outputPath The path to the bundle being built
 * @returns {Array} An array of Webpack plugins
 */
function extractRelevantNodeModules(outputPath) {
  return [copyYarnLockFile(outputPath), generatePackageJson()];
}

/**
 * Copy the Yarn lock file to the bundle to make sure that the right dependencies are
 * installed when running `yarn install`.
 *
 * @param {String} outputPath The path to the bundle being built
 * @returns {*} A Webpack plugin
 */
function copyYarnLockFile(outputPath) {
  return new CopyPlugin([{ from: 'yarn.lock', to: path.join(outputPath, 'yarn.lock') }]);
}

/**
 * Generate a package.json file that contains only the dependencies which are actually
 * used in the code.
 *
 * @returns {*} A Webpack plugin
 */
function generatePackageJson() {
  const implicitDeps = [
    'class-transformer',
    'class-validator',
    '@nestjs/platform-express',
    'reflect-metadata',
  ];
  const dependencies = implicitDeps.reduce((acc, dep) => {
    acc[dep] = packageJson.dependencies[dep];
    return acc;
  }, {});
  const basePackageJson = {
    dependencies,
  };
  const pathToPackageJson = path.join(__dirname, 'package.json');
  return new GeneratePackageJsonPlugin(basePackageJson, pathToPackageJson);
}

3. In the file angular.json, set webpackConfig to the file you have just created in the builder options.

      "architect": {
        "build": {
          "builder": "@nrwl/node:build",
          "options": {
            "webpackConfig": "webpack.config.js"

4. If you then build the backend application, the artifact will now contain a file package.json that lists only the dependencies which are actually used in the code. It will also contain the Yarn lock file, which ensures that you install the right versions of these dependencies when running yarn install.

5. If you cannot start the backend application because of a missing dependency, it might be because it is a transitive dependency. Just install it. It might also be that the dependency is implicit (aka not referenced from the code). Just declare it in webpack.config.js.

[bennymeg]

Sounds good!

If you prefer npm over yarn simply replace copyYarnLockFile function with this:

/**
 * Copy the NPM package lock file to the bundle to make sure that the right dependencies are
 * installed when running `npm install`.
 *
 * @param {String} outputPath The path to the bundle being built
 * @returns {*} A Webpack plugin
 */
function copyPackageLockFile(outputPath) {
  return new CopyPlugin([{ from: 'package-lock.json', to: join(outputPath, 'package-lock.json') }]);
}

[WonderPanda]

@thomas-jakemeyn You sir are a gentleman and a scholar. Was pulling my hair out over this for so long.

Am now deploying a NestJS app from an NX repo to AWS lambda shipping only it's own node_modules depdencies (35MB). I can't believe that the only other workarounds for this seem to be copying over the entire node_modules directory for the monorepo. Mine is over 500MB...

[JoeKolba]

@thomas-jakemeyn

I Implemented your solution, however after the project builds and I install the npm modules, it looks like the new package.json doesn't contain all of the dependencies from my original package.json.

[thomas-jakemeyn]

Hello @deviant32,

I am not sure that I do understand your question. Extracting the subset of dependencies that are relevant to a given backend application was exactly the goal of that solution. So you should indeed get fewer dependencies in the final package.json than in the original one...

If you are missing a specific dependency that is required by your application, then it might be because that dependency is not referenced explicitly by the code. You then need to add it into the list of implicit dependencies.

[FrozenPandaz]

Hi, sorry about this.

This was mislabeled as stale. We are testing ways to mark not reproducible issues as stale so that we can focus on actionable items but our initial experiment was too broad and unintentionally labeled this issue as stale.

[johnwest80]

I, too, am struggling with this. Will there be any effort to improve this experience?

[miking-the-viking]

There is a breaking change to the CopyPlugin where I had to change the options object to be:

  return new CopyPlugin({ patterns: [paths] });

(Note the object with patterns:)

[markusheinemann]

I am facing the exact same problem and the following approach seems to work. It is made for yarn but you can easily make it compatible with npm as well.

1. Install the following development dependencies.

yarn add copy-webpack-plugin generate-package-json-webpack-plugin --dev

1. Create a file webpack.config.js in the root folder of your project with the following content.

const CopyPlugin = require('copy-webpack-plugin');
const GeneratePackageJsonPlugin = require('generate-package-json-webpack-plugin');
const path = require('path');
const packageJson = require('./package.json');

/**
 * Extend the default Webpack configuration from nx / ng.
 */
module.exports = (config, context) => {
  // Extract output path from context
  const {
    options: { outputPath },
  } = context;

  // Install additional plugins
  config.plugins = config.plugins || [];
  config.plugins.push(...extractRelevantNodeModules(outputPath));

  return config;
};

/**
 * This repository only contains one single package.json file that lists the dependencies
 * of all its frontend and backend applications. When a frontend application is built,
 * its external dependencies (aka Node modules) are bundled in the resulting artifact.
 * However, it is not the case for a backend application (for various valid reasons).
 * Installing all the production dependencies would dramatically increase the size of the
 * artifact. Instead, we need to extract the dependencies which are actually used by the
 * backend application. We have implemented this behavior by complementing the default
 * Webpack configuration with additional plugins.
 *
 * @param {String} outputPath The path to the bundle being built
 * @returns {Array} An array of Webpack plugins
 */
function extractRelevantNodeModules(outputPath) {
  return [copyYarnLockFile(outputPath), generatePackageJson()];
}

/**
 * Copy the Yarn lock file to the bundle to make sure that the right dependencies are
 * installed when running `yarn install`.
 *
 * @param {String} outputPath The path to the bundle being built
 * @returns {*} A Webpack plugin
 */
function copyYarnLockFile(outputPath) {
  return new CopyPlugin([{ from: 'yarn.lock', to: path.join(outputPath, 'yarn.lock') }]);
}

/**
 * Generate a package.json file that contains only the dependencies which are actually
 * used in the code.
 *
 * @returns {*} A Webpack plugin
 */
function generatePackageJson() {
  const implicitDeps = [
    'class-transformer',
    'class-validator',
    '@nestjs/platform-express',
    'reflect-metadata',
  ];
  const dependencies = implicitDeps.reduce((acc, dep) => {
    acc[dep] = packageJson.dependencies[dep];
    return acc;
  }, {});
  const basePackageJson = {
    dependencies,
  };
  const pathToPackageJson = path.join(__dirname, 'package.json');
  return new GeneratePackageJsonPlugin(basePackageJson, pathToPackageJson);
}

1. In the file angular.json, set webpackConfig to the file you have just created in the builder options.

      "architect": {
        "build": {
          "builder": "@nrwl/node:build",
          "options": {
            "webpackConfig": "webpack.config.js"

1. If you then build the backend application, the artifact will now contain a file package.json that lists only the dependencies which are actually used in the code. It will also contain the Yarn lock file, which ensures that you install the right versions of these dependencies when running yarn install.
2. If you cannot start the backend application because of a missing dependency, it might be because it is a transitive dependency. Just install it. It might also be that the dependency is implicit (aka not referenced from the code). Just declare it in webpack.config.js.

Thank you! For me it was also necessary to adjust the package.json path in line 4 of the webpack.config.js

[Cammisuli]

Hey all, I'm just letting everyone know that I added support to generate a package.json based on whatever dependencies are currently being used in the node app. (This is similar to how buildable Nx libraries add their dependencies to the package.json as well.)

This would significantly help with deploying with docker because we'll be able to install the dependencies in the container.

Although, there's some peer dependencies needed with nestjs that would need to be installed within the container. So a layer like this should be done:

RUN npm install reflect-metadata tslib rxjs @nestjs/platform-express

So keeping externalDependencies as the default (i.e, "none"), and adding generatePackageJson, docker images can be made easier.

[hakimio]

@Cammisuli this option should be enabled by default for NestJS production builds.

[tmtron]

@Cammisuli this option should be enabled by default for NestJS production builds.

I disagree. for following reasons:

1. The default should be to get a working production build. With this option activated, the build may not work without further tweaks: e.g. may need to specify implicit dependencies
2. If someone wants to have a smaller build, they will anyway investigate (some may even read the docs, or search the issues) and may chose to enable this option - or to use a different approach: e.g. use the vercel/ncc compiler (see also [Feature request] ncc builder (for node apps) #5005) which can reduce the size even more.

[hakimio]

@tmtron For me by default NestJS production build didn't work at all. "externalDependencies": "none" was not set in angular.json. So, it was not including external depencies in the main.js. Not a good UX. Also, even when set, it produced loads of warnings and errors.
Anyway, I prefer generatePackageJson to "externalDependencies": "none".

[hakimio]

@tmtron Just for reference, here is output I get with "externalDependencies": "none" when using fastify instead of express (the app itself works perfectly well):

WARNING in ./node_modules/fastify-formbody/node_modules/fastify-plugin/plugin.js 81:22-29
Critical dependency: require function is used in a way in which dependencies cannot be statically extracted

WARNING in ./node_modules/@nestjs/core/helpers/optional-require.js 6:39-59
Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/@nestjs/common/utils/load-package.util.js 9:39-59
Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/fastify-formbody/node_modules/fastify-plugin/plugin.js 115:35-51
Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/@nestjs/core/helpers/load-adapter.js 9:39-63
Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/fast-json-stringify/index.js
Module not found: Error: Can't resolve 'long' in 'C:\Users\PC1\PhpstormProjects\fenerum\node_modules\fast-json-stringify'

WARNING in ./node_modules/pino/lib/tools.js
Module not found: Error: Can't resolve 'pino-pretty' in 'C:\Users\PC1\PhpstormProjects\fenerum\node_modules\pino\lib'

WARNING in license-webpack-plugin: could not find any license file for passport-headerapikey. Use the licenseTextOverrides option to add the license text if desired.

WARNING in license-webpack-plugin: could not find any license type for libphonenumber-js-min in its package.json

WARNING in license-webpack-plugin: could not find any license file for libphonenumber-js-min. Use the licenseTextOverrides option to add the license text if desired.

WARNING in license-webpack-plugin: could not find any license type for pause in its package.json

WARNING in license-webpack-plugin: could not find any license file for pause. Use the licenseTextOverrides option to add the license text if desired.

WARNING in license-webpack-plugin: could not find any license file for abstract-logging. Use the licenseTextOverrides option to add the license text if desired.

ERROR in ./node_modules/@nestjs/core/nest-application.js
Module not found: Error: Can't resolve '@nestjs/microservices' in 'C:\Users\PC1\PhpstormProjects\fenerum\node_modules\@nestjs\core'

ERROR in ./node_modules/@nestjs/core/nest-factory.js
Module not found: Error: Can't resolve '@nestjs/microservices' in 'C:\Users\PC1\PhpstormProjects\fenerum\node_modules\@nestjs\core'

ERROR in ./node_modules/@nestjs/core/nest-application.js
Module not found: Error: Can't resolve '@nestjs/microservices/microservices-module' in 'C:\Users\PC1\PhpstormProjects\fenerum\node_modules\@nestjs\core'

ERROR in ./node_modules/@nestjs/core/nest-factory.js
Module not found: Error: Can't resolve '@nestjs/platform-express' in 'C:\Users\PC1\PhpstormProjects\fenerum\node_modules\@nestjs\core'

ERROR in ./node_modules/@nestjs/core/nest-application.js
Module not found: Error: Can't resolve '@nestjs/websockets/socket-module' in 'C:\Users\PC1\PhpstormProjects\fenerum\node_modules\@nestjs\core'

ERROR in ./node_modules/@nestjs/common/cache/cache.providers.js
Module not found: Error: Can't resolve 'cache-manager' in 'C:\Users\PC1\PhpstormProjects\fenerum\node_modules\@nestjs\common\cache'

ERROR in ./node_modules/@nestjs/mapped-types/dist/type-helpers.utils.js
Module not found: Error: Can't resolve 'class-transformer/storage' in 'C:\Users\PC1\PhpstormProjects\fenerum\node_modules\@nestjs\mapped-types\dist'

ERROR in ./node_modules/@nestjs/swagger/dist/swagger-module.js
Module not found: Error: Can't resolve 'swagger-ui-express' in 'C:\Users\PC1\PhpstormProjects\fenerum\node_modules\@nestjs\swagger\dist'
error Command failed with exit code 1.

[hakimio]

1. The default should be to get a working production build. With this option activated, the build may not work without further tweaks: e.g. may need to specify implicit dependencies

You have to manually specify "implicit dependencies" when using the script provided by @thomas-jakemeyn but there is no need to do that when using generatePackageJson option provided by @Cammisuli . It just works.

Anyway, since "externalDependencies": "none" option was reverted in #1212, we now have broken production build by default which doesn't make any sense at all.

EDIT: here is what the creator of NestJS has to say about it:

In many real-world scenarios (depending on what libraries are being used), you should not bundle Node.js applications (not only NestJS applications) with all dependencies (external packages located in the node_modules folder). Although this may make your docker images smaller (due to tree-shaking), somewhat reduce the memory consumption, slightly increase the bootstrap time (which is particularly useful in the serverless environments), it won't work in combination with many popular libraries commonly used in the ecosystem.
nestjs/nest#1706 (comment)

[adam-arold]

Hey all, I'm just letting everyone know that I added support to generate a package.json based on whatever dependencies are currently being used in the node app. (This is similar to how buildable Nx libraries add their dependencies to the package.json as well.)

This would significantly help with deploying with docker because we'll be able to install the dependencies in the container.

Although, there's some peer dependencies needed with nestjs that would need to be installed within the container. So a layer like this should be done:

RUN npm install reflect-metadata tslib rxjs @nestjs/platform-express

So keeping externalDependencies as the default (i.e, "none"), and adding generatePackageJson, docker images can be made easier.

How can I make this work if I have all my dependencies in a single file (package.json in the root) and I'd like to prune my dependencies for only a single package (my backend code)? I tried generatePackageJson but it generates this file into the dist folder.

[hakimio]

@adam-arold you can use depcheck to remove unused dependencies. Here you can find a nodejs script for inspiration.

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.