Memory leak since miggration from node 20 to bun 1.1.43 of a nextjs website #16339
Labels
Comments
erwanriou
changed the title
|
I tried a bunch of things without success. The only thing i didn't tryed was to switch all the |
|
I believe axios will use bun's fetch implementation under the hood so switching to axios might not give you any advantages in that regard. |
|
Is there a chance you could email us a link to the heap snapshot - [email protected]? Can you show a screenshot of what else is in there (without ArrayBuffer expanded)? 315 MB of ArrayBuffer is sort of a lot, but it doesn't fit with the 10-ish gigabytes you're seeing there. Something is missing from the heap snapshot, unless there are a large number of similarly-sized objects types in there. Can you list out what modules are imported so we can get a sense of what packages are in use and might be causing this? Something like this should do it: console.log([...new Set([...Loader.registry.keys(), ...Object.keys(require.cache)])].sort())Could you also run this when it's using a lot of memory? If a large number of objects show up in console.log(require("bun:jsc").heapStats()) |
|
@Jarred-Sumner I will email you the heap snapshot but i am not sure if its a relevant one as its a one made from my local environment. I will add the console logs that you suggested on the staging environment and post the logs here. There it won't scale until 10GO but just spamming a few urls should already increase the memory by 10 fold and that should be already enough to investigate. Regarding the lib i am using, here are the logs (i had to stringify it sorry otherwise it would not paint them all) ["(rsc)/./node_modules/@archsplace/archsplace_client_common/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/constants/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/countries/data.json","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/countries/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/capitalize/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/formatContent/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/formatKebabToLower/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/formatName/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/formatPhone/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/formatPrice/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/formatStringToSlug/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/formatToUpperSnake/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/formatUpperSnakeToLower/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/getCookie/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/getCookieSearch/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/getRelativeTime/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/imageRender/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/isClean/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/isEmpty/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/isRedirect/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/isValidUrl/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/objectDeepFlat/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/objectFlattener/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/parseDomain/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/priceBuilder/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/removeEmpty/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/removeFalsy/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/removePluralSuffix/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/helpers/urlBuilder/index.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/tracking/factories/redirect.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/tracking/factories/trackingDataFormatter.js","(rsc)/./node_modules/@archsplace/archsplace_client_common/src/tracking/index.js","(rsc)/./node_modules/@formatjs/fast-memoize/lib/index.js","(rsc)/./node_modules/@formatjs/icu-messageformat-parser/lib/date-time-pattern-generator.js","(rsc)/./node_modules/@formatjs/icu-messageformat-parser/lib/error.js","(rsc)/./node_modules/@formatjs/icu-messageformat-parser/lib/index.js","(rsc)/./node_modules/@formatjs/icu-messageformat-parser/lib/parser.js","(rsc)/./node_modules/@formatjs/icu-messageformat-parser/lib/regex.generated.js","(rsc)/./node_modules/@formatjs/icu-messageformat-parser/lib/time-data.generated.js","(rsc)/./node_modules/@formatjs/icu-messageformat-parser/lib/types.js","(rsc)/./node_modules/@formatjs/icu-messageformat-parser/node_modules/tslib/tslib.es6.mjs","(rsc)/./node_modules/@formatjs/icu-skeleton-parser/lib/date-time.js","(rsc)/./node_modules/@formatjs/icu-skeleton-parser/lib/index.js","(rsc)/./node_modules/@formatjs/icu-skeleton-parser/lib/number.js","(rsc)/./node_modules/@formatjs/icu-skeleton-parser/lib/regex.generated.js","(rsc)/./node_modules/@formatjs/icu-skeleton-parser/node_modules/tslib/tslib.es6.mjs","(rsc)/./node_modules/@swc/helpers/esm/_interop_require_default.js","(rsc)/./node_modules/decode-uri-component/index.js","(rsc)/./node_modules/filter-obj/index.js","(rsc)/./node_modules/intl-messageformat/lib/index.js","(rsc)/./node_modules/intl-messageformat/lib/src/core.js","(rsc)/./node_modules/intl-messageformat/lib/src/error.js","(rsc)/./node_modules/intl-messageformat/lib/src/formatters.js","(rsc)/./node_modules/intl-messageformat/node_modules/tslib/tslib.es6.mjs","(rsc)/./node_modules/next-intl/dist/development/routing.js","(rsc)/./node_modules/next-intl/dist/development/routing/defineRouting.js","(rsc)/./node_modules/next-intl/dist/esm/_virtual/_rollupPluginBabelHelpers.js","(rsc)/./node_modules/next-intl/dist/esm/navigation/react-server/createNavigation.js","(rsc)/./node_modules/next-intl/dist/esm/navigation/react-server/getServerLocale.js","(rsc)/./node_modules/next-intl/dist/esm/navigation/shared/BaseLink.js","(rsc)/./node_modules/next-intl/dist/esm/navigation/shared/createSharedNavigationFns.js","(rsc)/./node_modules/next-intl/dist/esm/navigation/shared/utils.js","(rsc)/./node_modules/next-intl/dist/esm/react-server/NextIntlClientProviderServer.js","(rsc)/./node_modules/next-intl/dist/esm/react-server/getTranslator.js","(rsc)/./node_modules/next-intl/dist/esm/react-server/useConfig.js","(rsc)/./node_modules/next-intl/dist/esm/react-server/useTranslations.js","(rsc)/./node_modules/next-intl/dist/esm/routing/config.js","(rsc)/./node_modules/next-intl/dist/esm/server/react-server/RequestLocale.js","(rsc)/./node_modules/next-intl/dist/esm/server/react-server/RequestLocaleCache.js","(rsc)/./node_modules/next-intl/dist/esm/server/react-server/RequestLocaleLegacy.js","(rsc)/./node_modules/next-intl/dist/esm/server/react-server/getConfig.js","(rsc)/./node_modules/next-intl/dist/esm/server/react-server/getLocale.js","(rsc)/./node_modules/next-intl/dist/esm/server/react-server/getMessages.js","(rsc)/./node_modules/next-intl/dist/esm/server/react-server/getNow.js","(rsc)/./node_modules/next-intl/dist/esm/server/react-server/getRequestConfig.js","(rsc)/./node_modules/next-intl/dist/esm/server/react-server/getTimeZone.js","(rsc)/./node_modules/next-intl/dist/esm/server/react-server/getTranslations.js","(rsc)/./node_modules/next-intl/dist/esm/shared/NextIntlClientProvider.js","(rsc)/./node_modules/next-intl/dist/esm/shared/constants.js","(rsc)/./node_modules/next-intl/dist/esm/shared/utils.js","(rsc)/./node_modules/next/dist/api/app-dynamic.js","(rsc)/./node_modules/next/dist/api/headers.js","(rsc)/./node_modules/next/dist/api/navigation.react-server.js","(rsc)/./node_modules/next/dist/build/output/log.js","(rsc)/./node_modules/next/dist/build/webpack/loaders/next-app-loader.js?name=app%2F%5Blocale%5D%2Fpage&page=%2F%5Blocale%5D%2Fpage&appPaths=%2F%5Blocale%5D%2Fpage&pagePath=private-next-app-dir%2F%5Blocale%5D%2Fpage.js&appDir=%2Fapp%2Fsrc%2Fapp&pageExtensions=tsx&pageExtensions=ts&pageExtensions=jsx&pageExtensions=js&rootDir=%2Fapp&isDev=true&tsconfigPath=tsconfig.json&basePath=&assetPrefix=&nextConfigOutput=standalone&preferredRegion=&middlewareConfig=e30%3D!","(rsc)/./node_modules/next/dist/build/webpack/loaders/next-flight-loader/module-proxy.js","(rsc)/./node_modules/next/dist/client/components/app-router.js","(rsc)/./node_modules/next/dist/client/components/client-page.js","(rsc)/./node_modules/next/dist/client/components/draft-mode.js","(rsc)/./node_modules/next/dist/client/components/error-boundary.js","(rsc)/./node_modules/next/dist/client/components/headers.js","(rsc)/./node_modules/next/dist/client/components/hooks-server-context.js","(rsc)/./node_modules/next/dist/client/components/layout-router.js","(rsc)/./node_modules/next/dist/client/components/navigation.react-server.js","(rsc)/./node_modules/next/dist/client/components/not-found-boundary.js","(rsc)/./node_modules/next/dist/client/components/not-found-error.js","(rsc)/./node_modules/next/dist/client/components/not-found.js","(rsc)/./node_modules/next/dist/client/components/redirect-status-code.js","(rsc)/./node_modules/next/dist/client/components/redirect.js","(rsc)/./node_modules/next/dist/client/components/render-from-template-context.js","(rsc)/./node_modules/next/dist/client/components/search-params.js","(rsc)/./node_modules/next/dist/client/components/static-generation-bailout.js","(rsc)/./node_modules/next/dist/compiled/@edge-runtime/cookies/index.js","(rsc)/./node_modules/next/dist/compiled/@opentelemetry/api/index.js","(rsc)/./node_modules/next/dist/lib/constants.js","(rsc)/./node_modules/next/dist/lib/picocolors.js","(rsc)/./node_modules/next/dist/lib/url.js","(rsc)/./node_modules/next/dist/server/app-render/dynamic-rendering.js","(rsc)/./node_modules/next/dist/server/app-render/entry-base.js","(rsc)/./node_modules/next/dist/server/app-render/rsc/postpone.js","(rsc)/./node_modules/next/dist/server/app-render/rsc/preloads.js","(rsc)/./node_modules/next/dist/server/app-render/rsc/taint.js","(rsc)/./node_modules/next/dist/server/future/route-kind.js","(rsc)/./node_modules/next/dist/server/future/route-modules/app-page/vendored/rsc/react-dom.js","(rsc)/./node_modules/next/dist/server/future/route-modules/app-page/vendored/rsc/react-jsx-dev-runtime.js","(rsc)/./node_modules/next/dist/server/future/route-modules/app-page/vendored/rsc/react-jsx-runtime.js","(rsc)/./node_modules/next/dist/server/future/route-modules/app-page/vendored/rsc/react-server-dom-webpack-server-edge.js","(rsc)/./node_modules/next/dist/server/future/route-modules/app-page/vendored/rsc/react.js","(rsc)/./node_modules/next/dist/server/lib/clone-response.js","(rsc)/./node_modules/next/dist/server/lib/dedupe-fetch.js","(rsc)/./node_modules/next/dist/server/lib/patch-fetch.js","(rsc)/./node_modules/next/dist/server/lib/trace/constants.js","(rsc)/./node_modules/next/dist/server/lib/trace/tracer.js","(rsc)/./node_modules/next/dist/server/web/spec-extension/adapters/headers.js","(rsc)/./node_modules/next/dist/server/web/spec-extension/adapters/reflect.js","(rsc)/./node_modules/next/dist/server/web/spec-extension/adapters/request-cookies.js","(rsc)/./node_modules/next/dist/server/web/spec-extension/cookies.js","(rsc)/./node_modules/next/dist/shared/lib/app-dynamic.js","(rsc)/./node_modules/next/dist/shared/lib/lazy-dynamic/dynamic-bailout-to-csr.js","(rsc)/./node_modules/next/dist/shared/lib/lazy-dynamic/loadable.js","(rsc)/./node_modules/next/dist/shared/lib/lazy-dynamic/preload-css.js","(rsc)/./node_modules/next/font/google/target.css?{\"path\":\"src/app/[locale]/layout.js\",\"import\":\"Roboto\",\"arguments\":[{\"weight\":[\"100\",\"300\",\"500\"],\"style\":[\"normal\",\"italic\"],\"subsets\":[\"latin\"],\"display\":\"swap\",\"preload\":true}],\"variableName\":\"roboto\"}","(rsc)/./node_modules/nextjs-toploader/dist/index.js","(rsc)/./node_modules/query-string/base.js","(rsc)/./node_modules/query-string/index.js","(rsc)/./node_modules/split-on-first/index.js","(rsc)/./node_modules/use-intl/dist/development/core.js","(rsc)/./node_modules/use-intl/dist/development/createFormatter-QqAaZwGD.js","(rsc)/./node_modules/use-intl/dist/development/initializeConfig-BhfMSHP7.js","(rsc)/./src/api/actions/architectsActions.js","(rsc)/./src/api/actions/articlesActions.js","(rsc)/./src/api/actions/keywordsActions.js","(rsc)/./src/api/actions/partnersActions.js","(rsc)/./src/api/actions/pathsActions.js","(rsc)/./src/app/[locale]/App.js","(rsc)/./src/app/[locale]/layout.js","(rsc)/./src/app/[locale]/loading.js","(rsc)/./src/app/[locale]/not-found.js","(rsc)/./src/app/[locale]/page.js","(rsc)/./src/components/pages/error/Error.js","(rsc)/./src/components/registry/Registry.js","(rsc)/./src/i18n/request.js","(rsc)/./src/i18n/routing.js","(rsc)/./src/utils/imageLoader.js","(rsc)/./src/utils/index.js","(ssr)/./node_modules/next/dist/server/future/route-modules/app-page/module.compiled.js?d969","../../client/components/action-async-storage.external","../../client/components/request-async-storage.external","../../client/components/static-generation-async-storage.external","/app/cache-handler.js","/app/next.config.js","/app/node_modules/next/dist/server/lib/start-server.js","buffer","bun:jsc","bun:main","next/dist/compiled/next-server/app-page.runtime.dev.js","string_decoder","util/types"] |
|
regarging the logs, here i created in staging a state where i already am using more memory than my production environement in a normal state (it never goes more than 60MO usually)
And here are the requested logs. hopefully it helps! {
"protectedObjectCount": 1308,
"objectTypeCounts": {
"JSON": 2,
"URLSearchParams": 22,
"Set": 4227,
"Immutable Butterfly": 6610,
"Map Iterator": 20,
"ProxyObject": 50,
"JSLexicalEnvironment": 24897,
"Headers": 269,
"RegExp": 1651,
"PerformanceObserverEntryList": 2,
"EvalExecutable": 8,
"ProcessBindingConstants": 1,
"WebSocket": 2,
"Boolean": 11,
"CallSite": 102,
"EventEmitter": 1,
"URLSearchParams Iterator": 2,
"WeakSet": 30,
"Exception": 19,
"InternalPromisePrototype": 11,
"Float32ArrayPrototype": 1,
"NativeExecutable": 1107,
"JSAsyncGeneratorFunction": 11,
"Uint16ArrayPrototype": 1,
"ReadableByteStreamController": 262,
"StructureRareData": 3763,
"ShadowRealm": 11,
"NextTickQueue": 1,
"WebAssembly.Table": 2,
"FunctionExecutable": 17383,
"JSPropertyNameEnumerator": 364,
"NativeZlib": 9,
"GlobalObject": 1,
"Performance": 3,
"TransformStreamDefaultController": 34,
"Function": 45152,
"HTTPServer": 2,
"Date": 57,
"JSModuleEnvironment": 9,
"string": 159906,
"ReadableStreamDefaultController": 429,
"TextDecoderStream": 2,
"ModuleLoader": 11,
"ModuleNamespaceObject": 7,
"ReadableStream": 692,
"FileInternalReadableStreamSource": 1,
"WebAssemblyWrapperFunction": 7,
"Generator": 1291,
"Callee": 22,
"WritableStreamDefaultWriter": 30,
"WritableStream": 34,
"TemplateObjectDescriptor": 14,
"CustomGetterSetter": 146,
"PerformanceObserver": 2,
"Module": 884,
"ArrayBuffer": 824,
"NativeBrotli": 1,
"JSGlobalLexicalEnvironment": 11,
"symbol": 434,
"ErrorCodeCache": 1,
"UnlinkedModuleProgramCodeBlock": 2,
"ModulePrototype": 1,
"NodeJSFS": 2,
"Array Iterator": 11,
"String Iterator": 11,
"console": 2,
"BlobInternalReadableStreamSource": 187,
"JSSourceCode": 8,
"Timeout": 377,
"ProgramExecutable": 835,
"AsyncGenerator": 38,
"ModuleRecord": 8,
"Uint32ArrayPrototype": 1,
"PerformanceEntry": 2,
"Intl.RelativeTimeFormat": 2,
"Intl.ListFormat": 1,
"Reflect": 2,
"UnlinkedFunctionCodeBlock": 7490,
"NodeVMGlobalObject": 10,
"MessagePort": 2,
"File": 1,
"MessageChannel": 2,
"TextDecoder": 39,
"Iterator Helper": 11,
"ProgramCodeBlock": 3,
"UnlinkedProgramCodeBlock": 861,
"TCPSocket": 2,
"CryptoHasher": 4,
"BroadcastChannel": 2,
"BufferList": 124,
"AsyncFromSyncIterator": 34,
"Stats": 1,
"DOMAttributeGetterSetter": 147,
"WebAssemblyModuleRecord": 1,
"Uint8Array": 31394,
"GeneratorFunction": 74,
"Intl.DisplayNames": 1,
"UnlinkedEvalCodeBlock": 8,
"PerformanceMark": 2,
"ReadableStreamDefaultReader": 459,
"Iterator": 11,
"Map": 1178,
"InternalFieldTuple": 12,
"Intl.Collator": 1,
"Uint32Array": 48,
"Float64ArrayPrototype": 1,
"Intl": 11,
"Float16ArrayPrototype": 11,
"Script": 3,
"Response": 762,
"TransformStream": 34,
"Proxy": 2,
"BigUint64ArrayPrototype": 1,
"FunctionCodeBlock": 4385,
"TextEncoderStreamEncoder": 1,
"Int16ArrayPrototype": 1,
"TextEncoder": 15,
"Intl.DateTimeFormat": 1,
"Intl.NumberFormat": 1,
"WebAssemblyFunction": 42,
"AbortSignal": 722,
"SparseArrayValueMap": 2094,
"WritableStreamDefaultController": 94,
"Int32ArrayPrototype": 1,
"Structure": 24786,
"TextEncoderStream": 2,
"ResolveMessage": 1,
"PerformanceMeasure": 2,
"Promise": 7365,
"ReadableStreamBYOBRequest": 177,
"WeakRef": 224,
"Headers Iterator": 1,
"Dirent": 1,
"Arguments": 1397,
"FormData": 2,
"ScopedArgumentsTable": 15,
"EventTarget": 3,
"ByteLengthQueuingStrategy": 2,
"WebAssembly": 11,
"InternalPromise": 16,
"ReadableStreamBYOBReader": 2,
"require": 1,
"Crypto": 2,
"WebAssembly.Module": 2,
"AsyncIterator": 11,
"Set Iterator": 41,
"BytesInternalReadableStreamSource": 3,
"ArrayIterator": 1047,
"StringDecoder": 3,
"Blob": 1,
"Bun": 1,
"Math": 1,
"FunctionRareData": 1337,
"AsyncContextFrame": 746,
"resolve": 1,
"ReadableHTTPResponseSinkController": 39,
"Process": 1,
"DOMException": 659,
"Symbol": 11,
"AsyncGeneratorFunction": 22,
"AsyncFunction": 1397,
"String": 11,
"AbortController": 23,
"ModuleProgramExecutable": 2,
"JSGlobalProxy": 11,
"WeakMap": 330,
"Event": 21,
"Int8ArrayPrototype": 1,
"InternalModuleRegistry": 1,
"StructureChain": 899,
"CryptoKey": 2,
"Error": 145,
"Prototype": 11,
"GetterSetter": 4491,
"Uint16Array": 3,
"WebAssembly.Instance": 2,
"BigInt64ArrayPrototype": 1,
"FinalizationRegistry": 12,
"Int8Array": 3,
"Object": 59406,
"Intl.Locale": 1,
"BigInt": 14,
"PropertyTable": 1546,
"SymbolTable": 6867,
"Worker": 2,
"Number": 11,
"CountQueuingStrategy": 2,
"WebAssembly.Memory": 2,
"URL": 58,
"Array": 20908,
"Uint8ClampedArrayPrototype": 1,
"RegExp String Iterator": 1,
"Intl.PluralRules": 1,
"Uint8ArrayPrototype": 2,
"Request": 41,
"UnlinkedFunctionExecutable": 18292,
"SubtleCrypto": 3,
"Buffer": 1
},
"protectedGlobalObjectCount": 1,
"globalObjectCount": 11,
"heapCapacity": 126086332,
"protectedObjectTypeCounts": {
"UnlinkedProgramCodeBlock": 603,
"HTTPServer": 1,
"GlobalObject": 1,
"AsyncContextFrame": 188,
"Promise": 1,
"Function": 16,
"ReadableStream": 377,
"Timeout": 120,
"UnlinkedModuleProgramCodeBlock": 1
},
"heapSize": 112155372,
"extraMemorySize": 80807212,
"objectCount": 530727
} |
|
I suspect it's a memory leak in ReadableStream or otherwise something in the HTTP server holding on to the ReadableStream instance for longer than necessary. But still investigating |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What version of Bun is running?
1.1.43
What platform is your computer?
oven/bun:1.1.43-alpine docker image
What steps can reproduce the bug?
I recently today performed a migration from node 20 to bun 1.1.43 of a consequent nextjs application (9 vendor website with 500k + urls)
The performance on static generation are outstanding and we can definitely see bun advantage here. The only issue that we have is a potential memory leak...
More than trying to explain i will just share the screen of the GKE dashboard were we can clearly see when i deployed from node to bun and my recent changes in between to try see if a different bun version (older) would work better.
As u can see i deployed the change at 4h30PM (the blue line is what matter) and since it keep growing in memory despite me trying everything to contain it. I could observe in a staging environment that the memory only increase when an url is fetched (probably triggering some internal nextjs process to statify the data)
I technically don't use the memory to store data as i have these config setup here:
that mean that i externalize all the cache through a redis instance. I tried to do a heat snapshot of the website mounted locally (not sure if relevant) but unfortunatly this is not my area of expertise and i am not sure what it does.
What is the expected behavior?
The expected behavior would be that bun don't generate this increase in memory.
What do you see instead?
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: