[Memory Leak]: Builds incorporating Shiki Twoslash crash with JavaScript heap out of memory errors

[IMax153]

Describe the bug

Hello 👋 from the Effect-TS community :)

We have recently been experimenting with using different frameworks for building our documentation website. I decided to try building our documentation with Vitepress because of your native integration with Shiki Twoslash (via @shikijs/vitepress-twoslash), which we make extensive use of in our docs.

However, when I attempt to build our documentation with Vitepress and Shiki Twoslash, I consistently receive a JavaScript heap out of memory error. I have attempted to increase the --max-old-space-size to ~8GB and still encounter this issue.

With Twoslash Integration

In my .vitepress/config.ts, if I remove the transformerTwoslash from the markdown.codeTransformers array, the build succeeds without issue in ~20 seconds.

Without Twoslash Integration

Also, please let me know if it is more appropriate to move this issue to the Shiki project. If so, I have no problem closing this issue and moving it 👍 Thank you for your time!

Reproduction

1. Clone our docs experiment

   git clone https://github.com/IMax153/vitepress-memory-leak-repro.git

2. Install dependencies

   pnpm install

3. Attempt to build

   pnpm build

Expected behavior

My expected behavior would be for Vitepress to be able to build our docs with Shiki Twoslash integrated without issue.

System Info

System:
    OS: macOS 14.6.1
    CPU: (10) arm64 Apple M1 Pro
    Memory: 1.99 GB / 16.00 GB
    Shell: 5.9 - /nix/store/w53pgqwh5dm38skdd7n7wn0k6cw72jzj-zsh-5.9/bin/zsh
Binaries:
    Node: 20.17.0 - /nix/store/hxl1k8qgmrm1vfq5f419iv4wybz9szqq-nodejs-20.17.0/bin/node
    npm: 10.8.2 - /nix/store/hxl1k8qgmrm1vfq5f419iv4wybz9szqq-nodejs-20.17.0/bin/npm
    pnpm: 9.10.0 - /nix/store/ddjsxwcpw09llisgvw7drvr92dbkg4mi-corepack-nodejs-20.17.0/bin/pnpm
Browsers:
    Chrome: 129.0.6668.70
    Safari: 17.6
npmPackages:
    vitepress: ^1.3.4 => 1.3.4

Note: I use Nix for provisioning my system, hence the strange looking binary paths.

Additional context

I've included a script in the package.json that reduces the available NodeJS heap memory (to force an OOM earlier) and takes a heap snapshot right before an OOM error occurs in case that is useful for you folks.

pnpm build:snapshot

Validations

• Check if you're on the latest VitePress version.
• Follow our Code of Conduct
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

[brc-dd]

Yeah https://github.com/shikijs/shiki/issues might be better place to post this. I'm fine with keeping this open if it's indeed a memory leak with vitepress.

[IMax153]

https://github.com/shikijs/shiki/issues might be better place to post this

No problem, makes sense to me - I've gone ahead and opened an issue upstream. Linking here for tracking purposes: shikijs/shiki#796

[jasonkuhrt]

I started experiencing these issues today with Graffle's website as well. https://github.com/jasonkuhrt/graffle/actions/runs/11129348920/job/30926231362?pr=1151. Nothing more to add than what @IMax153 has already said.

[brc-dd]

@jasonkuhrt If possible, do you have the exact commit which started giving these? If it's recent then it might have something to do with newer shiki releases. 👀

[jasonkuhrt]

Hey @brc-dd, it started failing on this commit ade5a3e61ad536484968880d9218bdcdf4a66ac1. It's a huge commit unfortunately. You can see under the website directory no version of any dependencies changed in it fwiw.

[nekomeowww]

Hmmmm.

I am not using any twoslash plugins, but encountered the similar issue after bumpping up Nolebase's dependencies, will investigate more to find out which dependency was causing this. Similar setup with VitePress, with twoslash plugin (and actually the tool set used to build Nolebase website) in Nolebase Integrations cannot reproduce this OOM issue.

✓ building client + server bundles...
⠇ rendering pages...
<--- Last few GCs --->

[36265:0x148008000]   201737 ms: Mark-Compact 4030.9 (4139.8) -> 4005.2 (4138.0) MB, 3299.75 / 0.00 ms  (average mu = 0.099, current mu = 0.015) task; scavenge might not succeed
[36265:0x148008000]   207589 ms: Mark-Compact 4022.3 (4139.0) -> 4013.1 (4136.0) MB, 5711.75 / 0.00 ms  (average mu = 0.052, current mu = 0.024) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x102ac34c4 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
 2: 0x102c6836c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
 3: 0x102e48b4c v8::internal::Heap::stack() [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
 4: 0x102e4c470 v8::internal::Heap::CollectGarbageShared(v8::internal::LocalHeap*, v8::internal::GarbageCollectionReason) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
 5: 0x102e4b0bc v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
 6: 0x102e5be90 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)::$_6::operator()() const [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
 7: 0x102e5ba7c void heap::base::Stack::SetMarkerAndCallbackImpl<v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)::$_6>(heap::base::Stack*, void*, void const*) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
 8: 0x103495b08 PushAllRegistersAndIterateStack [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
 9: 0x102e46c1c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
10: 0x102e3e048 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
11: 0x102e3e8e4 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
12: 0x102e22698 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
13: 0x1031f07b0 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
14: 0x10358b954 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
15: 0x10358c378 Builtins_StringAdd_CheckNone [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
16: 0x10ad1e828 
17: 0x10adafd00 
18: 0x10ad66158 
19: 0x10ad66794 
20: 0x10ad66ff0 
21: 0x10ad65d18 
22: 0x10ad677d0 
23: 0x10ad694f4 
24: 0x1035ef354 Builtins_PromiseConstructor [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
25: 0x1034fdc3c Builtins_JSBuiltinsConstructStub [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
26: 0x10ad69914 
27: 0x10ad6d4d0 
28: 0x10ad73bec 
29: 0x10acd5924 
30: 0x103537b10 Builtins_AsyncFunctionAwaitResolveClosure [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
31: 0x1035f12d8 Builtins_PromiseFulfillReactionJob [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
32: 0x103526654 Builtins_RunMicrotasks [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
33: 0x1034fe794 Builtins_JSRunMicrotasksEntry [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
34: 0x102daa8b8 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
35: 0x102daad68 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
36: 0x102daaf14 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
37: 0x102dd2b24 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
38: 0x102dd32c0 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
39: 0x1029f0c4c node::InternalCallbackScope::Close() [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
40: 0x1029f07bc node::InternalCallbackScope::~InternalCallbackScope() [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
41: 0x102b27f90 node::PerIsolatePlatformData::RunForegroundTask(std::__1::unique_ptr<v8::Task, std::__1::default_delete<v8::Task>>) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
42: 0x102b26ce0 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
43: 0x1034df250 uv__async_io [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
44: 0x1034f1814 uv__io_poll [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
45: 0x1034df814 uv_run [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
46: 0x1029f16f0 node::SpinEventLoopInternal(node::Environment*) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
47: 0x102b0244c node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
48: 0x102b021f8 node::NodeMainInstance::Run() [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
49: 0x102a8dda4 node::Start(int, char**) [/Users/neko/.volta/tools/image/node/21.6.2/bin/node]
50: 0x18bd510e0 start [/usr/lib/dyld]
sh: line 1: 36265 Abort trap: 6           vitepress build

[nekomeowww]

I did a investigation, found that starting from [email protected], the OOM issue can be reproduced constantly. But only from my repository, it may not be the issue of any of the core package from Vue but rather something went bad for my components.

But not sure which line of change caused this... vuejs/core@v3.5.5...v3.5.6

Update:

NVM, I think this is related to Vite or other dependencies, even with [email protected] (previously working one), if vite is updated to 5.4.8 or vitepress updated to 1.4.0 (not sure which one yet), then this issue presist.