New build system

[vjpr]

I encountered some sporadic issues with the Makefile build. It would be nice to revamp it.

Build issues are very painful to deal with if you don't have the right tools. You end up make clean and rebuilding things and waiting 30s+ each time you tweak something, and you can lose a day easily.

CMake

The obvious choice is CMake as its used by WebKit (also Chrome, Android, LLVM, etc.). CMake's Ninja backend is cool.

JetBrains CLion also just released a CMake Debugger which should be very useful.

Proposal

We create a JS CLI tool that is the main way to build the codebase.

It will run cmake and have options to enable common debugging flags, targets, etc.

Other options

Bazel - people seem to struggle setting this up.

Interesting

Make never really goes away, so there is a tool called remake that provides debugging logs and an interactive debugger for Makefiles. It would be nice to have a doc on how to use this to track down common issues. I used it a bit.

Package Managers

We could also look at using cpp/c package managers to manage external dependencies.

Conan is integrated into CMake. It has good conventions. It would also allow us to make the codebase more modular. We could also use it for Zig modules too before a package manager is standardized.

vcpkg is another option.

Another left-field option is using a JS package manager like pnpm. It's very robust for working with a monorepo, and we do have a lot of JS code too. It would benefit us in creating a clear dependency graph.

[vjpr]

After some research, cmake might not be the best option.

Cmake is good when your project consists of other cmake packages, and you write mostly cpp code, and you want this all to build cross-platform.

Most of our code is Zig which handles the cross-platform building for us.

We rely on a few third party C packages that have automake/Makefile builds. Cmake doesn't help much with this - you still supply the make command to it and it just runs it. Here is an example: https://www.selectiveintellect.net/blog/2016/7/29/using-cmake-to-add-third-party-libraries-to-your-project-1

The big WebKit dep is cmake though.

Cmake syntax is not great. And it's an abstraction over Makefiles anyway, which can make things more difficult to debug and get how we want.

For the other Makefile deps we have, make is aware of recursive make executions. Cmake is not - I think it builds its own fixed hierachy.

If we went cmake we just get a different more high-level syntax for generating makefiles. It would only simplify things if all our deps were also cmake - like we wrap libarchive in CMakeLists.txt. I'm not sure it's worth it.

[Jarred-Sumner]

To improve C++ build times, the biggest win is probably doing something like WebKit's unified source builds and looking for unused/unnecessary #include's and cleaning them up

To improve Zig build times and memory usage, the biggest win is migrating to zig's stage2 compiler

To make bun's build system simpler, we could split it into more files I guess.

Code-generation steps:

• Generate the JSSink.cpp file (all the streams) (make generate-sink)
• Generate the classes/objects implemented in Zig (make generate-classes)
• Generate the JavaScript builtins (make generate-builtins)
• Generate the headers used for our bindings between C++ and Zig (make headers)

We don't need to run the code-generation steps unless things related to them change.

Dependencies:

• WebCrypto (make webcrypto, needs to be recompiled somewhat often)
• mimalloc (make mimalloc)
• BoringSSL (make boringssl)
• WebKit (just download the prebuilt archive)

• several more

MIsc things:

• make identifier-cache is for generating a 300 KB bitset used by the JS lexer. It eats up a number of seconds in zig build
• make all_js generates all the JS which we build ahead of time and embed into the binary

Then, it's a whole bunch of C++ files and Zig. For day-to-day building bun, this is the annoying part

The C++ files now take a similar amount of time as Zig when built from scratch. Which is unfortunate.

I wonder how much of this is a documentation problem?

The tricky part of working on build systems is it distracts from actually building the product that people use. So we really can only spend a small amount of time on things like this.

I think the overall largest barrier for contribution is the ~34 GB of memory requirement imposed by Zig's stage1 compiler. I'm hoping that this goes down a lot with stage2.