doc/manual: Add 'Debugging Nix' section

[michaelvanstraten]

This PR adds a new 'Debugging Nix' section to the Nix manual. It provides instructions on how to build Nix with debug symbols and how to debug the Nix binary using debuggers like lldb.

Motivation

Currently, it's hard to get started with launching a debuggable instance of Nix. Building Nix with debug symbols requires knowledge of internal environment variables like mesonBuildType, which are not well-documented.

Context

Please see #11502.

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[edolstra]

Perhaps this is needed for the Meson build, but our regular build has debug symbols by default.

[michaelvanstraten]

What do you mean by a regular build? I followed this guide:

https://hydra.nixos.org/build/274312601/download/1/manual/development/building.html#building-nix-with-flakes

[Mic92]

@edolstra is talking about the old autotools buildsystem that we are migrating away from. Do we want debugoptimized instead of debug here?

Suggested change

	[nix-shell]$ export mesonBuildType=debug
	[nix-shell]$ export mesonBuildType=debugoptimized

[michaelvanstraten]

Optimized makes sense to me if the debugger is this able to nicely link back into the source code.

[edolstra]

FWIW, I use gdb, and if you're in a nix develop shell, you can just do

gdb --args nix ...

i.e. no need to give a full path to the nix binary.

[Mic92]

So meson actually installs it to /home/joerg/git/nix/out/bin which is not in PATH. But in that case we would also need to run ninjaInstallPhase. I think it makes sense that both build system choose different locations for our own sanity while we still have two and not put both in PATH. I actually prefer being explicit about the executable location, because it's less error prone for users following the tutorial. Let's say one person has exited the devshell to get a debugger and no longer has the PATH from it set, in that case they might run the debugger against the global installed nix, which is not good.

[Mic92]

To get a proper nix installation, we might want to install it instead? The extra overhead is negligible.

Suggested change

	[nix-shell]$ ninjaBuildPhase
	[nix-shell]$ ninjaInstallPhase

[Mic92]

Maybe we want to suggest gdb on Linux still by default? At least for me it used to be more stable. But I am happily convinced if this has changed. And on macOS I believe we should not suggest to install lldb with nix-shell because it's not code-signed, meaning it won't be able to use the debugger api without disabling apples security protection.

[michaelvanstraten]

For Linux I can see why we would want to recommend gdb, it's just the case that I am on Mac an that's the only thing I was familiar with.

I haven't experienced needing to disable security protection on macOS to work with lldb in nix-shell.

[michaelvanstraten]

I haven't figured out how to both debug the nix-daemon and nix when the debugging user is not the owner of /nix.

We could use the --store option to point to a local store but that would make debugging harder.

Is there a way to point nix to a different socket in terms of the daemon?

[Mic92]

I haven't figured out how to both debug the nix-daemon and nix when the debugging user is not the owner of /nix.

We could use the --store option to point to a local store but that would make debugging harder.

Is there a way to point nix to a different socket in terms of the daemon?

if you run as root than nix won't use the nix-daemon but will have similar privileges. This might be helpful for debugging. Otherwise, you would need to run two debugger sessions.
One that runs the nix-daemon as root and a second one with the actual command.
Debuggers are not very good at tracking multiple processes.

[michaelvanstraten]

if you run as root than nix won't use the nix-daemon but will have similar privileges.

Then the problem becomes that you can't reproduce the IPC interactions between the two.

[Mic92]

Usually you won't to need a debugger on both processes at the same time. You can just have the debugger on one process and than maybe verbose logging or strace on the other instance.

[michaelvanstraten]

Usually you won't to need a debugger on both processes at the same time. You can just have the debugger on one process and than maybe verbose logging or strace on the other instance.

I recently had the problem that the nix-daemon wasn't able to access an ssh key needed to access a remote builder. This was do to the fact that the key was password protected and the subprocess was not piped back to the nix client.