Derivations built locally that have dependencies built remotely hang indefinitely on "Sending inputs"

[hacker1024]

Describe the bug

When a local build has a dependency built remotely, it [the local build] hangs indefinitely on "Sending inputs". When this occurs, such builds cannot even be cancelled: Attempting to do so labels it a such, but the job still shows up in the machine status page and prevents the build queue from advancing.

Nothing significant appears to be logged on the Hydra server or remote builder.

Single-host builds still complete as normal.

Practical examples of such cases:

• An aggregate job with constituents of another architecture
• A deployment runCommandHook using a NixOS system closure for another architecture
• A regular derivation depending on another with specific requiredSystemFeatures (untested)

To Reproduce

1. Set up two machines: A Hydra server, and a remote builder
2. Add a job to build a derivation on the server machine that depends on another derivation built on the remote builder. This can be done by either changing the system or requiredSystemFeatures.
3. Note that the server-built build hangs on "Sending inputs"

Expected behavior

The remote dependency should be copied onto the server machine, and the build using it should complete successfully.

Screenshots

I'm pretty sure the "ssh" here is cosmetic - normal jobs have no issue with this builder, it doesn't actually seem to be using SSH.

Hydra Server:

• OS and version: nixos-unstable @ 32fb99ba93fea2798be0e997ea331dd78167f814 (tested both with and without hydra: 0-unstable-2025-02-12 -> 0-unstable-2025-02-13 nixpkgs#382171)
• Version of Hydra: hydra-0-unstable-2025-02-18
• Version of Nix Hydra is built against: 2.26.3
• Version of the Nix daemon: 2.24.12 (both server and builder)

Additional context

I upgraded from 02e453f with Nix 2.19, which did not have this issue.