Integration with systemd-creds

[korfuri]

Hi all, I've been experimenting with agenix and other secret distribution mechanisms trying to find the one I like the most. I'm quickly converging on agenix as my top choice! One thing I've noticed is that ~nothing in the NixOS world seems to make use of systemd's credentials management feature. It's quite new so that's probably why.

The full doc on that feature is at https://systemd.io/CREDENTIALS/ and the tl;dr is that you can encrypt secrets to a host-specific key (TPM-backed, if a TPM2 is available) and let systemd manage the decryption and distribution of secrets to units. This offers some improvements over agenix's current method of distributing secrets (decrypting them all at activation time and chown/chmoding them to their intended user):

• systemd can decrypt secrets only as they are needed
• systemd always knows the right owner and group for a secret since it's configuring it based on the unit. This is significant if you need to share a secret among several services on the machine (but I don't have a real use-case for this)
• it's probably the systemd-idiomatic way of solving issue Feature: restart systemd service when relevant secret updated #84

It does have a significant downside: no support for "user" (as in, isNormalUser = true users) secrets. This can be hacked around, in that you could run "system" oneshot services as root to copy creds to a user's homedir, I guess.

I have a prototype implementation of a fork of agenix that does this. Essentially, it's creating oneshot units for each secret that basically do age --decrypt -o - | systemd-creds encrypt - $TMP_FILE at https://github.com/korfuri/agenix-systemd in case anyone is interested in this approach. I could have made it a tentative PR for agenix but I found that it made the agenix API quite cumbersome and that a prototype was probably a better way to start the discussion and gauge interest.

A couple notes to address some questions you may have:

• yes, my initial approach was "I can just add a | systemd-creds encrypt in the existing agenix implementation, unfortunately activation scripts run before systemd starts, which means systemd-creds can't talk to the bit of systemd that has access to the host key.
• unfortunately, systemd-creds does not support asymmetric crypto (it's on the TODO), so it's not possible to encrypt secret "to" another host. This is why I want to integrate with agenix, so I have secrets encrypted to a given host, then use systemd's native crypto to protect them while on-host. If one is willing to generate all secrets on the machine itself, the agenix part becomes irrelevant and you might as well check in the systemd-creds-encrypted secrets to your git repo :)
• yes, there's a demo that doubles as an integration test of how this works in practice with my implementation, including a couple of riffs on how the API could look like.
• No, none of the services in nixpkgs currently use LoadCredentialEncrypted= or SetCredentialEncrypted= so using this at scale will require some hacking with service configs. If your service supports environment variable expansion that's quite easy (secret foo is at $CREDENTIAL_DIRECTORY/foo), but if it doesn't you probably need to pre-process the config or something. $CREDENTIAL_DIRECTORY changes so you can't just hardcode it somewhere either.

I'm interested in what people think about this. I'm not entirely sold - the main benefit to me is pretty immaterial tbh, it's just that it's nice to use systemd idiomatically. Would you use this? Would it make sense to try to make this a native agenix feature? Did I miss something glaringly obvious and my approach is broken?

Thanks!

[ryantm]

@korfuri Thanks for the thoughtful post. Here are my concerns:

1. People are close to having agenix support for nix-darwin, which I believe does not support systemd, so adding systemd credentials might complicate this. (Though I keep pushing the people to do the nix-darwin code in a separate module file, so maybe this is a non-issue.)
2. Correct me if I'm wrong, but it sounds like it cannot handle the case of secrets that are needed for the creation of user accounts which happens as an activation script. Provisioning user account passwords is an important feature for me personally at least.
3. Since it doesn't support asymmetric cryptography, it seems like this could only be bolting something onto the end of the existing decryption code, which isn't going to simplify agenix.

[korfuri]

Thanks for the reply! Those are very valid points.

I wasn't thinking of replacing the secret delivery method, but more of providing users with a choice. Possibly in an extensible way, so there would be other credential delivery methods (e.g. deliver to the machine's keyring, or to the kernel's crypto API). I haven't followed the Darwin proposals but maybe these are two similar needs (different secret delivery mechanisms) and a common API could help there as well?

Your point 2 is indeed correct (AFAIK, systemd doesn't manage users... yet ;) ) which feeds into why this can't replace every secret delivery.

To your third point, it's precisely because systemd-creds doesn't support asymmetric cryptography that I want to use agenix in conjunction with it :) But yeah, I'd like agenix to deliver the secrets to a pipe, and then let systemd-creds take it from there. In terms of how these two can interface together in practice, see this line: https://github.com/korfuri/agenix-systemd/blob/abf6c934ba3af682a838670e51357ffa0069f9c0/modules/age.nix#L26

I don't want to be pushy or anything, I'm also happy to use my own fork, but if you have some thoughts on a "secret delivery API" for the output side of agenix, I'm also happy to contribute to the discussion.

[samhza]

Hello, I am very new to agenix and systemd-creds. What benefits would there be for using LoadCredentialEncrypted over LoadCredential? Why decrypt and re-encrypt the secret for systemd if it is to be immediately decrypted?

[korfuri]

Using LoadCredential in a non-encrypted way also works. It does bring most of the direct benefits of using systemd-creds (mostly: automatic management of secret permissions). However, I'd like to steer NixOS modules to use LoadCredentialEncrypted= for one main reason: it makes it less likely that secrets will be leaked to the store. You just can't make LoadCredentialEncrypted work with a derivation that writes the secret to the store, unless you write your derivation to call out to systemd-creds and that derivation should fail because of hermeticity.

The main benefit from standardizing on the Encrypted version, even if it adds some complexity, is that this lets us change the off-host secret distribution story without changing the on-host one. E.g., if modules all rely on LoadCredentialEncrypted we have a path to using systemd-creds without age once systemd-creds implements public-key crypto support (it is in systemd's TODO). Age is a great option for off-host secret distribution but it's likely to run into some hurdles for the enterprise use-case, where things like Hashicorp Vault would probably work better (or at least just have a better sales pitch). We'll need to standardize how NixOS modules accept credentials, and so I'd rather have it be on the encrypted variant rather than the unencrypted one.

[antifuchs]

I'm working on something similar right now that would benefit from a feature where agenix could be instructed to create a transparently re-encrypted (with systemd-creds) file on the host: I'm writing a program that runs in initrd (with boot.initrd.systemd.enable=true), which should take secrets encrypted with the host's TPM2, so that the initrd doesn't contain the unencrypted secret.

I'm not completely sure yet how the process of "place unencrypted file on the host's file system" -> "pack that re-encryted secret into the initrd" should go, but I feel that agenix would be a pretty good place for the transparent re-encryption to take place.