Feature: restart systemd service when relevant secret updated

[Radvendii]

If a systemd service depends on a secret, and agenix is managing that secret, it would be nice to restart the service on nixos-rebuild switch. Now, at the moment agenix has no way of knowing which service depends on what secret, so we could maybe add a field to specify that per-secret. Alternatively, a more general solution which is what morph does, is to just add a field for a command (or maybe script) to be run when the secret is deployed / changes.

The way I am currently managing this is via systemd paths:

{
    systemd.paths."secret-watcher" = {
      wantedBy = [ "multi-user.target" ];
      pathConfig = {
        PathModified = config.agenix."secret".path;
      };
    };
    systemd.services."secret-watcher" = {
      serviceConfig = {
        Type = "oneshot";
        ExecStart = "systemctl restart foo.service";
      };
    };
}

I'm not sure if there is a better nix-internal way to do this, but it shouldn't be too hard to translate this solution into an agenix-internal one. I'm happy to implement it if people like the idea.

[winterqt]

This is a great idea. How do you propose that a service's dependence on a secret be declared, though? (Instead of requiring the user to add a dependence on a secret to a service by manually, ideally there would be a cleaner solution that would let you specify the dependent services alongside the secrets.)

[Radvendii]

This is what I added as a separate module to accomplish my goals:

{ config, lib, ... }:

let cfg = config.age; in

{
  options.age.secrets = lib.mkOption {
    type = with lib.types;
      attrsOf (submodule ({config, ...}: {
        options = {
          action = lib.mkOption {
            type = nullOr string;
            default = null;
            description = "Action to run when secret is updated.";
            example = "systemctl restart wireguard-wg0.service";
          };
          service = lib.mkOption {
            type = nullOr string;
            default = null;
            description = "The systemd service that uses this secret.";
            example = "wireguard-wg0";
          };
        };

        config = {
          action = lib.mkIf (config.service != null)
            (lib.mkOverride 980 "systemctl restart ${config.service}.service");
        };
      }));
  };

  config.systemd = lib.mkMerge
    (lib.mapAttrsToList
      (name: {action, path, ...}: {

        paths."${name}-watcher" = {
          wantedBy = [ "multi-user.target" ];
          pathConfig = {
            PathModified = path;
          };
        };

        services."${name}-watcher" = {
          serviceConfig = {
            Type = "oneshot";
            ExecStart = action;
          };
        };

      }) cfg.secrets);
}

This way you can either specify it with

age.secrets.my-secret.service = "foo";

or

age.secrets.my-secret.action = "systemctl restart foo.service";

Which might be useful for different actions than just restarting a given service (in this case the two examples are equivalent).
action will take precidence over service if both are specified

[winterqt]

I like this a lot, nice work.

[winterqt]

@Radvendii Would you be open to PRing this implementation?

[EHfive]

There are now /run/nixos/{,dry-}activation-{restart,realod}-list that you can write service names to in activation step to restart,reload services.

https://github.com/NixOS/nixpkgs/blob/8d925cc/nixos/doc/manual/development/activation-script.section.md#activation-script-sec-activation-script

The feature has been used by sops-nix to implement on demand systemd unit restart/reload.

[lilyball]

I would like to see a solution similar to sops-nix, where I just list the units to restart or reload. If this works via the activation script approach that would be convenient, especially if it can assert that the services I listed actually exist.

In the meantime I'm tempted to just use systemd.services.«name».reloadTriggers = [ config.age.secrets."secret name".file ]. This should reload it if the age file changes, though it won't reload for any other attribute change. (EDIT: this might only work as-is with flakes, otherwise you might want to use something like [ "${config.age.secrets."secret name".file}" ] to force it to copy the path to the store)

Edit 2: systemd.services.«name».reloadTriggers = [ config.age.secrets."secret name".file ] with flakes will reload with every single change to the entire repo, not just changes to the file. Always use the antiquotation version as that will only change the store path if the local file changes due to editing or rekeying.

[Radvendii]

1. I'm not sure why you would have to quote-antiquote "${}" outside of a flake
2. If I recall correctly (which I might not be), I ran into problems with using the secret file as the reload trigger, because agenix will decrypt the secret even if there's been no change, which touches the file, so the reloadTrigger triggers on every nixos-rebuild, even if there's been no change to the file.

[Radvendii]

@EHfive Is there a reason you say that sops-nix is using the feature you mentioned? I see that they've implemented similar functionality, but as far as I can tell they just wrote go code to do it, rather than using a nixos feature.

[EHfive]

@Radvendii
https://github.com/Mic92/sops-nix/blob/c2614c4fe61943b3d280ac1892fcebe6e8eaf8c8/pkgs/sops-install-secrets/main.go#L742-L753

[Radvendii]

Oh, of course. Sorry, I was being silly and didn't actually look at the go file 🤦‍♀️

Hmm... Thanks for bringing this up. I'm not sure what advantage it has over the approach I implemented with setting restartTriggers on the service. It seems to me like they just re-implemented that functionality in the activation script.

[Radvendii]

Ohhhhh as I say that I realize what the advantage is: they're comparing the decrypted files, so it won't reload if, for instance, you just rekey the secret.

[Radvendii]

I think I ran into problems with this approach where we didn't have access to the old secrets while decrypting the new ones for some reason...

[lilyball]

@Radvendii

• I'm not sure why you would have to quote-antiquote "${}" outside of a flake

Because the way reloadTriggers works is it passes the value through toString and writes it to a special X-Reload-Triggers key in the unit file, and then activation parses unit files to see if the only difference is in that key. If that line is the only place where a difference occurs the unit is reloaded instead of restarted.

toString will convert paths into strings without copying to the store.

Given that, if I write systemd.services.«name».reloadTriggers = [ config.age.secrets."secret name".file ], that's equivalent to writing something like systemd.services.«name».reloadTriggers = [ ../secrets/foo.age ], which would then just write /path/to/local/repo/secrets/foo.age to the unit file. And any changes made to that file would result in the same path and therefore no reload.

That said, I was wrong about saying this might work fine with flakes. Flakes will copy the repo to the store so local changes produce new paths, but local changes anywhere in the repo will produce a new path. So you really do need the antiquotation always, to copy just that path to the store. I suppose you could write builtins.path { path = config.age.secrets."secret name".file; } but that's more verbose.

• If I recall correctly (which I might not be), I ran into problems with using the secret file as the reload trigger, because agenix will decrypt the secret even if there's been no change, which touches the file, so the reloadTrigger triggers on every nixos-rebuild, even if there's been no change to the file.

I'm pointing this at the source file, not the decrypted file. And it's the path that matters for reloadTriggers anyway, not the timestamp. Rekeying the source file would cause a reload of course, but merely decrypting it fresh each time won't.

An activation script that compares the actual decrypted file would be best of course, as that would allow it to actually do things like compare user/group/mode and avoid spuriously reloading due to rekeying.

[lilyball]

I suppose you could modify the reloadTriggers thing to stuff the file's owner/group/mode in there as well.

In theory agenix could actually define its secrets config to have a __toString lambda that resolves to a one-line string containing the store path, user, group, and owner specifically so you can just stuff the secret itself into reloadTriggers. That would still reload on rekeys of course, but it would be a relatively simple way of making reloading fairly painless without having to make an activation script.

[Radvendii]

Ahhh I understand. Yes, I was confused. Seems like we thought of the same solution. What you're describing is what I've implemented in the PR (more or less). It puts the user, group, permissions, file, etc. in the restartTriggers (I'm going to have an option for reloadTriggers. Working on that).