feature: nix based scripts

[DrRuhe]

Overview

Enable users to define scripts using Nix, leveraging its package management to ensure reproducible environments, precise dependency management, and access to Nixpkgs.

Benefits of Nix Integration

Central Nixpkgs Revision

Windmill can use a shared Nixpkgs revision, ensuring consistent versions of tools across all scripts. This centralization makes updating and managing dependencies easier, as only the nixpkgs reference has to be updated for all scripts to receive new versions.

Per-Script Dependency Isolation

Nix allows each script to declare its specific environment, avoiding global dependencies. For example, if only one script requires a particular Python library, it can install and use it independently, preventing conflicts with other scripts. This ensures robustness and avoids breaking other scripts due to version mismatches.

Efficient Distribution via the Nix Store

Once a derivation (Nix-built package) is created, it’s stored in the Nix store. Scripts can reuse these pre-built binaries, ensuring fast deployment and execution without recompilation.

Questions

• How to get jsonschema from the tools?
  • nix doesn't automatically produce jsonschema for derivations. For nix derivations that produce scripts in languages already supported by windmill, the existing infrastructure for jsonschemas can be used.
  • in the remaining cases a jsonschema could be added manually in the nix derivation under the meta attribute
  • Is it possible to omit jsonschema?

Example 1 : build a Go tool

An example script using Nix to build a Go tool:

Note that this example is the definition of the carapace tool from nixpkgs.

{ lib, buildGoModule, fetchFromGitHub }:

buildGoModule rec {
  pname = "carapace";
  version = "1.0.7";

  src = fetchFromGitHub {
    owner = "carapace-sh";
    repo = "carapace-bin";
    rev = "v${version}";
    hash = "sha256-...";
  };

  ldflags = [
    "-s" "-w" "-X main.version=${version}"
  ];

  meta = with lib; {
    description = "Multi-shell argument completer";
    license = licenses.mit;
  };
}

Example 2: A python script

This script takes an input MP4 file, extracts the audio using ImageMagick, and generates a spectrogram using Python libraries.

Note that this is a contrived example and the script was generated by ChatGPT, so it likely wont work as is. But that's beside the point of this issue. I believe it still conveys the desire for nix based script support in windmill

{ lib, python3, imagemagick, ffmpeg, matplotlib, numpy, scipy }:

# Define a Python environment with the required tools and libraries
python3.buildPythonApplication rec {
  pname = "audio-spectrogram";
  version = "1.0";

  # Inline Python script
  src = ''
    import numpy as np
    import matplotlib.pyplot as plt
    import subprocess
    from scipy.io import wavfile

    # Step 1: Extract audio from MP4 using ImageMagick
    def extract_audio(mp4_file, output_wav):
        subprocess.run(['convert', mp4_file, output_wav])

    # Step 2: Generate and plot the spectrogram
    def plot_spectrogram(wav_file, output_image):
        sample_rate, audio_data = wavfile.read(wav_file)
        plt.specgram(audio_data, Fs=sample_rate)
        plt.title('Spectrogram')
        plt.savefig(output_image)

    if __name__ == "__main__":
        mp4_input = "input.mp4"
        wav_output = "output.wav"
        spectrogram_output = "spectrogram.png"
        
        extract_audio(mp4_input, wav_output)
        plot_spectrogram(wav_output, spectrogram_output)
  '';

  propagatedBuildInputs = [
    imagemagick   # Tool for extracting audio from video
    ffmpeg        # Handles multimedia files
    matplotlib    # For plotting the spectrogram
    numpy         # For numerical computations
    scipy         # For signal processing
  ];

  buildInputs = [ imagemagick ];

  meta = with lib; {
    description = "Extract audio from an MP4 and plot its spectrogram";
    license = licenses.mit;
    maintainers = with maintainers; [ "yourname" ];
  };
}

[rubenfiszel]

Thanks for the input

Per-Script Dependency Isolation

Nix allows each script to declare its specific environment, avoiding global dependencies. For example, if only one script requires a particular Python library, it can install and use it independently, preventing conflicts with other scripts. This ensures robustness and avoids breaking other scripts due to version mismatches.

Just a note that's already what Windmill provides for Python, with a caching strategy that is much more efficient than nix and can be backed by S3.

The fact that the code of the script would be a string inline in the "src" field would give a terrible user experience, both locally and in the web editor, prevents to use LSP and the monaco tooling and go against the spirit of Windmill to put the code logic at the center of the DX.

I'm sure some form of nix integration is possible but it seems to me that under this form, it's not so different from calling nix with bash on an inline nix script and would not integrate well.

A better form of Nix integration would probably to do it per-language and likely have the whole nix script (except src) in a metadata field that could be updated but that is something that would require quite a bit of work and we have very little request so far for it so until we do, it would likely have to be a community contribution.