test: Vendor kcl-samples and add simulation tests for them

[jtran]

Overview

• kcl-samples is now copied in public/kcl-samples. We omit screenshots and step output files.
• ZMA uses these files when opening samples.
• We now run simulation tests on them, including generating other outputs like the AST, artifact graph, and operations.

To Do

• Artifact graph output shouldn't be non-deterministic for multi-axis-robot and walkie-talkie.
• Show which sample failed when there's a test failure.

  thread 'tokio-runtime-worker' panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/insta-1.42.1/src/runtime.rs:679:13:
  snapshot assertion for 'artifact_graph_flowchart' failed in line 235
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  
  thread 'tokio-runtime-worker' panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/insta-1.42.1/src/runtime.rs:679:13:
  snapshot assertion for 'artifact_graph_flowchart' failed in line 235
  

• We shouldn't run all of e2e tests when only kcl-samples changes.

Implementation

yarn fetch:samples downloads kcl-samples. @pierremtb implemented this and the ZMA integration.

yarn install is decoupled from fetching kcl-samples. It's part of the dev workflow that shouldn't force people to update kcl-samples if they don't want to.

Since Rust doesn't (out of the box) have a nice way to generate many tests, I made a single test for each parse, unparse, and kcl_test_execute. kcl_test_execute et al run on all KCL samples in a single Rust test. I chose this because the alternative is that we'd need an explicit list of samples in a .rs file to change every time the list of KCL samples changed, which seemed inconvenient. We could change this in the future pretty easily if we like.

Because of the chosen Rust module structure, just commands work.

just run-sim-test kcl_samples

You can also run a single one by filtering to a test prefix:

KCL_SAMPLES_ONLY=gear just run-sim-test kcl_samples

Future Work

I think that it would be nicer if modeling-app were the source of truth for kcl-samples, and we had a CI action that made a PR to the kcl-samples repo. We already use this approach for the OpenAPI specs. But, in my opinion, this PR is better than what we currently do, which is let bugs reach production.

[qa-wolf]

QA Wolf here! As you write new code it's important that your test coverage is keeping up.
Click here to request test coverage for this PR!

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
modeling-app	✅ Ready (Inspect)	Visit Preview	Feb 27, 2025 11:38pm

[jessfraz]

why not a submodule? its easier in my head than curling it all the time and when we update we just update the submodule plus dependabot could do that for us and we get auto tests thats kinda cool

also helps diffs etc

[jessfraz]

also if its a submodule people cant change individual files out of band of the og repo

[jtran]

why not a [git] submodule?

We'd like the source of truth of kcl-samples to be in modeling-app. Supposedly @jgomez720 and @nicboone8 agreed to this under the condition that changes to only kcl-samples don't run the full modeling-app e2e test suite. A git submodule would be a step backwards.