feat(devex): cypress roulette

[mariusandra]

Problem

We spend too much on cypress with not much to show.

Changes

Run only 4 random cypress tests each run.

How did you test this code?

See the above 🤪

[greptile-apps]

PR Summary

This PR modifies the Cypress E2E testing strategy to run only 4 randomly selected tests per CI run instead of all tests.

• Added a new step in the chunks job that selects 4 random spec files using the commit's SHA as a seeded random source
• Changed matrix variable from chunk to spec throughout the workflow for clarity
• Updated artifact naming to use the specific spec name instead of job index
• Modified the Cypress run command to execute only the selected spec files
• Implemented a reproducible selection mechanism using the commit SHA to ensure consistent test selection for the same code

_{1 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile}

[pauldambra]

I do think once a day we should run them all and send an event to posthog on failure so we can alert from CDP / track incidence

but otherwise 100

these tests are slow and low value

[mariusandra]

Yeah, that works for me. Though the only thing these tests actually test for me is that the plugin server is running. I didn't (yet) explicitly verify if the same issue causes playwright to break as well... will report back.

[mariusandra]

For clarity I put the name of the test into the action's name, but this just means none will actually be required:

[pauldambra]

yeah, they're a bit of a dead end cos everyone just finds them annoying...

[pauldambra]

yep, we'd make them not required and tell everyone, plz be careful-er now

any migration to playwright that is done carefully will just take too long

[pauldambra]

i'm in