fix: Be more tolerant of intra-cluster latency when running mutations on shards

[tkaemming]

Problem

MutationRunner.run_on_shards sometimes tries to check mutation status on shards before the shard is aware of the mutation. This seems especially likely when those servers are under heavy load.

Changes

Wraps the mutation waiter in a retry policy that retries several times in the event that a mutation is not yet available before giving up.

Does this work well for both Cloud and self-hosted?

Yes

How did you test this code?

Added unit test for retry policy, change to the job is covered by existing integration test

[greptile-apps]

PR Summary

Added a RetryPolicy class to handle intra-cluster latency issues when running mutations on shards, making the system more resilient when checking mutation status.

• Implemented RetryPolicy in posthog/clickhouse/cluster.py that wraps callables with configurable retry logic (max attempts, delay, exception types)
• Applied retry policy to mutation status checks in run_on_shards method with 3 retry attempts and 10-second delay
• Added comprehensive test cases in test_retry_policy() covering successful, flaky, and consistently failing functions
• Improved error handling with proper exception propagation and logging of retry attempts
• Added helper function format_exception_summary() to provide concise error messages in logs

_{2 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile}

[tkaemming]

I'd like to figure out if it's possible to get these logs in the Dagster UI without passing their logger down the stack everywhere but that seems reasonable to do as a follow up

[fuziontech]

this will def be important to have