feat(row-counts-for-saved-queries): Temporal row counts for saved queries

[phixMe]

Problem

Row counts are not updated and we don't bill for rows that are materialized in views... The first step is to have methods that count the records written via delta for our data-modeling-run.

Rows now updated:

Changes

Does this work well for both Cloud and self-hosted?

Yes

How did you test this code?

Works on scheduled queries, tested on 72k events in test env. Seems to work ok and update.

[greptile-apps]

PR Summary

This PR adds row counting functionality for materialized saved queries in the data warehouse, enabling proper billing and accurate reporting of data size.

• Added count_delta_table_rows function in run_workflow.py to efficiently count rows in Delta tables using batched processing
• Implemented update_table_row_count to update the row_count field in the corresponding DataWarehouseTable model
• Modified materialize_model to call these functions after materializing data
• Added test assertions in test_run_workflow.py to verify row counts are correctly updated after materialization
• Included error handling with proper logging to ensure robustness in production environments

_{2 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile}

[EDsCODE]

looks good. as a reminder the billing system works by pulling "rows_synced" from external data job which means we aggregate the total rows synced across all syncs. An analogy with materialization would be saving the row count/row size each sync vs just the rows in the table.

but, as mentioned, we should rethiink what we're counting in the first place

[phixMe]

looks good. as a reminder the billing system works by pulling "rows_synced" from external data job which means we aggregate the total rows synced across all syncs. An analogy with materialization would be saving the row count/row size each sync vs just the rows in the table.

but, as mentioned, we should rethiink what we're counting in the first place

Yep, I just wanted to get this row updated before tackling the bigger problem... I wasn't even sure if we wanted to bill in the same way in terms of rows, but that's a larger discussion...