update lars partition function #12
Conversation
|
|
||
| for partition_index, chunk in enumerate(pg_lar_data): |
There was a problem hiding this comment.
It looks like we depend on the # of chunks to be equal to the # of partitions. What determines the TextFileReader chunk size, and where?
There was a problem hiding this comment.
the f"pg_lar_{year}" and f"pg_lar_{year}_{quarter}" sql queries in pipeline.py returned chunks based on the sql query result size. For example, pg_lar_2023 sql returned 47 chunks and `pg_lar_2023_q2' returned 9 chunks.
*These sql queries originated from raw_typed.yaml
There was a problem hiding this comment.
I now see the chunksize: 250000 avals for those targets in dev_postgres.yaml. If we can confirm each of those parquet files stored in s3 has 250K or fewer data rows, I think we're good.
There was a problem hiding this comment.
I confirmed the output files are 250k rows locally (I set it to output files into local folders)
Additionally, I also compared the file content from previous code and this new code, and I confirmed the result/parquet file count, files' size and content are the same.
update lars partition function removing
lar_row_counts_by_leiparameter.Originally, it's used to calculate the size of the lar data chunks returned by sql query. this means that we need to run another sql query.
Found out that the chunks list is returned as iterable type so we can use regular loop to into each chunk (and skip the
lar_row_counts_by_leisql query)Tested by running 2023 annual and Q3 lars data locally (
lar_raw_parquets_2023andlar_raw_parquets_2023_q2outputs) and verified that the parquet output files count and contents are the samecloses #11