Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Explore file read config heuristics in the plugin/auto-tuner to reduce spilling but increase throughput #12133

Open
revans2 opened this issue Feb 13, 2025 · 0 comments
Labels
feature request New feature or request

Comments

@revans2
Copy link
Collaborator

revans2 commented Feb 13, 2025

Is your feature request related to a problem? Please describe.
This is very similar to #12122 but is intended for configs like

spark.sql.files.maxPartitionBytes

The simpler part is that we know how to override this value for GPU reads. We have total control over that. The harder part is that we have less information and may need to rely more on historical information like #12121 to augment any heuristics we can come up with.

The issue is that we know the size of the compressed data on disk, but we don't know the compression ratio of the data in that. We don't know the selectivity of the predicate push down. We don't know the selectivity of the filter after the predicate push down. We also don't really know the size of the row groups so that we can avoid launching tasks that are just going to read a footer and exit.

Just like in the others we want to start off by implementing the triple buffering feature #11343 and then work on heuristics to figure out what the decompressed data sizes are going to look like. After that the input size estimation should look very similar to how shuffle ends up working.

@revans2 revans2 added ? - Needs Triage Need team to review and classify feature request New feature or request labels Feb 13, 2025
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants