Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] user defined hints to better optimize query execution #12162

Open
binmahone opened this issue Feb 18, 2025 · 1 comment
Open

[FEA] user defined hints to better optimize query execution #12162

binmahone opened this issue Feb 18, 2025 · 1 comment
Assignees
Labels
feature request New feature or request

Comments

@binmahone
Copy link
Collaborator

binmahone commented Feb 18, 2025

The idea origins from a discussion in #12122 (comment) , @revans2 @mattahrens and I had some offline discussions are we decide to open a issue to track this idea first.

apache/spark#25071 (there's an example in the PR) enables us to create user defined hints to optimze query execution. Like #12121, given historical exeuction eventlogs, we may come up with some useful hints to inject into the query, either by manually or by help of some kind of automatic tools. With these hints rapids plugin can be smarter in choosing different operator implementation, e.g. Velox Scan vs. Gpu Scan, or deciding whether to do join reordering or aggregation push down.

Possible hint candidates include:

  1. if a join explodes
  2. if a join does not add rows
  3. if a join is highly selective
  4. if a scan is higly selective after predicate filter pushdown
  5. etc.

Compared with #12121, hints might be able to support more fine grained optimizations. For example, in a single query one can specify different hints for different tables, so that some of the Scans can use Velox Scan and others using GPU Scan. However, 12121 and hints should NOT be mutually exclusive, as they cover slightly different use cases.

@binmahone binmahone added ? - Needs Triage Need team to review and classify feature request New feature or request labels Feb 18, 2025
@binmahone
Copy link
Collaborator Author

@revans2 please feel free to refine the issue description

@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants