You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The idea origins from a discussion in #12122 (comment) , @revans2@mattahrens and I had some offline discussions are we decide to open a issue to track this idea first.
apache/spark#25071 (there's an example in the PR) enables us to create user defined hints to optimze query execution. Like #12121, given historical exeuction eventlogs, we may come up with some useful hints to inject into the query, either by manually or by help of some kind of automatic tools. With these hints rapids plugin can be smarter in choosing different operator implementation, e.g. Velox Scan vs. Gpu Scan, or deciding whether to do join reordering or aggregation push down.
Possible hint candidates include:
if a join explodes
if a join does not add rows
if a join is highly selective
if a scan is higly selective after predicate filter pushdown
etc.
Compared with #12121, hints might be able to support more fine grained optimizations. For example, in a single query one can specify different hints for different tables, so that some of the Scans can use Velox Scan and others using GPU Scan. However, 12121 and hints should NOT be mutually exclusive, as they cover slightly different use cases.
The text was updated successfully, but these errors were encountered:
The idea origins from a discussion in #12122 (comment) , @revans2 @mattahrens and I had some offline discussions are we decide to open a issue to track this idea first.
apache/spark#25071 (there's an example in the PR) enables us to create user defined hints to optimze query execution. Like #12121, given historical exeuction eventlogs, we may come up with some useful hints to inject into the query, either by manually or by help of some kind of automatic tools. With these hints rapids plugin can be smarter in choosing different operator implementation, e.g. Velox Scan vs. Gpu Scan, or deciding whether to do join reordering or aggregation push down.
Possible hint candidates include:
Compared with #12121, hints might be able to support more fine grained optimizations. For example, in a single query one can specify different hints for different tables, so that some of the Scans can use Velox Scan and others using GPU Scan. However, 12121 and hints should NOT be mutually exclusive, as they cover slightly different use cases.
The text was updated successfully, but these errors were encountered: