You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
The auto tuner code currently sets the number of shuffle threads to 28 for CSPs and 1.5 * numExecutorCores for on prem environments. We should look into which if either of these we might want to use in the plugin by default, or if there is something better we can use to figure this out. We do know the number of executor cores, because if the parallelism is less than the number of executor cores, then a warning is output. We should run some performance tests to verify that whatever change we make is better generally.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
The auto tuner code currently sets the number of shuffle threads to 28 for CSPs and
1.5 * numExecutorCores
for on prem environments. We should look into which if either of these we might want to use in the plugin by default, or if there is something better we can use to figure this out. We do know the number of executor cores, because if the parallelism is less than the number of executor cores, then a warning is output. We should run some performance tests to verify that whatever change we make is better generally.The text was updated successfully, but these errors were encountered: