You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
In an effort to reduce the number of configs that a user must set we should try and have a better default for spark.rapids.sql.multiThreadedRead.numThreads. The auto-tuner currently sets this to max(80, executor.core) for CSPs and max(20, 2*executor.core) for on prem. We should know approximately the number of executor cores in the plugin, but we don't know always if we are in a CSP or on prem. We should run some tests and see if we can figure out a good default, beyond the 20 we have today.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
In an effort to reduce the number of configs that a user must set we should try and have a better default for
spark.rapids.sql.multiThreadedRead.numThreads
. The auto-tuner currently sets this tomax(80, executor.core)
for CSPs andmax(20, 2*executor.core)
for on prem. We should know approximately the number of executor cores in the plugin, but we don't know always if we are in a CSP or on prem. We should run some tests and see if we can figure out a good default, beyond the 20 we have today.The text was updated successfully, but these errors were encountered: