[FEA] See if we can set spark GPU resource configs from the plugin

[revans2]

Is your feature request related to a problem? Please describe.
In order to reduce the friction in adopting the RAPIDS accelerator it would be great if we could set/update Spark resource configs for the user when the plugin is enabled.

These configs are

spark.executor.resource.gpu.amount=1 and spark.task.resource.gpu.amount

We need to test to see if we can set these values from the plugin and have them properly apply to all of the various spark versions and configurations that we support. As for the value of spark.task.resouce.gpu.amount we can either set it to 1/16 (0.0625) to try to limit the task to GPU ration to be 16 to 1 at the most, or set it to be very very small so that we are not limited by the GPU.

[revans2]

I have tried to set these configs in the plugin for both the driver and the executor, but it looks like the configs are not showing up. The issue appears to be with the worker. The worker does not include the plugin, but it is the one that reads some of these configs and reports back to the driver.

[revans2]

I just confirmed that our plugin also comes up too late, even on the driver to set the configs we want. So this is something that can only be set by the auto-tuner or some kind of a bootstrap script.

I know that the auto-tuner already sets the task amount.

https://github.com/NVIDIA/spark-rapids-tools/blob/86cacaf968cdd1434d4bf1bb6a5fb54a587854ea/user_tools/src/spark_rapids_pytools/rapids/rapids_tool.py#L398

But spark complains if you do not set the executor.gpu.amount too. @amahussein have we seen issue with the executor.gpu.amount not being set? If not then I will just close this issue as cannot fix. If so then we need to spend some time to understand what is the right way/value to set it.