[FEA] Investigate/Refactor jenkins/databricks scripts to support use databricks CLI/API only

[pxLi]

Is your feature request related to a problem? Please describe.

from @gerashegalov We could use the AWS issue with SSH as an opportunity to rewrite the test scripts more in terms of the frontdoor APIs

1. git clone and build the PR branch using an init script
2. run actual tests as Jobs submitted using databricks CLI from Jenkins

@NvTimLiu to help investigate, and we also have the same scripts in the spark-rapids-private project

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context, code examples, or references to existing implementations about the feature request here.