Clean up runtime code for finding/using standalone python and entering a venv

[jdstrand]

#25969 lists this as a followup once it is merged:

• influxdb3/src/main.rs has some hacks to help find things. This is clearly not the right place for this code and there may be ways to not need it at all. Specifically:
  • set_pythonhome() sets PYTHONHOME based on where it finds the runtime. I kinda feel like this shouldn't be needed, but the concept of it isn't terrible (even if the location in the code and probably the code itself is)
  • set_pythonpath() sets PYTHONPATH and is required on Windows for some reason. This should be fixed properly
• influxdb3_processing_engine/src/virtualenv.rs has a few changes (but not enough):
  • get_python_version() is adjusted to use python on Windows as python3 doesn't exist there
  • initialize_venv() is adjusted to use Scripts/activate and call cmd.exe on Windows
• despite the above, --virtual-env-location doesn't work on any platform. Linux and OSX can be made to work with venv using source /path/to/venv/bin/activate and launching influxdb3 serve ... under it (client doesn't need this). Windows needs to set PYTHONPATH=\path\to\venv\Lib\site-packages

Tentatively assigning to @jacksonrnewhouse for (at least) the virtualenv.rs and --virtual-env-location. I can handle the move away from main.rs but I suspect @jacksonrnewhouse is best positioned for this.

[jdstrand]

I'll mention that there is a pyvenv.cfg in the top-level directory of any created virtual environment. Eg:

$ cat ./test-venv/pyvenv.cfg 
home = /usr/lib/influxdb3/python/bin
include-system-site-packages = false
version = 3.11.11
executable = /usr/lib/influxdb3/python/bin/python3
command = /usr/lib/influxdb3/python/bin/python3 -m venv /home/jamie/test-venv

I wonder if we should first run the activate script upon entering, then parsing pyvenv.cfg to fix up sys.prefix and sys.path as needed?

[jdstrand]

Some more info:

The server must be launched on Linux/OSX (assumes tarball unpacked at ~/tmp/influxdb3-pe/install):

# venv isn't setup automatically currently, so do that
$ ~/tmp/influxdb3-pe/install/python/bin/python3 -m venv ~/tmp/influxdb3-pe/data/plugins/.venv

# --virtual-env-location is not honored, so activate the venv
$ source ~/tmp/influxdb3-pe/data/plugins/.venv/bin/activate
$ ~/tmp/influxdb3-pe/install/influxdb3 serve --node-id=local01 --object-store=file --data-dir ~/tmp/influxdb3-pe/data --plugin-dir ~/tmp/influxdb3-pe/data/plugins

# this works now and is all that is needed for plugins to start using it
$ ~/tmp/influxdb3-pe/install/influxdb3 install package requests

Windows works less well, but can be made to work. The server must by launched (assumes zip unpacked at influxdb3-pe\install):

# venv isn't setup automatically currently, so do that
$ influxdb3-pe\install\python\python.exe -m venv influxdb3-pe\data\plugins\.venv
# 'influxdb3.exe install package ...' doesn't work, so activate the venv and install it manually
$ influxdb3-pe\data\plugins\.venv\Scripts\activate
(.venv) $ python -m pip install requests
...

# neither --virtual-env-location nor launching under an activated venv works, so deactivate and
# set PYTHONPATH manually
$ deactivate
$ set PYTHONPATH=Z:\influxdb3-pe\data\plugins\.venv\Lib\site-packages
$ influxdb3-pe\install\influxdb3.exe serve --node-id=local01 --object-store=file --data-dir influxdb3-pe\data --plugin-dir influxdb3-pe\data\plugins

# 'influxdb3.exe install package ...' doesn't work, but plugins can use what was installed manually

[jdstrand]

@jacksonrnewhouse found that updating PATH to include /path/to/python/runtime/python/bin addresses the OSX and Linux issues (including use of --virtual-env-location) and he is creating a PR (thanks!).

That leaves Windows:
Windows works less well, but can be made to work. The server must by launched (assumes zip unpacked at influxdb3-pe\install):

# venv isn't setup automatically currently, so do that
$ influxdb3-pe\install\python\python.exe -m venv influxdb3-pe\data\plugins\.venv
# 'influxdb3.exe install package ...' doesn't work, so activate the venv and install it manually
$ influxdb3-pe\data\plugins\.venv\Scripts\activate
(.venv) $ python -m pip install requests
...

# neither --virtual-env-location nor launching under an activated venv works, so deactivate and
# set PYTHONPATH manually
$ deactivate
$ set PYTHONPATH=Z:\influxdb3-pe\data\plugins\.venv\Lib\site-packages
$ influxdb3-pe\install\influxdb3.exe serve --node-id=local01 --object-store=file --data-dir influxdb3-pe\data --plugin-dir influxdb3-pe\data\plugins

# 'influxdb3.exe install package ...' doesn't work, but plugins can use what was installed manually