Improvements to Testing

[dlstadther]

Problem

Luigi's test suites are difficult to get setup on a new local machine and the documentation is not sufficient to get up and running quickly.

While setting up Luigi on a new computer to aid in PR review (in order to run tests locally), I discovered that documentation is lacking on a recommended way to run the tox tests on a user's machine. After installing a desired python version and tox, I was unable to get all tests to run and complete successfully.

Currently, all the local development documentation is limited to the CONTRIBUTING.rst document. However, the content there is insufficient to getting up and running in 2024. (At least on an Apple Silicon Mac).

Resolution Proposal

I intend to use this Issue to track and discuss a route forward to improve testing the luigi project.

Goals include:

• straightforward setup for developer
• deterministic test execution, regardless of host compute environment
• alignment between local test execution and CI test execution

Additional aspirations which may be tackled separately include:

• ensure all tests are run through CI (a few tox test suites are currently ignored)
  • (cdh, dropbox, gcloud, hdp, and visualiser are not run by CI)
• improve speed of test execution
• cleanup verbosity of test execution logs (lots of warnings and other console output)

[dlstadther]

pyxx-cdh and pyxx-hdp test suites are broken due to dependence on legacy hortonworks and cloudera artifacts which are no longer available or are now behind a paywall

[dlstadther]

The visualiser test suite requires PhantomJS to be installed; however, even after brew install phantomjs, tests fail.

This test suite isn't ran in CI, so I'm not yet sure if the issue is user error or broken tests.

[dlstadther]

The following test suites require actual credentials:

• pyxx-dropbox
• pyxx-gcloud

The following subset of tests require actual credentials:

• pyxx-aws test/contrib/batch_test.py (the tests in this file are currently excluded from CI execution)

[dlstadther]

There are a few other tests which include the @skipOnTravisAndGithubActions decorator which attempt to run locally and fail.

[dlstadther]

Attempted local setup

pyenv local 3.11.10
python -m venv .venv
.venv/bin/python -m pip install "tox<4.0"

# tox.ini needs to be updated such that flake8 ignores .venv/
.venv/bin/tox -e flake8
_____________________________________________________ summary _____________________________________________________
  flake8: commands succeeded
  congratulations :)


.venv/bin/tox -e docs
_____________________________________________________ summary _____________________________________________________
  docs: commands succeeded
  congratulations :)


.venv/bin/tox -e py311-apache
==================== 11 failed, 56 passed, 11 skipped, 1740 deselected, 121 warnings in 23.24s ====================


.venv/bin/tox -e py311-contrib
==================== 13 failed, 67 passed, 11 skipped, 1727 deselected, 82 warnings in 34.50s =====================


.venv/bin/tox -e py311-postgres
========================== 6 passed, 11 skipped, 1801 deselected, 37 warnings in 18.56s ===========================
_____________________________________________________ summary _____________________________________________________
  py311-postgres: commands succeeded
  congratulations :)


.venv/bin/tox -e py311-unixsocket
================ 5 failed, 11 skipped, 1802 deselected, 36 warnings, 1 error in 229.69s (0:03:49) =================

So the simple virtualenv with tox<4.0 installed does ok for some of the suites, but not for others.

My first assumption here is that the CI tests benefit from being run on ubuntu environments (and possibly some existing system packages too).

To verify this assumption, I will create a docker image which can run tests.

[dlstadther]

See this branch for drafted changes to support local execution of Luigi tests, using docker.

Execution flexibility:

# default python version and lint test
make test
# default python version and specific tox suite
make test CMD="tox -e doc"
# specific python version and specific tox suite
make test TAG=3.10 CMD="tox -e visualiser"
# specific python version and specific tox test from suite
make test TAG=3.10 CMD="tox -e py311-core test/mypy_test.py::TestMyMypyPlugin::test_plugin_no_issue"

Tox test suite support (initially running with Python 3.11):

• doc
• flake8
• core
  • a few failures on tests which otherwise have @skiponTravisAndGithubActions decorator
• unixsocket
• visualiser
  • requires phantomjs to be installed
  • tests still failing for unknown reasons after installing phantomjs (not yet commited)
• contrib
• apache
• aws
  • all pass EXCEPT test/contrib/batch_test.py
• azureblob
  • 3 tests fail, 3 test pass inside of docker, unsure why yet
• cdh
  • cdh downloads are now behind a paywall. the currently configured tar.gz cannot be accessed
• contrib
• dropbox
  • requires valid dropbox oauth2 token
• gcloud
  • requires gcloud credentials
• hdp
  • hdp downloads are now behind a paywall. the currently configured tar.gz cannot be accessed
• postgres