What would be the best way to offer a suite of unit tests for testing a plugin?

[couling]

For a plugin framework, I'd like to offer a suit of unit tests which could be used to test third party plugins. This makes sense where the plugin is offering to be a provider (eg providing access to a remote file system). The framework that uses that provider plugin may have strict requirements around behaviour and it would be good to offer unit tests that can test any given provider against those requirements.

For single unit tests I believe that fixtures offer some flexability here because of the way that fixtures are assembled in context of the current module.

Eg a test could be written as:

def test_does_thing(provider):
    assert provider.does_thing() == "Expected Result"

Then, to use the test, a third party provider could do this in their own unit tests:

import pytest
from my_code import Provider

from plugin_framework.unit_tests import test_does_thing  # 👈 Inject a unit test from the framework here

@pytest.fixture()
def provider() -> Provider:
    return Provider()

---

Is there is a more appropriate way to do this or a slicker way to offer entire test suits?

[RonnyPfannschmidt]

There currently is no built-in mechanism for using/integrating test suites from packages

[nicoddemus]

What I have done and works well is to provide a base class, containing the skeleton for the fixtures and the tests that I want to distribute, for example:

# contents of plugin_framework.testing.standard_tests
class StandardTests:    

    __test__ = False

    @pytest.fixture()
    def provider(self) -> Provider:
        # Declaring this fixture here is not strictly required, but good to have 
        # as it stabilishes what subclasses need to implement.
        assert 0, "not implemented"

    def test_does_thing(self, provider):
        assert provider.does_thing() == "Expected Result"    

Then a third-party plugin needs to subclass that class and implement the required fixtures:

# contents of my_code.tests.test_standard_tests
from plugin_framework.testing.standard_tests import StandardTests

from my_code import DirectoryProvider

class Test(StandardTests):

    @pytest.fixture()
    def provider(self, tmp_path) -> Provider:
        return DirectoryProvider(tmp_path)        

Now pytest can collect Test and execute the tests using the implementation of the fixture as provided by the third-party plugin. Note that the neat thing is that the provider fixture can request other fixtures, regardless of how provider was declared in the StandardTests class.