testdir runs are affecting coverage, can I tweak to prevent this?

[joshua-badger]

#8868
While working on the issue I'm asking about in the above link, I am running into an issue with code coverage.

Specifically, when I add a test to my test suite (which uses testdir, specifically testdir.runpytest_subprocess(), all of a sudden my coverage drops on my django settings files.

Is there something I can do to prevent my test from reverting this coverage?

The added test file:

import inspect
from os import pathsep

import pytest

from tests import pytest_common_hooks


def test_nplusone_catches_marked_test(testdir, monkeypatch):
    test_contents = """
import logging
import pytest

log = logging.getLogger("nplusone")

def test_single_nplusone_catches():
    # Expect this test to pass but err
    log.warning("Potential n+1 query detected on `Model.column`")

def test_multiple_nplusone_queries_catch():
    # Expect this test to pass but err
    log.warning("Potential n+1 query detected on `Model.column`")
    log.warning("Potential n+1 query detected on `Model.column`")

@pytest.mark.allow_nplusone(column="Model.column", count=3)
def test_marked_allowed_nplusone_queries_pass():
    # Expect no error here, we marked it sufficiently
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column`")

@pytest.mark.allow_nplusone(column="Model.column2", count=1)
@pytest.mark.allow_nplusone(column="Model.column", count=3)
def test_multiple_marks_allowed_nplusone_queries_pass():
    # Expect no error here, we marked it sufficiently
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column`")
    log.warning("Potential n+1 query detected on `Model.column2`")

@pytest.mark.allow_nplusone(column="Model.column2", count=1)
@pytest.mark.allow_nplusone(column="Model.column", count=3)
def test_multiple_marks_allowed_nplusone_queries_still_err_if_not_sufficiently_accounted_for():
    # Expect this test to pass but err; we didn't sufficiently catch it
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column`")
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column2`")

@pytest.mark.allow_nplusone(column="Model.column", count=3)
def test_multiple_marks_allowed_nplusone_queries_still_err_if_an_extra_nplusone_happens():
    # Expect this test to pass but err; we didn't sufficiently catch it
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column`")
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column2`")
    """

    testdir.makepyfile(test_things_work=test_contents)

    testdir.makeconftest(inspect.getsource(pytest_common_hooks))
    monkeypatch.setenv("PYTHONPATH", "/<redacted>", prepend=pathsep)

    result = testdir.runpytest_subprocess()

    result.assert_outcomes(passed=6, errors=4)  # all the tests pass, but only 4 err

The coverage report output:

----------- coverage: platform linux, python 3.8.2-final-0 -----------
Name                          Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------------------
<redacted>/settings/base.py     144      3      4      2    97%   212->217, 415, 593-594
<redacted>/settings/test.py      20      0      2      1    95%   37->41
<redacted>/urls.py               11     11      2      0     0%   1-30
<redacted>/wsgi.py                4      4      0      0     0%   10-16
-------------------------------------------------------------------------
TOTAL                          5878     18   1086      3    99%

134 files skipped due to complete coverage.
Coverage HTML written to dir /test-output/coverage
Coverage XML written to file /test-output/coverage.xml

FAIL Required test coverage of 100% not reached. Total coverage: 99.67%

coveragerc file:

[run]
omit =
    # These files run in a configuration not present on the test suite.
    <redacted>/celery.py
    <redacted>/settings/*
    <redacted>/urls.py
    <redacted>/wsgi.py
    **/tests/**
    **/migrations/**