allocation tests fail with coverage=true after dfe6a13e5038c8cbe0f1720d190629225ec1a19b

[lmiq]

See discussion at: https://discourse.julialang.org/t/tests-fail-on-ci-pass-locally/125140/11

After this commit, allocation tests in packages started failing on CI runs because, with Pkg.test(;coverage=true), allocations appear where with coverage=false they do not.

My simple example of such behavior (used to bisect the commit is):

import Pkg
Pkg.add(url="https://github.com/lmiq/MeuNovoPacote.jl")
Pkg.test("MeuNovoPacote"; coverage=true)

This implied that CI tests that passed until v1.11.2 started failing on v1.11.3 and later versions.

The actual test that is being run in that package is:

using PDBTools
using BenchmarkTools
atoms = read_pdb(PDBTools.TESTPDB, "protein")
a = @benchmark sum($mass, $atoms) samples=1 evals=1
@test a.allocs == 0

[Seelengrab]

I think this is conceptually the same thing as #49978 , where effects are influenced by code coverage.

[lmiq]

Yes probably. Is there a workaround? Some way to get the coverage report but in parallel run CI tests with the same compiler options that we use for production? (and thus, the ones which we test locally)

[nsajko]

Is there a workaround? Some way to get the coverage report but in parallel run CI tests with the same compiler options that we use for production? (and thus, the ones which we test locally)

FixedSizeArrays works around this issue by splitting CI into coverage-enabled and coverage-disabled jobs. CI informs the test suite that "the build is a production build" when coverage is disabled, thus enabling the no-allocation tests:

The test suite needs to parse the environment variable:

https://github.com/JuliaArrays/FixedSizeArrays.jl/blob/d17373edf4144a672cd80a062bf24d017f01e42f/test/runtests.jl#L6-L17

The CI is configured like so:

https://github.com/JuliaArrays/FixedSizeArrays.jl/blob/d17373edf4144a672cd80a062bf24d017f01e42f/.github/workflows/UnitTests.yml#L16-L77

Perhaps it'd be good if a Github Action was available to make this easier.