Better way to address memory leak with pytest logging

[tjsmart]

We came across an interesting issue at work where a users test scripts were running out of memory. We root caused the issue to be due to:

1. An overabundance of logging in one of our tools.
2. pytest's caplog.handler holding on to a record of every logged message.

We have limited 1, but 2 still presents a problem as our tests are intended to create large amounts of logs as a single test could run as long as multiple days (we're in the drive industry so the whole point is to make sure we can remain stable over long periods of time).

---

I was able to create a MWE which demonstrates the issue we run into (save the contents to a file test.py):

import logging

def test_chatterbox():
    while True:
        logging.debug("hello")

Then run this using pytest test.py --log-file-level DEBUG.

If you then monitor the memory usage of this process (I used htop to manually track it), it will grow about ~20MB/sec. Eventually the process gets killed by the OS but I suggest not waiting that long before killing it. 😁

---

Checking pytest's issues forum I find that some similar issues have popped up and suggest -p no:logging. While this fixes the memory issue, it also eliminates any logging that would be written to the file (we'd prefer to use pytest's logging configuration as opposed to turning to python stdlib's API or an alternate tool).

I think what I need is to just disable the caplog.handler from holding onto any records.

One workaround I've attempted is to replace caplog.handler.records with a collections.deque with a reasonable maxlen (or zero). For example:

import logging
from collections import deque

import pytest

def test_chatterbox(caplog: pytest.LogCaptureFixture):
    caplog.handler.records = deque(maxlen=0)
    while True:
        logging.debug("hello")

This works in some cases but in this MWE we'll still see a memory leak. Last thing I have tried is adding gc.collect:

import gc
import logging
from collections import deque

import pytest

def test_chatterbox(caplog: pytest.LogCaptureFixture):
    caplog.handler.records = deque(maxlen=0)
    while True:
        logging.debug("hello")
        gc.collect()

This final form removes the memory leak while keeping the logging to a file in-tact, but it is obviously an impractical solution.

I'm hoping someone smarter and more familiar with the pytest framework can help point me in the right direction. 🙏

---

Additional details:

$ python --version
Python 3.10.13
$ pytest --version
pytest 7.4.3
$ lsb_release --description --short
Ubuntu 20.04.6 LTS

[tjsmart]

It turns out that deque + gc.collect didn't fix the memory leak, if I instead call gc.collect every 5 seconds (as below) the process still steadily increases in memory usage:

import gc
import logging
import time
from collections import deque

import pytest


def test_foo(caplog: pytest.LogCaptureFixture):
    caplog.handler.records = deque(maxlen=0)  # type: ignore

    gc_time = time.monotonic() + 5
    while True:
        if time.monotonic() > gc_time:
            gc.collect()
            gc_time = time.monotonic() + 5

        logging.debug(f"hello: {len(caplog.handler.records)=}")

So perhaps caplog.handler.records isn't the culprit.

[tjsmart]

I think this is related to #8307.