Differences in how modules are imported between running pytest on a directory or a file

[Kodiologist]

We use pytest for Hy, both for the compiler's own tests and for tests of other Hy programs. Mostly, this works well, but one flaw I've noticed is that calling pytest on a specific file, like pytest tests/native_tests/with.hy, attempts to interpret the file as Python instead of Hy, which fails. By contrast, calling pytest alone, or calling it on a directory of Hy code, works fine. It's quite possible that the underlying issue is in the code that Hy uses to twist the import system for its own purposes. But, I wonder if pytest imports files differently in these two cases—calling pytest on a directory and calling it on a file—in a way that's significant here.

Here's a minimal example:

mkdir -p tests

echo >tests/__init__.py '
import hy'

echo >tests/test_foo.hy '
(defn test_bar []
    (assert (= (+ 1 1) 2)))'

echo >conftest.py '
import hy, pytest
def pytest_collect_file(file_path, parent):
    if file_path.suffix == ".hy":
        return pytest.Module.from_parent(parent, path=file_path)'

You should see that pytest and pytest tests run the one test successfully, and python3 -c 'import tests.test_foo; tests.test_foo.test_bar()' also works, but pytest tests/test_foo.hy fails during collection with a SyntaxError:

====================================== ERRORS =======================================
_______________________ ERROR collecting tests/test_foo.hy _________________________
/home/hippo/Scratch/Python-venv/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/home/hippo/Scratch/Python-venv/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/home/hippo/Scratch/Python-venv/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:177: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
/home/hippo/Scratch/Python-venv/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:359: in _rewrite_test
    tree = ast.parse(source, filename=strfn)
/usr/lib/python3.11/ast.py:50: in parse
    return compile(source, filename, mode, flags,
E     File "/tmp/chippy/tests/test_foo.hy", line 2
E       (defn test_bar []
E        ^^^^^^^^^^^^^
E   SyntaxError: invalid syntax. Perhaps you forgot a comma?
============================== short test summary info ==============================
ERROR tests/test_foo.hy
!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!

[bluetech]

The reason is this: pytest uses "assertion rewriting" as a way to implement its assert magic. It rewrites test modules at the AST level and loads the rewritten AST instead of the original. The python import system doesn't have an "AST transform pipeline" concept, only "find" (path -> module spec) and "load" (module spec -> module object). So the flow is this (pytest part in [...]):

path -> module spec -> hy -> module object -> [path -> ast -> ast rewrite -> module object]

The "[path -> ast]" (ast.parse()) part can't work because the file is hy syntax not python syntax. There is no way to go "module object -> ast" without reparsing the file.

OK so pytest's assertion rewriting feature seemingly can't work with hy, so how come pytest tests works? The answer to this lies in how pytest decides which files to rewrite in the first place:

https://github.com/pytest-dev/pytest/blob/52db918a27b2eb5043de6e80215076a98b0b9fff/src/_pytest/assertion/rewrite.py#L216-L235C56

"modules not passed explicitly on the command line are only rewritten if they match the naming convention for test files". The default value of the python_files option is test_*.py *_test.py which doesn't match test_foo.hy. However if you were to set python_files = test_*.py test_*.hy you'd see the error also in this case.

I can't think of any good solutions for you other than turning off assertion rewriting (--assert=plain).

[Kodiologist]

Thanks for the thorough explanation. --assert=plain works fine considering that assertion rewriting isn't going to work for Hy anyway.

Maybe there should be a configuration option that disables assertion rewriting for files not matched by python_files even if the file is explicitly listed on the command line.

[bluetech]

Maybe there should be a configuration option that disables assertion rewriting for files not matched by python_files even if the file is explicitly listed on the command line.

I do think we should provide some way to handle this, however a specific configuration option just for this seems too specific. I think perhaps it should come from the plugin/conftest not from the user, but haven't thought too much about it.