_pyproject_wheeldir of "build" is way too generic

[s-t-e-v-e-n-k]

Now that we have a module named "build", some of the (ahem) build machinery for it is getting extremely confused by multiple directories matching it. We should probably pick something way less generic.

[dirkmueller]

I ran into the same issue while trying to resolve setuptools update / build update build issues in factory.

[s-t-e-v-e-n-k]

I was also trying to update build, and I added %define _pyproject_wheeldir project-build to the spec file, which gets it building, but then it fails when building tests.

[danigm]

I tried to build without modifying the python-rpm-macros, that's what worked for me in the python-build.spec file:

%build
# Can't use %%pyproject_wheel because the module name is build and
# conflicts with the temp build folder created in that macro
%{python_expand #
rm -rf project-build
mkdir -p project-build
$python -mpip wheel --verbose --progress-bar off --disable-pip-version-check --use-pep517 --no-build-isolation --no-deps --wheel-dir ./project-build .
mv project-build _build.%{$python_prefix}
}

[bnavigator]

Are you talking about %_pyproject_wheeldir, which just tells %pyproject_wheel where to place the built wheels and %pyproject_install where to look for it, or about the build/ directory which gets shuffled around in between all flavor expansions? The latter predates the %_pyproject_wheeldir by years. Note that cmake uses the same directory name, and other build tools probably too.

[s-t-e-v-e-n-k]

[   94s] _________________________ test_build_wheel[from_sdist] _________________________
[   94s] [gw4] linux -- Python 3.8.16 /usr/bin/python3.8
[   94s] 
[   94s] monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f9c11850b80>
[   94s] tmpdir = local('/tmp/pytest-of-abuild/pytest-15/popen-gw4/test_build_wheel_from_sdist_0')
[   94s] args = ()
[   94s] 
[   94s]     @pytest.mark.parametrize('args', ((), ('--wheel',)), ids=('from_sdist', 'direct'))
[   94s]     def test_build_wheel(monkeypatch, tmpdir, args):
[   94s]     
[   94s]         monkeypatch.chdir(MAIN_DIR)
[   94s]     
[   94s] >       subprocess.run(
[   94s]             [
[   94s]                 sys.executable,
[   94s]                 '-m',
[   94s]                 'build',
[   94s]                 *args,
[   94s]                 '--outdir',
[   94s]                 str(tmpdir),
[   94s]             ],
[   94s]             check=True,
[   94s]         )
[   94s] 
[   94s] tests/test_self_packaging.py:71: 
[   94s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   94s] 
[   94s] input = None, capture_output = False, timeout = None, check = True
[   94s] popenargs = (['/usr/bin/python3.8', '-m', 'build', '--outdir', '/tmp/pytest-of-abuild/pytest-15/popen-gw4/test_build_wheel_from_sdist_0'],)
[   94s] kwargs = {}, process = <subprocess.Popen object at 0x7f9c11850a60>
[   94s] stdout = None, stderr = None, retcode = 1
[   94s] 
[   94s]     def run(*popenargs,
[   94s]             input=None, capture_output=False, timeout=None, check=False, **kwargs):
[   94s]         """Run command with arguments and return a CompletedProcess instance.
[   94s]     
[   94s]         The returned instance will have attributes args, returncode, stdout and
[   94s]         stderr. By default, stdout and stderr are not captured, and those attributes
[   94s]         will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
[   94s]     
[   94s]         If check is True and the exit code was non-zero, it raises a
[   94s]         CalledProcessError. The CalledProcessError object will have the return code
[   94s]         in the returncode attribute, and output & stderr attributes if those streams
[   94s]         were captured.
[   94s]     
[   94s]         If timeout is given, and the process takes too long, a TimeoutExpired
[   94s]         exception will be raised.
[   94s]     
[   94s]         There is an optional argument "input", allowing you to
[   94s]         pass bytes or a string to the subprocess's stdin.  If you use this argument
[   94s]         you may not also use the Popen constructor's "stdin" argument, as
[   94s]         it will be used internally.
[   94s]     
[   94s]         By default, all communication is in bytes, and therefore any "input" should
[   94s]         be bytes, and the stdout and stderr will be bytes. If in text mode, any
[   94s]         "input" should be a string, and stdout and stderr will be strings decoded
[   94s]         according to locale encoding, or by "encoding" if set. Text mode is
[   94s]         triggered by setting any of text, encoding, errors or universal_newlines.
[   94s]     
[   94s]         The other arguments are the same as for the Popen constructor.
[   94s]         """
[   94s]         if input is not None:
[   94s]             if kwargs.get('stdin') is not None:
[   94s]                 raise ValueError('stdin and input arguments may not both be used.')
[   94s]             kwargs['stdin'] = PIPE
[   94s]     
[   94s]         if capture_output:
[   94s]             if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
[   94s]                 raise ValueError('stdout and stderr arguments may not be used '
[   94s]                                  'with capture_output.')
[   94s]             kwargs['stdout'] = PIPE
[   94s]             kwargs['stderr'] = PIPE
[   94s]     
[   94s]         with Popen(*popenargs, **kwargs) as process:
[   94s]             try:
[   94s]                 stdout, stderr = process.communicate(input, timeout=timeout)
[   94s]             except TimeoutExpired as exc:
[   94s]                 process.kill()
[   94s]                 if _mswindows:
[   94s]                     # Windows accumulates the output in a single blocking
[   94s]                     # read() call run on child threads, with the timeout
[   94s]                     # being done in a join() on those threads.  communicate()
[   94s]                     # _after_ kill() is required to collect that and add it
[   94s]                     # to the exception.
[   94s]                     exc.stdout, exc.stderr = process.communicate()
[   94s]                 else:
[   94s]                     # POSIX _communicate already populated the output so
[   94s]                     # far into the TimeoutExpired exception.
[   94s]                     process.wait()
[   94s]                 raise
[   94s]             except:  # Including KeyboardInterrupt, communicate handled that.
[   94s]                 process.kill()
[   94s]                 # We don't call process.wait() as .__exit__ does that for us.
[   94s]                 raise
[   94s]             retcode = process.poll()
[   94s]             if check and retcode:
[   94s] >               raise CalledProcessError(retcode, process.args,
[   94s]                                          output=stdout, stderr=stderr)
[   94s] E               subprocess.CalledProcessError: Command '['/usr/bin/python3.8', '-m', 'build', '--outdir', '/tmp/pytest-of-abuild/pytest-15/popen-gw4/test_build_wheel_from_sdist_0']' returned non-zero exit status 1.
[   94s] 
[   94s] /usr/lib64/python3.8/subprocess.py:516: CalledProcessError
[   94s] ----------------------------- Captured stdout call -----------------------------
[   94s] * Creating venv isolated environment...
[   94s] * Installing packages in isolated environment... (flit-core >= 3.4)
[   94s] * Getting build dependencies for sdist...
[   94s] 
[   94s] ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist
[   94s] ----------------------------- Captured stderr call -----------------------------
[   94s] Traceback (most recent call last):
[   94s]   File "/usr/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
[   94s]     main()
[   94s]   File "/usr/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
[   94s]     json_out['return_val'] = hook(**hook_input['kwargs'])
[   94s]   File "/usr/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 287, in get_requires_for_build_sdist
[   94s]     return hook(config_settings)
[   94s]   File "/tmp/build-env-v8xpqmib/lib64/python3.8/site-packages/flit_core/buildapi.py", line 31, in get_requires_for_build_wheel
[   94s]     module = Module(info.module, Path.cwd())
[   94s]   File "/tmp/build-env-v8xpqmib/lib64/python3.8/site-packages/flit_core/common.py", line 50, in __init__
[   94s]     raise ValueError(
[   94s] ValueError: Multiple files or folders could be module build: /home/abuild/rpmbuild/BUILD/build-0.10.0/build, /home/abuild/rpmbuild/BUILD/build-0.10.0/src/build

[bnavigator]

• https://docs.python.org/3/library/sys_path_init.html#pth-files
• https://docs.python.org/3/using/cmdline.html#cmdoption-P
• https://docs.python.org/3/using/cmdline.html#envvar-PYTHONSAFEPATH

[s-t-e-v-e-n-k]

New in version 3.11. which won't help us for 3.8

[s-t-e-v-e-n-k]

@bnavigator But also, cmake isn't trying to build a module called 'build', which is more the problem here.

I've tracked down the test failures to %python_flavored_alternatives, which is separate from this issue, but still related, since it hardcodes mkdir -p build/flavorbin and then flit-core dies with the above ValueError.

[bnavigator]

But there are python modules building with cmake as backend

[bnavigator]

since it hardcodes mkdir -p build/flavorbin and then flit-core dies with the above ValueError.

It merely reuses the hardcoded shuffle dir:

python-rpm-macros/macros.in

Lines 12 to 26 in 050c1a4

	%_python_use_flavor() \
	last_flavor=`[ -f _current_flavor ] && cat _current_flavor || true` \
	if [ -z "$last_flavor" ]; then last_flavor="tmp"; fi \
	if [ "$last_flavor" != "%1" ]; then \
	if [ -d build ]; then mv build _build.$last_flavor; fi \
	if [ -d _build.%1 ]; then mv _build.%1 build; fi \
	fi \
	echo %1 > _current_flavor \
	python_flavor=%1 \
	%{nil}
	%_python_stash_flavor() \
	if [ -d build ]; then mv build _build.%1; fi \
	if [ -d _build.tmp ]; then mv _build.tmp build; fi \
	%{nil}

Note the the creation of the build/ folder traditionally comes from python3 setup.py build ... and python3 -m pip wheel .... It is completely independent of python-rpm-macros.

[bnavigator]

... isn't trying to build a module called 'build', which is more the problem here.

And how often is that a problem so that it warrants a general change in the macros?

[bnavigator]

SR#1073633

[mcepl]

Has this been fixed by sr#1073633?

[s-t-e-v-e-n-k]

I'd consider that a workaround, but I'm not certain @bnavigator and I are going to agree.

[bnavigator]

I still believe that this is a wontfix issue.