rerun-if-changed doesn't trigger if a symlink points to a different file with the same mtime

[ngoldbaum]

Problem

Last week I ran into crashes working on the Python bcrypt project, which uses Rust via PyO3. Ultimately I tracked the crash down to the Rust build cache not being invalidated correctly while switching Python interpreters. See this PyO3 issue for more context.

Steps

Clone this repository and run the steps in the README.

The repository sets up symlinks in a manner analogous to the problematic symlinks that caused issues on the bcrypt CI. In the real-world case the two python symlinks were pointing at interpreters with different filesystem paths but happened to have the same mtime.

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.84.1 (66221abde 2024-11-19)
release: 1.84.1
commit-hash: 66221abdeca2002d318fde6efff516aab091df0e
commit-date: 2024-11-19
host: aarch64-apple-darwin
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.7.1 (sys:0.4.74+curl-8.9.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Mac OS 15.2.0 [64-bit]

[epage]

rerun-if-changed compares mtimes of the specified file (or files within a specified directory).

Were you able to confirm if the mtime's changed?

[epage]

Are you able to simplify the reproduction steps to not include third-party tools? At least for myself, I'd rather not go through the trouble of setting up the needed nox environment (and making sure I get it correct).

[ngoldbaum]

Were you able to confirm if the mtime's changed?

Yes, the files have different mtimes:

$ cargo build
   Compiling test-rerun-if-changed v0.1.0 (/Users/goldbaum/Documents/test-rerun-if-changed)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.51s

$ ls -lh .nox/test/bin/python
lrwxr-xr-x@ 1 goldbaum  staff    53B Feb  3 08:53 .nox/test/bin/python -> /Users/goldbaum/.pyenv/versions/3.13.1/bin/python3.13

$ rm -r .nox

$ nox
nox > Running session test
nox > Creating virtual environment (virtualenv) using python3.13 in .nox/test
nox > python -m pip install pytest
nox > Session test was successful.

$ ls -lh .nox/test/bin/python
lrwxr-xr-x@ 1 goldbaum  staff    53B Feb  3 08:54 .nox/test/bin/python -> /Users/goldbaum/.pyenv/versions/3.13.1/bin/python3.13

$ cargo build
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.02s

Are you able to simplify the reproduction steps to not include third-party tools? At least for myself, I'd rather not go through the trouble of setting up the needed nox environment (and making sure I get it correct).

Sure I can try.

[epage]

I wonder if we are checking the mtime for what the symlink resolves to.

[ngoldbaum]

I wonder if we are checking the mtime for what the symlink resolves to.

But then why does running touch on the symlink cause a rebuild?

[ngoldbaum]

I wonder if we are checking the mtime for what the symlink resolves to.
But then why does running touch on the symlink cause a rebuild?

Oh it's because touch dereferences the symlink.

[ngoldbaum]

Thanks for the hint, I think this definitely is because of symlinks. I updated the test repo to use shell commands to set up the test repository.

[ngoldbaum]

Hmm although in the problematic case that crashed the bcrypt CI the symlink was deleted and recreated to point at a different file, so not quite what my test is demonstrating. I'll edit it to hopefully make it clear what's happening in the real problematic case.

[epage]

That depends on what the mtimes were for the files that were being pointed at.

If we had checksum support for build scripts, that would at least avoid the problem but that has its own list of problems that I'm unsure whether its worth doing.

[ngoldbaum]

OK, I think I understand what's happening. It is checking the mtime of the file the symlink points at, but not checking the name or path of the file it points at. I managed to trigger it by creating a symlink with the same name that points to /bin/sh, building, blowing away the symlink, then creating a new one with the same name that points to /bin/cp. See the latest version of the test repo.

[weihanglo]

Thanks for updating the test repo!

It is checking the mtime of the file the symlink points at, but not checking the name or path of the file it points at.

I am not sure if checking the name or path is the correct way to fix it. It may depend on how we define "the same file". It may fail if two symilnks target are actaully two hardlinks to the same file.

The same-file crate Cargo uses in some places defines it if two files share the device ID and inode on Unix. The nightly -Zchecksum-freshness checks content length and content checksum but not name nor path.