Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combining nextest, llvm-cov and delayed execution #1036

Closed
MiniaczQ opened this issue Oct 13, 2023 · 14 comments
Closed

Combining nextest, llvm-cov and delayed execution #1036

MiniaczQ opened this issue Oct 13, 2023 · 14 comments

Comments

@MiniaczQ
Copy link

MiniaczQ commented Oct 13, 2023

Hello, I'm investing how I can create an optimal integration test environment and that includes code coverage and junit compatible test logs, while efficiently using build cache.
I believe cargo-llvm-cov combined with nextest it possible, but I'm having some trouble executing it and I was hoping someone could help me out here.

FROM rust:1.73-bookworm as build
WORKDIR /app/

RUN rustup component add llvm-tools-preview
RUN curl -L --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/cargo-bins/cargo-binstall/main/install-from-binstall-release.sh | bash
RUN cargo binstall cargo-nextest --secure -y
RUN cargo binstall cargo-llvm-cov --secure -y

COPY . .
# Skipped cache mounting, that works fine
# Prepare build flags for coverage, save it in a file to execute in each RUN statement
RUN echo $(cargo llvm-cov show-env --export-prefix) >> ./envvars.sh 
RUN chmod +x ./envvars.sh
# Build all dependencies with the coverage flags and nextest
RUN ./envvars.sh && cargo nextest run --no-run

FROM build as unit-test
WORKDIR /app/
# Everything is already built, just export the correct tests to an archive
RUN ./envvars.sh && cargo nextest archive --archive-file test.tar.zst --lib --bins
ENTRYPOINT ./tests/run-report.sh

The run-report.sh

set -e
# Run tests from the archive
./envvars.sh && cargo nextest run --archive-file test.tar.zst
# Use generated profraws for report generation
cargo llvm-cov report --html

The issue I'm running into is that the profraws aren't being generated and I believe this has to do with the nextest building process.
Any ideas?

@MiniaczQ
Copy link
Author

Don't get confused by the example, I named the target unit-test, but I just treat it as special case integration tests with no external services.

@taiki-e
Copy link
Contributor

taiki-e commented Oct 13, 2023

echo $(cargo llvm-cov show-env --export-prefix) >> ./envvars.sh 

IIRC, this echo converts the output from show-env to a space-separated single line, which will result in environment variables being set incorrectly. I think you have to use cargo llvm-cov show-env --export-prefix >> ./envvars.sh

@MiniaczQ
Copy link
Author

You're correct, I tried the second one and it suffers from the same issue

@taiki-e
Copy link
Contributor

taiki-e commented Oct 13, 2023

./envvars.sh && 

envvars.sh internally call export NAME=VAL, so I think you have to load (source ./envvars.sh or . ./envvars.sh) it instead of run it.

@MiniaczQ
Copy link
Author

source doesn't exist in whichever shell RUN is using, so I left it early, but . seems to at least go through, I'm checking whether it exports properly

@MiniaczQ
Copy link
Author

Export works, no profraws though :/

@taiki-e
Copy link
Contributor

taiki-e commented Oct 13, 2023

set -e
# Run tests from the archive
./envvars.sh && cargo nextest run --archive-file test.tar.zst
# Use generated profraws for report generation
cargo llvm-cov report --html

@taiki-e
Copy link
Contributor

taiki-e commented Oct 13, 2023

Oh, that's about the builtin nextest subcommand, so that may not work for show-env as-is. (That said, the underlying problem should probably be the same.)

@MiniaczQ
Copy link
Author

I'm not sure I understand the issue, profraws should be generated during compilation and they don't get packaged with nextest archive? so tests don't generate anything?

  • cargo llvm-cov report also needs to reference the environment variables from show-env.

Added it, but I checked and there are just no .profraw's after the test run

@MiniaczQ
Copy link
Author

MiniaczQ commented Oct 13, 2023

Would moving the file by hand potentially fix it?
No files are generated during compilation with nextest it seems

@taiki-e
Copy link
Contributor

taiki-e commented Oct 13, 2023

profraws should be generated during compilation

No, binary (or cdylib) will generate *.profraw file on exit. See also taiki-e/cargo-llvm-cov#320 (comment).

no profraws

Does this mean it cannot be found using find or fd -I? Or does it mean that cargo llvm-cov report cannot find it?

In the latter case, I guess the problem is that the target directory used by --archive-file is different from cargo's default, just like the problem that occurred in the linked PR, so I think it can be avoided by adjusting CARGO_LLVM_COV_TARGET_DIR (maybe adding /target at last) before calling cargo llvm-cov report.

@MiniaczQ
Copy link
Author

Docker desktop highlights changed and new files in a container, they just don't exist.
There are only my tmp files for tests and this:
image

@MiniaczQ
Copy link
Author

Here, I made a repro repository, the problem is in the issue
https://github.com/MiniaczQ/nextext-llvm-cov-repro

@MiniaczQ
Copy link
Author

MiniaczQ commented Oct 13, 2023

Solved thanks to a friend, repro contains the solution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants