LLVM coverage instrumentation

[xd009642]

So when llvm coverage instrumentation is stable on rust we can look at removing the ptrace based coverage completely and instead relying on llvm and rustc to help. This has a few advantages

1. None of the weirdness of ptrace
2. More accurate results
3. First class support in the compiler - less chance of behaviour changing in nightly
4. Linker issues, large target folders and slowness should be greatly mitigated
5. Easier route to get branch/condition and other coverage metrics by coupling llvm_lines and source analysis

In this instance tarpaulin should make the whole experience still nicer because:

1. Handles calling cargo and right flags
2. Doc tests coverage without needing to remember the nightly flags
3. results collation/report generation
4. tagging unused generics that won't be instrumented

OSX TODOs:

• doc_coverage::doc_test_coverage
• doc_coverage::doc_test_env
• doc_coverage::doc_test_panics
• doc_coverage::doc_test_panics_workspace
• failures::error_if_test_fails
• follow_exes_down
• handle_forks
• issue_966_follow_exec
• kill_used_in_test
• match_expr_coverage
• rustflags_handling
• test_types::only_test_coverage

Windows TODOs

• failures::error_if_test_fails
• follow_exes_down
• issue_966_follow_exec
• match_expr_coverage
• rustflags_handling
• picking_up_shared_objects
• doc_coverage::doc_test_panics
• doc_coverage::doc_test_panics_workspace

Linux TODOs

• doc_coverage::doc_test_panics
• failures::error_if_test_fails
• handle_forks
• follow_exes_down
• issue_966_follow_exec
• match_expr_coverage
• rustflags_handling
• test_types::only_test_coverage

General TODO

• Coverage behaviour with loops

[xd009642]

https://hammer-vlsi.readthedocs.io/en/stable/CoverageMappingFormat.html some docs and the rust issue I keep having to find again rust-lang/rust#34701

[caemor]

Is there already a branch where we can try this?

[xd009642]

Not yet, given my current work schedule I'm hoping to have something early January

[xd009642]

Work schedule and covid restrictions came back to bite me I'm likely to be abroad away from my home until march or april until I can return. I plan on starting the work to make it so tarpaulin compiles on other OSs even if it doesn't do anything and then bring in llvm coverage

resources found:
https://opensource.apple.com/source/clang/clang-700.0.72/src/docs/CoverageMappingFormat.rst.auto.html

[xd009642]

So I've merged the first PR, didn't want to be constantly building up a giant PR that becomes hard to merge 😅. This one simply adds a future option (not visible to the user) to set the tracing engine. It has 3 options Auto, Ptrace and llvm (and with #675 I hope to add probe-rs). So currently this is hard set to Ptrace, but if it's set to Auto or llvm and the compiler version is compatible it will build the test binaries with the llvm-cov instrumentation included. If set to llvm and the compiler isn't compatible it prints an error log warning the user the compiler doesn't support it and falls back to ptrace.

There are some issues with doctests marked as should_panic and the llvm coverage but everything else works fine and generates the profdata files for the tests and passes as expected.

Looking at what I can see of the profdata it looks like I have all the info to parse it with the binary and I'd ideally like to do that from the get-go to avoid making the user install the llvm-tools component and using llvm-profdata. Another benefit is it also gives me ability to access the information I need to derive other coverage metrics than line coverage which would be a big future win. I haven't found a source for the format other than the llvm source code which isn't quite a joy to read...

[mikeumus]

💖 Another Sponsor here rooting for #549!

Go @xd009642!