rust-covfix

KO | EN

gitlite — search

#python #react #ai #llm #golang #go #machine-learning #rust #deep-learning #linux #cli #kubernetes #ios #openai

(★ 17)

Fix Rust coverage data based on source code

rust-covfix Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# rust-covfix

[![Build Status](https://travis-ci.org/Kogia-sima/rust-covfix.svg?branch=master)](https://travis-ci.org/Kogia-sima/rust-covfix)
[![Build status](https://ci.appveyor.com/api/projects/status/xc9jauk9nah5webf/branch/master?svg=true)](https://ci.appveyor.com/project/Kogiasima/rust-covfix/branch/master)
[![codecov](https://codecov.io/gh/Kogia-sima/rust-covfix/branch/master/graph/badge.svg)](https://codecov.io/gh/Kogia-sima/rust-covfix)
[![Version](https://img.shields.io/crates/v/rust-covfix)](https://crates.io/crates/rust-covfix)
[![docs](https://docs.rs/rust-covfix/badge.svg)](https://docs.rs/rust-covfix)
![GitHub Release Date](https://img.shields.io/github/release-date/Kogia-sima/rust-covfix?label=last%20release)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/Kogia-sima/rust-covfix/blob/master/LICENSE)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)

Rustc is known to report an incorrect coverage for some lines <https://stackoverflow.com/questions/32521800/why-does-kcov-calculate-incorrect-code-coverage-statistics-for-rust-programs>.
`rust-covfix` will read coverage from the file generated by [grcov](https://github.com/mozilla/grcov/), fix it, then outputs the correct coverage.

Though only `lcov` format is supported at current, Another formats are going to be supported in future releases.

## Features

- Compatible with the latest stable/beta/nightly Rust compiler
- Windows/OSX/Linux are all supported
- Lightweight (small dependencies)
- Fast and safe (implemented in Rust language)
- `rust-covfix` is also available with Rust API ([Documentation](https://docs.rs/rust-covfix))
- Show summary of coverage difference.

### Optional features

Optional features are available with cargo's `--features` option. You can specify the features like:

```console
$ cargo install --no-default-features --features "cli lcov"
```

|Feature name|Description|Default?|
|:--:|--|:--:|
|cli|Command Line Interface. This feature is required to build `rust-covfix` executable.|yes|
|lcov|Make LcovParser available|yes|
|noinline|Avoid adding `#cfg[inline]` attribute on function. (deprecated)|no|
|backtrace|Dump backtrace information on every time the error has occured.|no|

## Install

Download the latest release from [GitHub Release Page](https://github.com/Kogia-sima/rust-covfix/releases).

You can also install via `cargo` command.

```console
$ cargo install rust-covfix
```

## How to generate correct code coverage from Rust program?

### 1. Avoid inlining the functions (optional)

It seems that the current version of rustc (1.42) will automatically inline the function that is only called from one place.
This behaviour causes incorrect coverage for your tests.

To avoid this, you have to add `#[inline(never)]` attributes for their functions manually. I recommend defining a new feature flag `coverage` in your crate. In Cargo.toml, append the following lines.

```Cargo.toml
[features]
coverage = []
```

And then, add the attribute like `#[cfg(feature = "coverage", inline(never))]` to the functions which is called from just one location.

```rust
#[cfg(feature = "coverage", inline(never))]
fn foo() {
  // ...
}
```

This will avoid inlining the functions only when you enable `coverage` feature flag.

#### 2. Compile your crate with `-Zprofile` option

In order to generate code coverage with rustc, you must specify `-Zprofile` option.
This option is currently (1.42) unstable and only available from nightly toolchain.

Also, some other flags will be required to generate **correct** coverage. Here is my recommend.

```console
$ export CARGO_INCREMENTAL=0
$ export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort"
```

Then, compile your crate and run tests.

```console
$ cargo test --features coverage
```

#### 3. aggregate code coverage data

Now there are code coverage data in `target/debug/deps` directory.
The next step is aggregating them and convert the format so that `rust-covfix` can read coverage data.

I highly recommend to use [grcov](https://github.com/mozilla/grcov/) to aggregate them. This project is developed by mozilla team and supports the latest Rust toolchains.

Install latest `grcov` and run the following commands from your project root directory.

```console
zip -0 ccov.zip `find . $ -name "YOUR_PROJECT_NAME*.gc*" -o -name "test-*.gc*" $ -print`
./grcov ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov.info
```

where `YOUR_PROJECT_NAME` is your crate name specified in `Cargo.toml`.

#### 4. Fix coverage data using rust-covfix

Now rust-covfix can read coverage from `lcov.info`.

```console
$ rust-covfix -o lcov_correct.info lcov.info
```

This command will write a **correct** coverage into `lcov_correct.info`. You can upload them into codecov.io, or generate HTML summary using `genhtml`.

#### Use rust-covfix on Travis CI

Here is an example script to use `rust-covfix` on Travis CI environment.

```sh
#!/bin/bash
# ci/script.sh

set -ex

if [ "$TRAVIS_RUST_VERSION" = "nightly" ] && [ -z "$TRAVIS_TAG" ]; then
  # Setup grcov
  wget https://github.com/mozilla/grcov/releases/download/v0.5.7/grcov-linux-x86_64.tar.bz2
  tar xvf grcov-linux-x86_64.tar.bz2

# Setup environmental variables
  export CARGO_INCREMENTAL=0
  export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort"
fi

# Compile and run tests
cargo test --all-features

if [ "$TRAVIS_RUST_VERSION" = "nightly" ] && [ -z "$TRAVIS_TAG" ]; then
  # collect coverages
  zip -0 ccov.zip `find . $ -name "YOUR_PROJECT_NAME*.gc*" -o -name "test-*.gc*" $ -print`
  ./grcov ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov.info

# fix coverage using rust-covfix
  rust-covfix lcov.info -o lcov.info

# upload coverage to codecov
  bash <(curl -s https://codecov.io/bash) -f lcov.info
fi
```

Then, call this script from `travis.yml`

```yaml
language: rust

matrix:
  include:
    - os: linux
      rust: nightly

script:
  - bash ci/script.sh
```

#### Use rust-covfix on Gitlab CI

```yaml
codecov:
  stage:                           coverage
  artifacts:
    name:                          "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
    paths:
      - artifacts/
  variables:
    # Variables partly came from https://github.com/mozilla/grcov/blob/master/README.md
    CARGO_INCREMENTAL:             0
    RUSTFLAGS:                     "-Zprofile -Zmir-opt-level=0 -Ccodegen-units=1
                                      -Copt-level=0 -Clink-dead-code -Coverflow-checks=off"
    # We removed the `-Cinline-threshold=0` flag from the above `RUSTFLAGS` since it was bugged
    # at the time and lead to inlining of functions that shouldn't be inlined for the coverage
    # report. (More information here: https://github.com/Kogia-sima/rust-covfix/issues/2)
  script:
    # RUSTFLAGS are the cause target cache can't be used here
    - unset "CARGO_TARGET_DIR" # only needed if it's not default
    - cargo clean
    - cargo test --verbose --all-features --no-fail-fast --workspace
    - cargo build --verbose --all-features --workspace
    # coverage with branches
    - grcov ./target -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov-w-branch.info
    - rust-covfix -o lcov-w-branch_correct.info lcov-w-branch.info
    # We'd like to not use a remote bash script for uploading the coverage reports,
    # however this job seems to be more tricky than we hoped.
    - bash <(curl -s https://codecov.io/bash) -t "$CODECOV_P_TOKEN" -f lcov-w-branch_correct.info
    - cp *.info /artifacts # this is to output the report files, for the debug purposes
```

#### Use rust-covfix on GitHub actions

```yaml
name: Coverage

on: [push, pull_request]

jobs:
  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install toolchain
        uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          target: x86_64-unknown-linux-gnu
          toolchain: nightly
          override: true
      - name: Install grcov
        run: curl -L https://github.com/mozilla/grcov/releases/download/v0.6.1/grcov-linux-x86_64.tar.bz2 | tar jxf -
      - name: Install rust-covfix
        run: |
          curl -L https://github.com/Kogia-sima/rust-covfix/releases/download/v0.2.2/rust-covfix-linux-x86_64.tar.xz |tar Jxf -
          mv rust-covfix-linux-x86_64/rust-covfix ./
      - name: Test all crates
        env:
          CARGO_INCREMENTAL: 0
          RUSTFLAGS: -Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort
          RUSTDOCFLAGS: -C panic=abort
        run: |
          cargo build --all-features --workspace
          cargo test --all-features --workspace
      - name: collect coverages
        run: |
          zip -0 ccov.zip `find . $ -name "YOUR_PROJECT_NAME*.gc*" -o -name "test-*.gc*" $ -print`
          ./grcov ccov.zip --llvm --branch -t lcov -o lcov.info --ignore "/*" --ignore "sailfish-tests/*"
      - name: fix coverages
        run: ./rust-covfix -o lcov.info lcov.info
      - name: upload coverage
        uses: codecov/codecov-action@v1
        with:
          file: ./lcov.info
```

## Why is this project developed as a standalone package?

There are 3 reasons for this.

1. A grcov collaborator is aware of rustc reporting incorrect coverage <https://github.com/mozilla/grcov/issues/249#issuecomment-465154051>. I'm trying to compare original Rust code and transformed LLVM IR, but still don't fully understand why Rustc report incorrect coverage. The `rust-covfix` was originally developed as a temporal workaround for this problem until I understand when and where rustc reports incorrect coverage. It is not completely unravelled yet.

2. `grcov` is just a coverage collection tool, not a coverage generator. It only aggregates the raw coverage data, and convert the format. It does not read the source code, nor manipulate any coverage data. Also, although grcov is targetting many programming languages such as C/C++, Nim and Rust, `rust-covfix` only supports Rust.

3. `rust-covfix` is targetting not only grcov, but also [cargo-kcov](https://github.com/kennytm/cargo-kcov) and [Tarpaulin](https://github.com/xd009642/tarpaulin) in the future. Actually, `Tarpaulin` can now generate the `lcov.info`. It means `rust-covfix` is already able to fix the coverage data generated by `Tarpaulin`. I'm working on [feature/cobertura branch](https://github.com/Kogia-sima/rust-covfix/tree/feature/cobertura) to support `cargo-kcov` too. The final objective is to support these 3 tools.

## How is the incorrect line coverage detected?

`rust_covfix` fixes the coverage information using some rules. You can pass `--rules` option to specify which rules are used to fix coverages.

### Rules

#### close

closing brackets, line of `else` block will be ignored.

```rust
if a > 0 {
    b = a
} else {  // <-- marked as "not executable"
    b = -a
};  // <-- marked as "not executable"
```

#### test

module block named `test` or `tests` which has attribute `cfg(test)` will be ignored.

all functions with `#[test]` attribute will be also ignored.

```rust
#[cfg(test)]
mod tests {  // <-- removed from coverage
    fn util() { ... }  // <-- removed from coverage

#[test]
    fn test_hoge() { ... }  // <-- removed from coverage
}
```

#### loop

Fix rust internal bugs that loop branches are not correctly passed.

```rust
for i in 0..10 {  // <-- fix branch coverage information
    println!("{}", i);
}
```

#### derive

structs with `derive(...)` attribute will be ignored

```rust
#[derive(Clone, Debug)]  // <-- removed from coverage
struct Point {  // <-- removed from coverage
  x: f64,  // <-- removed from coverage
  y: f64  // <-- removed from coverage
}  // <-- removed from coverage
```

#### comment

ignore coverage based on comment marker

```rust
fn main() {
    let a = 1 + 2; // cov:ignore-line

// cov:begin-ignore-branch
    println!("Hello");
    println!("world!");
    // cov:end-ignore-branch

// cov: begin-ignore-line
    if a > 2 {
        println!("a is large!");
    } else if a == 0 {
        println!("a is small!");
    }
    // cov: end-ignore-line

// cov:begin-ignore
    println!("a = {}", a);
    // cov:end-ignore

println!("finish."); // cov:ignore-branch

return (); // cov:ignore
}
```

## Roadmap

- Support `cobertura.xml` file. (WIP)
- Add option for uploading the correct coverages to coveralls.
- Use syntax tree generated using [syn](https://crates.io/crates/syn) crate.
- Performance improvement

## Author

👤 **Kogia-sima**

- Twitter: [@Kogia\_sima](https://twitter.com/Kogia\_sima)
- Github: [@Kogia-sima](https://github.com/Kogia-sima)

## 🤝 Contributing

Contributions, issues and feature requests are welcome!

Feel free to check [issues page](https://github.com/Kogia-sima/rust-covfix/issues).

## Show your support

Give a ⭐️ if this project helped you!

## 📝 License

This project is [MIT](https://github.com/Kogia-sima/rust-covfix/blob/master/LICENSE) licensed.

***
_This README was generated with ❤️ by [readme-md-generator](https://github.com/kefranabg/readme-md-generator)_

rust-covfix

rust-covfix Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

rust-covfix

rust-covfix Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README