benchdiff

KO | EN

gitlite — search

#python #javascript #golang #nodejs #react #linux #aws #machine-learning #go #php #neovim #pytorch

★ 68 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# benchdiff

[![Go Reference](https://pkg.go.dev/badge/github.com/willabides/benchdiff.svg)](https://pkg.go.dev/github.com/willabides/benchdiff)
[![ci](https://github.com/WillAbides/benchdiff/workflows/ci/badge.svg?branch=main&event=push)](https://github.com/WillAbides/benchdiff/actions?query=workflow%3Aci+branch%3Amain+event%3Apush)

Benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show
the delta.

## What it does

Benchdiff automates comparing the results of the same go benchmarks on your worktree and a git reference.

It essentially combines multiple `git`, `go test -bench` and `benchstat` commands into one.

These are the basic steps `benchdiff` performs:
- Runs `go test -bench ...` on your current worktree and saves the results to a cache directory.
- Creates a new worktree at the base ref
- Runs `go test -bench ...` on the base worktree and saves the results to cache again.
- Runs `benchstat` to compare the base and head results from cache.

## GitHub Action

benchdiff can be run as a GitHub Action. https://github.com/WillAbides/benchdiff-action

## Usage

### `benchdiff --help` output


```
Usage: benchdiff

benchdiff runs go benchmarks on your current git worktree and a base ref then uses benchstat to show
the delta.

More documentation at https://github.com/willabides/benchdiff.

Flags:
  -h, --help       Show context-sensitive help.
      --version    Output the benchdiff version and exit.
      --debug      write verbose output to stderr

--base-ref="HEAD"    The git ref to be used as a baseline.
  --cooldown=100ms     How long to pause for cooldown between head and base runs.
  --force-base         Rerun benchmarks on the base reference even if the output already exists.
  --git-cmd="git"      The executable to use for git commands.
  --json               Format output as JSON.
  --on-degrade=0       Exit code when there is a statistically significant degradation in the
                       results.
  --tolerance=10.0     The minimum percent change before a result is considered degraded.

benchmark command line
  --bench="."              Run only those benchmarks matching a regular expression. To run all
                           benchmarks, use '--bench .'.
  --benchmark-args=args    Override the default args to the go command. This may be a template.
                           See https://github.com/willabides/benchdiff for details."
  --benchmark-cmd="go"     The command to use for benchmarks.
  --benchmem               Memory allocation statistics for benchmarks.
  --benchtime=STRING       Run enough iterations of each benchmark to take t, specified as a
                           time.Duration (for example, --benchtime 1h30s). The default is 1 second
                           (1s). The special syntax Nx means to run the benchmark N times (for
                           example, -benchtime 100x).
  --count=10               Run each benchmark n times. If --cpu is set, run n times for each
                           GOMAXPROCS value.'
  --cpu=GOMAXPROCS,...     Specify a list of GOMAXPROCS values for which the benchmarks should be
                           executed. The default is the current value of GOMAXPROCS.
  --packages="./..."       Run benchmarks in these packages.
  --show-bench-cmdline     Instead of running benchmarks, output the command that would be used and
                           exit.
  --tags=STRING            Set the -tags flag on the go test command
  --warmup-count=INT       Run benchmarks with -count=n as a warmup
  --warmup-time=STRING     When warmups are run, set -benchtime=n

benchstat options
  --alpha=0.05                 consider change significant if p < α
  --benchstat-output="text"    format for benchstat output (csv,html,markdown or text)
  --delta-test="utest"         significance test to apply to delta: utest, ttest, or none
  --geomean                    print the geometric mean of each file
  --norange                    suppress range columns (CSV and markdown only)
  --reverse-sort               reverse sort order
  --sort="none"                sort by order: delta, name, none
  --split="pkg,goos,goarch"    split benchmarks by labels

benchmark result cache
  --cache-dir=STRING    Override the default directory where benchmark output is kept.
  --clear-cache         Remove benchdiff files from the cache dir.
  --show-cache-dir      Output the cache dir and exit.
```

### `--benchmark-args`

The default should work for most cases.

`--benchmark-args` is everything after "go" in the `go test` command that runs the benchmarks. You can use either 
pass in the raw values or use a go template.

Default value:

```
test {{ .Packages }} -run '^$'
{{- if .Bench }} -bench {{ .Bench }}{{end}}
{{- if .Count }} -count {{ .Count }}{{end}}
{{- if .Benchtime }} -benchtime {{ .Benchtime }}{{end}}
{{- if .CPU }} -cpu {{ .CPU }}{{ end }}
{{- if .Tags }} -tags "{{ .Tags }}"{{ end }}
{{- if .Benchmem }} -benchmem{{ end }}
```

## Install

### go get

`go get -u github.com/willabides/benchdiff/cmd/benchdiff`

### bindown

Add a [bindown](https://github.com/willabides/bindown) dependency:

```
$ bindown template-source add benchdiff https://raw.githubusercontent.com/WillAbides/benchdiff/main/bindown.yml
$ bindown dependency add benchdiff benchdiff#benchdiff
Please enter a value for required variable "version":	<latest version>
```

benchdiff

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README