KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #ai-agents #dsh-plugin #codex #developer-tools #claude-code #windows #desktop-app #typescript #rust #ai-agent #cli

runn

★ 644 Open GitHub ↗

runn is a package/tool for running operations following a scenario.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![build](https://github.com/k1LoW/runn/actions/workflows/ci.yml/badge.svg)](https://github.com/k1LoW/runn/actions/workflows/ci.yml) ![Coverage](https://raw.githubusercontent.com/k1LoW/octocovs/main/badges/k1LoW/runn/coverage.svg) ![Code to Test Ratio](https://raw.githubusercontent.com/k1LoW/octocovs/main/badges/k1LoW/runn/ratio.svg) ![Test Execution Time](https://raw.githubusercontent.com/k1LoW/octocovs/main/badges/k1LoW/runn/time.svg) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/k1LoW/runn)

`runn` ( means "Run N". is pronounced /rʌ́n én/. ) is a package/tool for running operations following a scenario.

Key features of `runn` are:

- **As a tool for scenario based testing.**
- **As a test helper package for the Go language.**
- **As a tool for workflow automation.**
- **Support HTTP request, gRPC request, DB query, Chrome DevTools Protocol, and SSH/Local command execution**
- **OpenAPI Document-like syntax for HTTP request testing.**
- **Single binary = CI-Friendly.**

## Online book

- [runn Tutorial (Japanese)](https://zenn.dev/katzumi/books/runn-tutorial)
- [runn cookbook (Japanese)](https://zenn.dev/k1low/books/runn-cookbook)

## Quickstart

You can use the `runn new` command to quickly start creating scenarios ([runbooks](#runbook--runn-scenario-file-)).

**:rocket: Create and run scenario using `curl` or `grpcurl` commands:**

![docs/runn.svg](https://raw.githubusercontent.com/k1LoW/runn/main/docs/runn.svg)

<summary>Command details</summary>

``` console
$ curl https://httpbin.org/json -H "accept: application/json"
{
 "slideshow": {
 "author": "Yours Truly",
 "date": "date of publication",
 "slides": [
 {
 "title": "Wake up to WonderWidgets!",
 "type": "all"
 },
 {
 "items": [
 "Why WonderWidgets are great",
 "Who buys WonderWidgets"
 ],
 "title": "Overview",
 "type": "all"
 }
 ],
 "title": "Sample Slide Show"
 }
}
$ runn new --and-run --desc 'httpbin.org GET' --out http.yml -- curl https://httpbin.org/json -H "accept: application/json"
$ grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello
{
 "reply": "hello alice"
}
$ runn new --and-run --desc 'grpcb.in Call' --out grpc.yml -- grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello
$ runn list *.yml
 Desc Path If
---------------------------------
 grpcb.in Call grpc.yml
 httpbin.org GET http.yml
$ runn run *.yml
..

2 scenarios, 0 skipped, 0 failures
```

</details>

**:rocket: Create scenario using access log:**

![docs/runn_axslog.svg](https://raw.githubusercontent.com/k1LoW/runn/main/docs/runn_axslog.svg)

<summary>Command details</summary>

``` console
$ cat access_log
183.87.255.54 - - [18/May/2019:05:37:09 +0200] "GET /?post=%3script%3ealert(1); HTTP/1.0" 200 42433
62.109.16.162 - - [18/May/2019:05:37:12 +0200] "GET /core/files/js/editor.js/?form=\xeb\x2a\x5e\x89\x76\x08\xc6\x46\x07\x00\xc7\x46\x0c\x00\x00\x00\x80\xe8\xdc\xff\xff\xff/bin/sh HTTP/1.0" 200 81956
87.251.81.179 - - [18/May/2019:05:37:13 +0200] "GET /login.php/?user=admin&amount=100000 HTTP/1.0" 400 4797
103.36.79.144 - - [18/May/2019:05:37:14 +0200] "GET /authorize.php/.well-known/assetlinks.json HTTP/1.0" 200 9436
$ cat access_log| runn new --out axslog.yml
$ cat axslog.yml| yq
desc: Generated by `runn new`
runners:
  req: https://dummy.example.com
steps:
  - req:
      /?post=%3script%3ealert(1);:
        get:
          body: null
  - req:
      /core/files/js/editor.js/?form=xebx2ax5ex89x76x08xc6x46x07x00xc7x46x0cx00x00x00x80xe8xdcxffxffxff/bin/sh:
        get:
          body: null
  - req:
      /login.php/?user=admin&amount=100000:
        get:
          body: null
  - req:
      /authorize.php/.well-known/assetlinks.json:
        get:
          body: null
$
```

</details>

## Usage

`runn` can run a multi-step scenario following a `runbook` written in YAML format.

### As a tool for scenario based testing / As a tool for automation.

`runn` can run one or more runbooks as a CLI tool.

``` console
$ runn list path/to/**/*.yml
  id:      desc:             if:       steps:  path
-------------------------------------------------------------------------
  a1b7b02  Only if included  included       2  p/t/only_if_included.yml
  85ccd5f  List projects.                   4  p/t/p/list.yml
  47d7ef7  List users.                      3  p/t/u/list.yml
  97f9884  Login                            2  p/t/u/login.yml
  2249d1b  Logout                           3  p/t/u/logout.yml
$ runn run path/to/**/*.yml
S....

5 scenarios, 1 skipped, 0 failures
```

### As a test helper package for the Go language.

`runn` can also behave as a test helper for the Go language.

#### Run N runbooks using [httptest.Server](https://pkg.go.dev/net/http/httptest#Server) and [sql.DB](https://pkg.go.dev/database/sql#DB)

``` go
func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	ts := httptest.NewServer(NewRouter(db))
	t.Cleanup(func() {
		ts.Close()
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Runner("req", ts.URL),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.Load("testdata/books/**/*.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}
```

#### Run single runbook using [httptest.Server](https://pkg.go.dev/net/http/httptest#Server) and [sql.DB](https://pkg.go.dev/database/sql#DB)

``` go
func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	ts := httptest.NewServer(NewRouter(db))
	t.Cleanup(func() {
		ts.Close()
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Book("testdata/books/login.yml"),
		runn.Runner("req", ts.URL),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.New(opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.Run(ctx); err != nil {
		t.Fatal(err)
	}
}
```

#### Run N runbooks using [grpc.Server](https://pkg.go.dev/google.golang.org/grpc#Server)

``` go
func TestServer(t *testing.T) {
	addr := "127.0.0.1:8080"
	l, err := net.Listen("tcp", addr)
	if err != nil {
		t.Fatal(err)
	}
	ts := grpc.NewServer()
	myapppb.RegisterMyappServiceServer(s, NewMyappServer())
	reflection.Register(s)
	go func() {
		ts.Serve(l)
	}()
	t.Cleanup(func() {
		ts.GracefulStop()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Runner("greq", fmt.Sprintf("grpc://%s", addr),
	}
	o, err := runn.Load("testdata/books/**/*.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}
```

#### Run N runbooks with [http.Handler](https://pkg.go.dev/net/http#Handler) and [sql.DB](https://pkg.go.dev/database/sql#DB)

``` go
func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	t.Cleanup(func() {
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.HTTPRunnerWithHandler("req", NewRouter(db)),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.Load("testdata/books/**/*.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}
```

## Examples

See the [details](./examples)

## Runbook ( runn scenario file )

The runbook file has the following format.

`step:` section accepts **list** or **ordered map**.

**List:**

``` yaml
desc: Login and get projects.
runners:
  req: https://example.com/api/v1
  db: mysql://root:mypass@localhost:3306/testdb
vars:
  username: alice
  password: ${TEST_PASS}
steps:
  -
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  -
    req:
      /login:
        post:
          body:
            application/json:
              email: "{{ steps[0].rows[0].email }}"
              password: "{{ vars.password }}"
    test: steps[1].res.status == 200
  -
    req:
      /projects:
        get:
          headers:
            Authorization: "token {{ steps[1].res.body.session_token }}"
          body: null
    test: steps[2].res.status == 200
  -
    test: len(steps[2].res.body.projects) > 0
```

**Map:**

``` yaml
desc: Login and get projects.
runners:
  req: https://example.com/api/v1
  db: mysql://root:mypass@localhost:3306/testdb
vars:
  username: alice
  password: ${TEST_PASS}
steps:
  find_user:
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  login:
    req:
      /login:
        post:
          body:
            application/json:
              email: "{{ steps.find_user.rows[0].email }}"
              password: "{{ vars.password }}"
    test: steps.login.res.status == 200
  list_projects:
    req:
      /projects:
        get:
          headers:
            Authorization: "token {{ steps.login.res.body.session_token }}"
          body: null
    test: steps.list_projects.res.status == 200
  count_projects:
    test: len(steps.list_projects.res.body.projects) > 0
```

#### Grouping of related parts by color

**List:**

![color](https://raw.githubusercontent.com/k1LoW/runn/main/docs/runbook.svg)

**Map:**

![color](https://raw.githubusercontent.com/k1LoW/runn/main/docs/runbook_map.svg)

### JSON Schema

A JSON Schema for the runbook YAML format is available at [`runbook.schema.yaml`](runbook.schema.yaml).

You can use it with [YAML Language Server](https://github.com/redhat-developer/yaml-language-server) for editor validation and autocompletion by adding the following comment to the top of your runbook:

``` yaml
# yaml-language-server: $schema=https://raw.githubusercontent.com/k1LoW/runn/main/runbook.schema.yaml
```

### `desc:`

Description of runbook.

``` yaml
desc: Login and get projects.
runners:
  req: https://example.com/api/v1
vars:
  username: alice
steps:
[...]
```

### `labels:`

Labels of runbook.

``` yaml
desc: Login
runners:
  req: https://example.com/api/v1
labels:
  - users
  - auth
steps:
[...]
```

Runbooks to be run can be filtered by labels.

``` console
$ runn run path/to/**/*.yml --label users --label projects
```

``` console
$ runn run path/to/**/*.yml --label 'users and auth'
```

### `runners:`

Mapping of runners that run `steps:` of runbook.

In the `steps:` section, call the runner with the key specified in the `runners:` section.

Built-in runners such as test runner do not need to be specified in this section.

``` yaml
runners:
  ghapi: ${GITHUB_API_ENDPOINT}
  idp: https://auth.example.com
  db: my:dbuser:${DB_PASS}@hostname:3306/dbname
```

In the example, each runner can be called by `ghapi:`, `idp:` or `db:` in `steps:`.

### `hostRules:`

Allows remapping any request hostname to another hostname, IP address in HTTP/gRPC/DB/CDP/SSH runners.

``` yaml
hostRules:
  example.com: 127.0.0.1:8080
  '*.example.test': 192.168.0.16
```

### `vars:`

Mapping of variables available in the `steps:` of runbook.

``` yaml
vars:
  username: alice@example.com
  token: ${SECRET_TOKEN}
```

In the example, each variable can be used in `{{ vars.username }}` or `{{ vars.token }}` in `steps:`.

### `secrets:`

List of secret var names to be masked.

``` yaml
secrets:
  - vars.secret_token
  - binded_password
  - current.res.message.token
```

### `debug:`

Enable debug output for runn.

``` yaml
debug: true
```

### `interval:`

Interval between steps.

```yaml
interval: 1
```

### `if:`

Conditions for skip all steps.

``` yaml
if: included # Run steps only if included
```

### `skipTest:`

Skip all `test:` sections

``` yaml
skipTest: true
```

### `force:`

Force all steps to run.

``` yaml
force: true
```

### `trace:`

Add tokens for tracing to headers and queries by default.

Currently, HTTP runner, gRPC runner and DB runner are supported.

``` yaml
trace: true
```

### `loop:`

Loop setting for runbook.

#### Simple loop runbook

``` yaml
loop: 10
steps:
  [...]
```

``` yaml
loop:
  count: 10
steps:
  [...]
```

#### Retry runbook

It can be used as a retry mechanism by setting a condition in the `until:` section.

If the condition of `until:` is met, the loop is broken without waiting for the number of `count:` to be run.

Also, if the run of the number of `count:` completes but does not satisfy the condition of `until:`, then the step is considered to be failed.

``` yaml
loop:
  count: 10
  until: 'outcome == "success"' # until the runbook outcome is successful.
  minInterval: 0.5 # sec
  maxInterval: 10  # sec
  # jitter: 0.0
  # interval: 5
  # multiplier: 1.5
steps:
  waitingroom:
    req:
      /cart/in:
        post:
          body:
[...]
```

- `outcome` ... the result of a completed (`success`, `failure`, `skipped`).

### `concurrency:`

Runbooks with the same key are assured of a single run at the same time.

``` yaml
concurrency: use-shared-db
```

``` yaml
concurrency:
  - use-shared-db
  - use-shared-api
```

### `needs:`

It is possible to identify runbooks that must be pre-run.

``` yaml
needs:
  prebook: path/to/prebook.yml
  prebook2: path/to/prebook2.yml
```

Values bound by the bind runner can be referenced by `needs.<key>. *`.

### `steps:`

Steps to run in runbook.

The steps are invoked in order from top to bottom.

Any return values are recorded for each step.

When `steps:` is array, recorded values can be retrieved with `{{ steps[*].* }}`.

``` yaml
steps:
  -
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  -
    req:
      /users/{{ steps[0].rows[0].id }}:
        get:
          body: null
```

When `steps:` is map, recorded values can be retrieved with `{{ steps.<key>.* }}`.

``` yaml
steps:
  find_user:
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  user_info:
    req:
      /users/{{ steps.find_user.rows[0].id }}:
        get:
          body: null
```

### `steps[*].desc:` `steps.<key>.desc:`

Description of step.

``` yaml
steps:
  -
    desc: Login
    req:
      /login:
        post:
          body:
[...]
```

### `steps[*].if:` `steps.<key>.if:`

Conditions for skip step.

``` yaml
steps:
  login:
    if: 'len(vars.token) == 0' # Run step only if var.token is not set
    req:
      /login:
        post:
          body:
[...]
```

### `steps[*].loop:` `steps.<key>.loop:`

Loop setting for step.

#### Simple loop step

``` yaml
steps:
  multicartin:
    loop: 10
    req:
      /cart/in:
        post:
          body:
            application/json:
              product_id: "{{ i }}" # The loop count (0..9) is assigned to `i`.
[...]
```

``` yaml
steps:
  multicartin:
    loop:
      count: 10
    req:
      /cart/in:
        post:
          body:
            application/json:
              product_id: "{{ i }}" # The loop count (0..9) is assigned to `i`.
[...]
```

#### Retry step

It can be used as a retry mechanism by setting a condition in the `until:` section.

If the condition of `until:` is met, the loop is broken without waiting for the number of `count:` to be run.

Also, if the run of the number of `count:` completes but does not satisfy the condition of `until:`, then the step is considered to be failed.

``` yaml
steps:
  waitingroom:
    loop:
      count: 10
      until: 'steps.waitingroom.res.status == "201"' # Store values of latest loop
      minInterval: 500ms
      maxInterval: 10 # sec
      # jitter: 0.0
      # interval: 5
      # multiplier: 1.5
    req:
      /cart/in:
        post:
          body:
[...]
```

#### Execution order within a loop

When using `loop:`, it's important to understand the execution order of runners within each iteration:

1. **Main runner** (HTTP, DB, gRPC, CDP, SSH, Exec, Include, or Runner) - Only one is executed
2. **Dump runner** - Records request/response data (if specified)
3. **Bind runner** - Binds variables (if specified)
4. **Test runner** - Performs assertions (if specified)

This sequence repeats for each loop iteration.

#### ⚠️ Warning: Avoid using `test:` with `loop.until:`

When using `loop:` with an `until:` condition, **avoid** also using `test:` in the same step. The `test:` assertion runs on **every loop iteration**, which can cause the step to fail on the first iteration before the retry logic has a chance to work.

**❌ Incorrect usage:**
```yaml
steps:
  retry_until_success:
    req:
      /status:
        get:
    loop:
      count: 10
      until: 'current.res.status == 200'  # Retry until successful
    test: 'current.res.status == 200'     # This runs every iteration and may fail early!
```

**✅ Correct usage:**
```yaml
steps:
  retry_until_success:
    req:
      /status:
        get:
    loop:
      count: 10
      until: 'current.res.status == 200'  # This is sufficient for retry logic
```

The `until:` condition already serves as your test condition for retry scenarios.

### `steps[*].defer:` `steps.<key>.defer:` [THIS IS EXPERIMENT]

Deferring setting for step.

```yaml
steps:
  -
    defer: true
    req:
      /cart:
        delete:
          body: null
[...]
```

The step marked `defer` behaves as follows.

- If `defer: true` is set, run of the step is deferred until finish of the runbook.
- Steps marked with `defer` are always run even if the running of intermediate steps fails.
- If there are multiple steps marked with `defer`, they are run in LIFO order.
    - Also, the included steps are added to run sequence of the parent runbook's deferred steps.

### `steps[*].force:` `steps.<key>.force:`

Force step to run.

```yaml
steps:
[...]
  -
     force: true
     dump: previous.res.body
[...]
```

## Variables to be stored

runn can use variables and functions when running step.

Also, after step runs, HTTP responses, DB query results, etc. are automatically stored in variables.

The values are stored in predefined variables.

| Variable name | Description |
| --- | --- |
| `vars` | Values set in the `vars:` section |
| `steps` | Return values for each step |
| `i` | Loop index (only in `loop:` section) |
| `env` | Environment variables |
| `current` | Return values of current step |
| `previous` | Return values of previous step |
| `parent` | Variables of parent runbook (only included) |

## Variable Expansion

runn uses `{{ }}` syntax for variable expansion.

### Object variable expansion

When an object variable is used directly in `headers:` or `body:`, runn expands it as a map structure, not as a JSON string.

``` yaml
vars:
  auth_headers:
    X-Token: xxx
    X-Api-Key: yyy
steps:
  - req:
      /api:
        get:
          headers: "{{vars.auth_headers}}"
```

In the example, `vars.auth_headers` expands to multiple headers.

### Converting object to JSON string

To convert an object to a JSON string, wrap the entire `{{...}}` template expression in single quotes. This signals `runn` to serialize the object to JSON.

``` yaml
vars:
  metadata:
    user: alice
    role: admin
steps:
  - req:
      /api:
        get:
          headers:
            X-Metadata: "'{{vars.metadata}}'"
```

## Runner

### HTTP Runner: Do HTTP request

Use `https://` or `http://` scheme to specify HTTP Runner.

When the step is invoked, it sends the specified HTTP Request and records the response.

``` yaml
runners:
  req: https://example.com
steps:
  -
    desc: Post /users                     # description of step
    req:                                  # key to identify the runner. In this case, it is HTTP Runner.
      /users:                             # path of http request
        post:                             # method of http request
          headers:                        # headers of http request
            Authorization: 'Bearer xxxxx'
          body:                           # body of http request
            application/json:             # Content-Type specification. In this case, it is "Content-Type: application/json"
              username: alice
              password: passw0rd
          trace: false                    # add `X-Runn-Trace` header to HTTP request for tracing
    test: |                               # test for current step
      current.res.status == 201
```

See [testdata/book/http.yml](testdata/book/http.yml) and [testdata/book/http_multipart.yml](testdata/book/http_multipart.yml).

#### Structure of recorded responses

The following response

```
HTTP/1.1 200 OK
Content-Length: 29
Content-Type: application/json
Date: Wed, 07 Sep 2022 06:28:20 GMT
Set-Cookie: cookie-name=cookie-value

{"data":{"username":"alice"}}
```

is recorded with the following structure.

``` yaml
[`step key` or `current` or `previous`]:
  res:
    status: 200                              # current.res.status
    headers:
      Content-Length:
        - '29'                               # current.res.headers["Content-Length"][0]
      Content-Type:
        - 'application/json'                 # current.res.headers["Content-Type"][0]
      Date:
        - 'Wed, 07 Sep 2022 06:28:20 GMT'    # current.res.headers["Date"][0]
      Set-Cookie:
        - 'cookie-name=cookie-value'         # current.res.headers["Set-Cookie"][0]
    cookies:
      cookie-name: *http.Cookie              # current.res.cookies["cookie-name"].Value
    body:
      data:
        username: 'alice'                    # current.res.body.data.username
    rawBody: '{"data":{"username":"alice"}}' # current.res.rawBody
```

#### Do not follow redirect

The HTTP Runner interprets HTTP responses and automatically redirects.
To disable this, set `notFollowRedirect` to true.

``` yaml
runners:
  req:
    endpoint: https://example.com
    notFollowRedirect: true
```

#### Enable Cookie Sending

The HTTP Runner automatically saves cookies by interpreting HTTP responses.
To enable cookie sending during requests, set `useCookie` to true.

``` yaml
runners:
  req:
    endpoint: https://example.com
    useCookie: true
```

See [testdata/book/cookie.yml](testdata/book/cookie.yml) and [testdata/book/cookie_in_requests_automatically.yml](testdata/book/cookie_in_requests_automatically.yml).

#### Validation of HTTP request and HTTP response

HTTP requests sent by `runn` and their HTTP responses can be validated.

**OpenAPI v3:**

``` yaml
runners:
  myapi:
    endpoint: https://api.example.com
    openapi3: path/to/openapi.yaml
    # skipValidateRequest: false
    # skipValidateResponse: false
    # skipCircularReferenceCheck: false # skip checking circular references in OpenAPIv3 document.
```

#### Custom CA and Certificates

``` yaml
runners:
  myapi:
    endpoint: https://api.github.com
    cacert: path/to/cacert.pem
    cert: path/to/cert.pem
    key: path/to/key.pem
    # skipVerify: false
```

#### Add `X-Runn-Trace` header to HTTP request for tracing

``` yaml
runners:
  myapi:
    endpoint: https://api.github.com
    trace: true
```

### gRPC Runner: Do gRPC request

Use `grpc://` scheme to specify gRPC Runner.

When the step is invoked, it sends the specified gRPC Request and records the response.

``` yaml
runners:
  greq: grpc://grpc.example.com:80
steps:
  -
    desc: Request using Unary RPC                     # description of step
    greq:                                             # key to identify the runner. In this case, it is gRPC Runner.
      grpctest.GrpcTestService/Hello:                 # package.Service/Method of rpc
        headers:                                      # headers of rpc
          authentication: tokenhello
        message:                                      # message of rpc
          name: alice
          num: 3
          request_time: 2022-06-25T05:24:43.861872Z
        trace: false                                  # add `x-runn-trace` header to gRPC request for tracing
  -
    desc: Request using Server streaming RPC
    greq:
      grpctest.GrpcTestService/ListHello:
        headers:
          authentication: tokenlisthello
        message:
          name: bob
          num: 4
          request_time: 2022-06-25T05:24:43.861872Z
        timeout: 3sec                                 # timeout for rpc
    test: |
      steps.server_streaming.res.status == 0 && len(steps.server_streaming.res.messages) > 0
  -
    desc: Request using Client streaming RPC
    greq:
      grpctest.GrpcTestService/MultiHello:
        headers:
          authentication: tokenmultihello
        messages:                                     # messages of rpc
          -
            name: alice
            num: 5
            request_time: 2022-06-25T05:24:43.861872Z
          -
            name: bob
            num: 6
            request_time: 2022-06-25T05:24:43.861872Z
```

``` yaml
runners:
  greq:
    addr: grpc.example.com:8080
    tls: true
    cacert: path/to/cacert.pem
    cert: path/to/cert.pem
    key: path/to/key.pem
    # skipVerify: false
    # importPaths:
    #   - protobuf/proto
    # protos:
    #   - general/health.proto
    #   - myapp/**/*.proto
```

See [testdata/book/grpc.yml](testdata/book/grpc.yml).

#### Structure of recorded responses

The following response

```protocol-buffer
message HelloResponse {
  string message = 1;

int32 num = 2;

google.protobuf.Timestamp create_time = 3;
}
```

```json
{"create_time":"2022-06-25T05:24:43.861872Z","message":"hello","num":32}
```

and headers

```yaml
content-type: ["application/grpc"]
hello: ["this is header"]
```

and trailers

```yaml
hello: ["this is trailer"]
```

are recorded with the following structure.

``` yaml
[`step key` or `current` or `previous`]:
  res:
    status: 0                                      # current.res.status
    headers:
      content-type:
        - 'application/grpc'                       # current.res.headers[0].content-type
      hello:
        - 'this is header'                         # current.res.headers[0].hello
    trailers:
      hello:
        - 'this is trailer'                        # current.res.trailers[0].hello
    message:
      create_time: '2022-06-25T05:24:43.861872Z'   # current.res.message.create_time
      message: 'hello'                             # current.res.message.message
      num: 32                                      # current.res.message.num
    messages:
      -
        create_time: '2022-06-25T05:24:43.861872Z' # current.res.messages[0].create_time
        message: 'hello'                           # current.res.messages[0].message
        num: 32                                    # current.res.messages[0].num
```

#### Add `x-runn-trace` header to gRPC request for tracing

``` yaml
runners:
  greq:
    addr: grpc.example.com:8080
    trace: true
```

#### Buf

gRPC Runner supports Buf ecosystem includes [Buf Schema Registry](https://buf.build/product/bsr).

It can use the buf modules ( and protos ) it depends on.

``` yaml
runners:
  greq:
    addr: grpc.example.com:8080
    bufDirs:
      - path/to # Set buf directories for registering buf modules and protos
```

``` yaml
runners:
  greq:
    addr: grpc.example.com:8080
    bufLocks:
      - path/to/buf.lock # Register buf modules using buf.lock
```

``` yaml
runners:
  greq:
    addr: grpc.example.com:8080
    bufConfigs:
      - path/to/buf.yaml # Register buf modules using buf.yaml
```

``` yaml
runners:
  greq:
    addr: grpc.example.com:8080
    bufModules:
        - buf.build/owner/repository
        - buf.build/owner2/repository2
```

### DB Runner: Query a database

Use dsn (Data Source Name) to specify DB Runner.

When step is invoked, it executes the specified query the database.

``` yaml
runners:
  db: postgres://dbuser:dbpass@hostname:5432/dbname
steps:
  -
    desc: Select users            # description of step
    db:                           # key to identify the runner. In this case, it is DB Runner.
      query: SELECT * FROM users; # query to execute
      trace: false                # add comment with trace token to query for tracing
```

See [testdata/book/db.yml](testdata/book/db.yml).

#### Structure of recorded responses

If the query is a SELECT clause, it records the selected `rows`,

``` yaml
[`step key` or `current` or `previous`]:
  rows:
    -
      id: 1                           # current.rows[0].id
      username: 'alice'               # current.rows[0].username
      password: 'passw0rd'            # current.rows[0].password
      email: 'alice@example.com'      # current.rows[0].email
      created: '2017-12-05T00:00:00Z' # current.rows[0].created
    -
      id: 2                           # current.rows[1].id
      username: 'bob'                 # current.rows[1].username
      password: 'passw0rd'            # current.rows[1].password
      email: 'bob@example.com'        # current.rows[1].email
      created: '2022-02-22T00:00:00Z' # current.rows[1].created
```

otherwise it records `last_insert_id` and `rows_affected` .

``` yaml
[`step key` or `current` or `previous`]:
  last_insert_id: 3 # current.last_insert_id
  rows_affected: 1  # current.rows_affected
```

#### Add comment with trace token to query for tracing

``` yaml
runners:
  db:
    dsn: mysql://dbuser:dbpass@hostname:3306/dbname
    trace: true
```

#### Support Databases

**PostgreSQL:**

``` yaml
runners:
  mydb: postgres://dbuser:dbpass@hostname:5432/dbname
```

``` yaml
runners:
  db: pg://dbuser:dbpass@hostname:5432/dbname
```

**MySQL:**

``` yaml
runners:
  testdb: mysql://dbuser:dbpass@hostname:3306/dbname
```

``` yaml
runners:
  db: my://dbuser:dbpass@hostname:3306/dbname
```

**SQLite3:**

``` yaml
runners:
  db: sqlite:///path/to/dbname.db
```

``` yaml
runners:
  local: sq://dbname.db
```

**Cloud Spanner:**

``` yaml
runners:
  testdb: spanner://test-project/test-instance/test-database
```

``` yaml
runners:
  db: sp://test-project/test-instance/test-database
```

### CDP Runner: Control browser using Chrome DevTools Protocol (CDP)

Use `cdp://` or `chrome://` scheme to specify CDP Runner.

When the step is invoked, it controls browser via Chrome DevTools Protocol.

``` yaml
runners:
  cc: chrome://new
steps:
  -
    desc: Navigate, click and get h1 using CDP  # description of step
    cc:                                         # key to identify the runner. In this case, it is CDP Runner.
      actions:                                  # actions to control browser
        - navigate: https://pkg.go.dev/time
        - click: 'body > header > div.go-Header-inner > nav > div > ul > li:nth-child(2) > a'
        - waitVisible: 'body > footer'
        - text: 'h1'
  -
    test: |
      previous.text == 'Install the latest version of Go'
```

#### CDP Configuration Options

The CDP runner supports additional configuration options:

``` yaml
runners:
  cc:
    addr: chrome://new  # or cdp://new
    timeout: 120sec     # Timeout for each CDP action (default: 60s)
    flags:              # Chrome browser flags
      headless: true
      disable-gpu: true
      no-sandbox: true
```

**Configuration parameters:**
- `addr`: Chrome DevTools Protocol address. Use `chrome://new` or `cdp://new` to launch a new browser instance
- `timeout`: Timeout duration for each CDP action/step (e.g., "30s", "2m", "1m30s"). Default is 60 seconds
- `flags`: Chrome browser launch flags as key-value pairs

See [testdata/book/cdp.yml](testdata/book/cdp.yml).

#### Functions for action to control browser

**`attributes`** (aliases: `getAttributes`, `attrs`, `getAttrs`)

Get the element attributes for the first element node matching the selector (`sel`).

```yaml
actions:
  - attributes:
      sel: "h1"
# record to current.attrs:
```

```yaml
actions:
  - attributes: "h1"
```

**`click`**

Send a mouse click event to the first element node matching the selector (`sel`).

```yaml
actions:
  - click:
      sel: "nav > div > a"
```

```yaml
actions:
  - click: "nav > div > a"
```

**`doubleClick`**

Send a mouse double click event to the first element node matching the selector (`sel`).

```yaml
actions:
  - doubleClick:
      sel: "nav > div > li"
```

```yaml
actions:
  - doubleClick: "nav > div > li"
```

**`evaluate`** (aliases: `eval`)

Evaluate the Javascript expression (`expr`).

```yaml
actions:
  - evaluate:
      expr: "document.querySelector(\"h1\").textContent = \"hello\""
```

```yaml
actions:
  - evaluate: "document.querySelector(\"h1\").textContent = \"hello\""
```

**`fullHTML`** (aliases: `getFullHTML`, `getHTML`, `html`)

Get the full html of page.

```yaml
actions:
  - fullHTML
# record to current.html:
```

**`innerHTML`** (aliases: `getInnerHTML`)

Get the inner html of the first element node matching the selector (`sel`).

```yaml
actions:
  - innerHTML:
      sel: "h1"
# record to current.html:
```

```yaml
actions:
  - innerHTML: "h1"
```

**`localStorage`** (aliases: `getLocalStorage`)

Get localStorage items.

```yaml
actions:
  - localStorage:
      origin: "https://github.com"
# record to current.items:
```

```yaml
actions:
  - localStorage: "https://github.com"
```

**`location`** (aliases: `getLocation`)

Get the document location.

```yaml
actions:
  - location
# record to current.url:
```

**`navigate`**

Navigate the current frame to `url` page.

```yaml
actions:
  - navigate:
      url: "https://pkg.go.dev/time"
```

```yaml
actions:
  - navigate: "https://pkg.go.dev/time"
```

**`outerHTML`** (aliases: `getOuterHTML`)

Get the outer html of the first element node matching the selector (`sel`).

```yaml
actions:
  - outerHTML:
      sel: "h1"
# record to current.html:
```

```yaml
actions:
  - outerHTML: "h1"
```

**`screenshot`** (aliases: `getScreenshot`)

Take a full screenshot of the entire browser viewport.

```yaml
actions:
  - screenshot
# record to current.png:
```

**`scroll`** (aliases: `scrollIntoView`)

Scroll the window to the first element node matching the selector (`sel`).

```yaml
actions:
  - scroll:
      sel: "body > footer"
```

```yaml
actions:
  - scroll: "body > footer"
```

**`sendKeys`**

Send keys (`value`) to the first element node matching the selector (`sel`).

```yaml
actions:
  - sendKeys:
      sel: "input[name=username]"
      value: "k1lowxb@gmail.com"
```

**`sessionStorage`** (aliases: `getSessionStorage`)

Get sessionStorage items.

```yaml
actions:
  - sessionStorage:
      origin: "https://github.com"
# record to current.items:
```

```yaml
actions:
  - sessionStorage: "https://github.com"
```

**`setUploadFile`** (aliases: `setUpload`)

Set upload file (`path`) to the first element node matching the selector (`sel`).

```yaml
actions:
  - setUploadFile:
      sel: "input[name=avator]"
      path: "/path/to/image.png"
```

**`setUserAgent`** (aliases: `setUA`, `ua`, `userAgent`)

Set the default User-Agent

```yaml
actions:
  - setUserAgent:
      userAgent: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"
```

```yaml
actions:
  - setUserAgent: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"
```

**`submit`**

Submit the parent form of the first element node matching the selector (`sel`).

```yaml
actions:
  - submit:
      sel: "form.login"
```

```yaml
actions:
  - submit: "form.login"
```

**`tabTo`**

Change current frame to the tab with the specified `url`.

```yaml
actions:
  - tabTo:
      url: "https://pkg.go.dev/time"
```

```yaml
actions:
  - tabTo: "https://pkg.go.dev/time"
```

**`text`** (aliases: `getText`)

Get the visible text of the first element node matching the selector (`sel`).

```yaml
actions:
  - text:
      sel: "h1"
# record to current.text:
```

```yaml
actions:
  - text: "h1"
```

**`textContent`** (aliases: `getTextContent`)

Get the text content of the first element node matching the selector (`sel`).

```yaml
actions:
  - textContent:
      sel: "h1"
# record to current.text:
```

```yaml
actions:
  - textContent: "h1"
```

**`title`** (aliases: `getTitle`)

Get the document `title`.

```yaml
actions:
  - title
# record to current.title:
```

**`value`** (aliases: `getValue`)

Get the Javascript value field of the first element node matching the selector (`sel`).

```yaml
actions:
  - value:
      sel: "input[name=address]"
# record to current.value:
```

```yaml
actions:
  - value: "input[name=address]"
```

**`wait`** (aliases: `sleep`)

Wait for the specified `time`.

```yaml
actions:
  - wait:
      time: "10sec"
```

```yaml
actions:
  - wait: "10sec"
```

**`waitReady`**

Wait until the element matching the selector (`sel`) is ready.

```yaml
actions:
  - waitReady:
      sel: "body > footer"
```

```yaml
actions:
  - waitReady: "body > footer"
```

**`waitVisible`**

Wait until the element matching the selector (`sel`) is visible.

```yaml
actions:
  - waitVisible:
      sel: "body > footer"
```

```yaml
actions:
  - waitVisible: "body > footer"
```

### SSH Runner: execute commands on a remote server connected via SSH

Use `ssh://` scheme to specify SSH Runner.

When step is invoked, it executes commands on a remote server connected via SSH.

``` yaml
runners:
  sc: ssh://username@hostname:port
steps:
  -
    desc: 'execute `hostname`' # description of step
    sc:
      command: hostname
```

``` yaml
runners:
  sc:
    hostname: hostname
    user: username
    port: 22
    # host: myserver
    # sshConfig: path/to/ssh_config
    # keepSession: false
    # localForward: '33306:127.0.0.1:3306'
    # keyboardInteractive:
    #   - match: Username
    #     answer: k1low
    #   - match: OTP
    #     answer: ${MY_OTP}
```

See [testdata/book/sshd.yml](testdata/book/sshd.yml).

#### Structure of recorded responses

The response to the run command is always `stdout` and `stderr`.

``` yaml
[`step key` or `current` or `previous`]:
  stdout: 'hello world' # current.stdout
  stderr: ''            # current.stderr
```

### Exec Runner: execute command

> **Note**
> Exec runner requires `run:exec` scope to run.

The `exec` runner is a built-in runner, so there is no need to specify it in the `runners:` section.

It executes command using `command:`, `stdin:`, `shell:`, `background:`, `liveOutput:` and `env:`.

``` yaml
-
  exec:
    command: grep hello
    stdin: '{{ steps[3].res.rawBody }}'
```

``` yaml
-
  exec:
    command: echo $0
    shell: bash
```

`background:` set to `true` to run the command in the background.

``` yaml
-
  exec:
    command: kubectl port-forward svc/nginx 8080:80
    liveOutput: true
```

`liveOutput:` set to `true` to output the command output live.

``` yaml
-
  exec:
    command: |
      echo "Start heavy commands"
      sleep 5
      echo "Heavy command finished"
    liveOutput: true
```

`env:` sets additional environment variables for the command execution.

``` yaml
-
  exec:
    command: printenv MY_VAR
    env:
      MY_VAR: hello
      ANOTHER_VAR: "{{ vars.value }}"
```

See [testdata/book/exec.yml](testdata/book/exec.yml).

#### Structure of recorded responses

The response to the run command is always `stdout`, `stderr` and `exit_code`.

``` yaml
[`step key` or `current` or `previous`]:
  stdout: 'hello world' # current.stdout
  stderr: ''            # current.stderr
  exit_code: 0          # current.exit_code
```

#### `exec.shell:`

Use `shell:` to define the shell and options to be used by the Exec runner.

| Parameter| Command run internally |
|:----|:----|
| unspecified | `bash -e -c {0}` |
| `bash` | `bash --noprofile --norc -eo pipefail -c {0}` |
| `sh` | `sh -e -c {0}` |

### Test Runner: test using recorded values

The `test` runner is a built-in runner, so there is no need to specify it in the `runners:` section.

It evaluates the conditional expression using the recorded values.

``` yaml
-
  test: steps[3].res.status == 200
```

The `test` runner can run in the same steps as the other runners.

### Dump Runner: dump recorded values

The `dump` runner is a built-in runner, so there is no need to specify it in the `runners:` section.

It dumps the specified recorded values.

``` yaml
-
  dump: steps[4].rows
```

``` yaml
-
  dump:
    expr: steps[4].rows
    out: path/to/dump.out
    disableTrailingNewline: true # disable trailing newline. default is false
    disableMaskingSecrets: true  # disable masking secrets. default is false
```

The `dump` runner can run in the same steps as the other runners.

### Include Runner: include other runbook

The `include` runner is a built-in runner, so there is no need to specify it in the `runners:` section.

Include runner reads and runs the runbook in the specified path.

Recorded values are nested.

``` yaml
-
  include: path/to/get_token.yml
```

It is also possible to override `vars:` of included runbook.

``` yaml
-
  include:
    path: path/to/login.yml
    vars:
      username: alice
      password: alicepass
-
  include:
    path: path/to/login.yml
    vars:
      username: bob
      password: bobpass
```

It is also possible to skip all `test:` sections in the included runbook.

``` yaml
-
  include:
    path: path/to/signup.yml
    skipTest: true
```

It is also possible to force all steps in the included runbook to run.

``` yaml
-
  include:
    path: path/to/signup.yml
    force: true
```

#### Inline include

Instead of referencing an external file, you can write runbook steps inline.

``` yaml
-
  include:
    steps:
      - exec:
          command: echo hello
        test: current.stdout == "hello\n"
```

Inline includes can also define `desc:`, `runners:`, and `vars:`.

Note that `vars:` in inline includes defines the **inline runbook's own variables** (equivalent to the top-level `vars:` in an external runbook), not override variables.

``` yaml
-
  include:
    desc: setup user
    runners:
      db: ${TEST_DB}
    vars:
      name: alice
    steps:
      - db:
          query: "INSERT INTO users (name) VALUES ('{{ vars.name }}')"
    skipTest: true
```

### Bind Runner: bind variables

The `bind` runner is a built-in runner, so there is no need to specify it in the `runners:` section.

It bind runner binds any values with another key.

``` yaml
  -
    req:
      /users/k1low:
        get:
          body: null
  -
    bind:
      user_id: steps[0].res.body.data.id
  -
    dump: user_id
```

The `bind` runner can run in the same steps as the other runners.

### Runner Runner: Define runner in the middle of steps.

The `runner` runner is a built-in runner, so there is no need to specify it in the `runners:` section.

It defines a runner in the middle of steps.

``` yaml
  -
    runner:
      sc: ssh://username@hostname:port
  -
    sc:
      command: hostname
```

The `runner` runner can not run in the same steps as the other runners.

### Agent Runner: Interact with AI agents (experimental)

The agent runner sends prompts to external AI agents and records responses. It requires `run:agent` scope.

``` console
$ runn run --scopes run:agent runbook.yml
```

#### Supported agents

| Agent | SDK | Description |
|-------|-----|-------------|
| `claude` | [claude-agent-sdk-go](https://github.com/k1LoW/claude-agent-sdk-go) | Claude Code agent |
| `copilot` | [copilot-sdk](https://github.com/github/copilot-sdk) | GitHub Copilot agent |
| `codex` | [codex-agent-sdk-go](https://github.com/k1LoW/codex-agent-sdk-go) | OpenAI Codex agent |

#### Basic usage

``` yaml
runners:
  claude:
    agent: claude
    model: sonnet
    system: "You are a helpful assistant."
    permissions:
      - "allow:*"
steps:
  -
    claude:
      prompt: "What is Go?"
    test: current.res.content != ''
```

The expanded prompt is stored in `steps[N].prompt` and the response in `steps[N].res.content`.

#### Conversation context

Each step sends a single prompt. Conversation context is maintained per runner definition across steps. Use `clearContext: true` to reset.

``` yaml
steps:
  -
    claude:
      prompt: "What is Go?"
  -
    claude:
      prompt: "Tell me more about its concurrency model"
  -
    claude:
      prompt: "Completely different topic"
      clearContext: true
```

Different runner definitions have independent contexts, even if they use the same agent and model.

#### Collaboration with other runners

Agent runner results can be referenced by other runners via expression expansion.

``` yaml
runners:
  req:
    endpoint: http://api.example.com
  claude:
    agent: claude
    model: sonnet
    system: "Summarize articles in Japanese."
    permissions:
      - "allow:*"
steps:
  -
    req:
      /articles/1:
        get:
          body: null
  -
    claude:
      prompt: "{{ steps[0].res.body.content }}"
    test: current.res.content != ''
```

#### Configuration fields

| Field | Description |
|-------|-------------|
| `agent` | Agent framework (`claude`, `copilot`, `codex`) |
| `provider` | LLM provider (optional if same as agent. e.g. `openai` for copilot) |
| `model` | Model name |
| `system` | System prompt |
| `tools` | Available tools (restricts which tools the agent can see) |
| `permissions` | Tool permission rules (see below) |
| `interactive` | Enable interactive mode (`true`/`false`) |

#### Permissions

The `permissions` field controls what the agent is allowed to do. Rules are evaluated in array order (**last match wins**).

``` yaml
permissions:
  - "allow:*"          # allow all tools
  - "deny:Write"       # but deny Write (overrides allow:*)
```

``` yaml
permissions:
  - "deny:*"           # deny all tools
  - "allow:Read"       # but allow Read (overrides deny:*)
```

**Rule formats:**

| Format | Description |
|--------|-------------|
| `allow:*` | Allow all tools/operations |
| `deny:*` | Deny all tools/operations |
| `allow:ToolName` | Allow a specific tool |
| `deny:ToolName` | Deny a specific tool |
| `sandbox:mode` | Sandbox mode (Codex only) |
| Other values | SDK-specific mode (see per-agent sections below) |

**`interactive: true`:** When set, tools not matched by any rule will prompt the user for confirmation. Agent questions (e.g. AskUserQuestion) are also enabled. When `false` (default), unmatched tools are denied and agent questions cause an error.

#### Permissions per agent

##### Claude

Claude uses [Claude Code CLI](https://github.com/anthropics/claude-code) under the hood.

``` yaml
runners:
  claude:
    agent: claude
    model: sonnet
    tools:
      - Read
      - Bash
    permissions:
      - "allow:Read"
      - "allow:Bash(git:*)"    # pattern matching supported
      - "deny:Bash(rm *)"
      - acceptEdits             # SDK-specific permission mode
    interactive: true
```

| Format | Claude mapping |
|--------|----------------|
| `allow:ToolName` | `--allowedTools` (auto-approved, no callback) |
| `deny:ToolName` | `--disallowedTools` (blocked) |
| `acceptEdits`, `plan`, `bypassPermissions`, ... | `--permission-mode` |

Claude supports tool name patterns (e.g. `Bash(git:*)`) in both `allow:` and `deny:` rules. Available tools are configured via `tools` field (`--tools`).

When `interactive: true`, unmatched tools are confirmed via terminal prompt, and agent questions (`AskUserQuestion`) are answered interactively.

##### Copilot

Copilot uses [GitHub Copilot CLI](https://docs.github.com/en/copilot) under the hood.

``` yaml
runners:
  helper:
    agent: copilot
    provider: openai
    model: gpt-5-nano
    system: "You are a data analyst."
    tools:
      - web_search
    permissions:
      - "allow:*"
    interactive: true
```

| Format | Copilot mapping |
|--------|-----------------|
| `allow:ToolName` | Approved via `OnPermissionRequest` callback |
| `deny:ToolName` | `ExcludedTools` + denied via callback |
| SDK-specific modes | Not supported (error) |

Copilot's permission model is based on operation kinds (`shell`, `write`, `read`, `url`, etc.) rather than tool names. The `allow:`/`deny:` rules match against the tool name in the permission request.

When `interactive: true`, unmatched operations are confirmed via terminal prompt, and user input requests are answered interactively.

##### Codex

Codex uses [OpenAI Codex CLI](https://github.com/openai/codex) under the hood.

``` yaml
runners:
  codex:
    agent: codex
    model: o3
    permissions:
      - "allow:*"
      - full-auto
      - "sandbox:workspace-write"
```

| Format | Codex mapping |
|--------|---------------|
| `allow:*` (with no mode set) | `--approval-policy full-auto` |
| `deny:ToolName` | Declined via `OnCommandApproval` callback |
| `full-auto`, `auto-edit`, `suggest`, ... | `--approval-policy` |
| `sandbox:workspace-write`, `sandbox:workspace-read`, ... | `--sandbox` |

Codex has two SDK-specific settings:
- **Approval policy** — Controls how commands/file changes are approved. Set as a plain string in permissions (e.g. `full-auto`).
- **Sandbox mode** — Controls the execution sandbox. Set with `sandbox:` prefix (e.g. `sandbox:workspace-write`).

When `interactive: true`, command executions and file changes are confirmed via terminal prompt.

## Expression evaluation engine

runn has embedded [expr-lang/expr](https://github.com/expr-lang/expr) as the evaluation engine for the expression.

See [Language Definition](https://expr-lang.org/docs/language-definition).

### Additional built-in functions

- `urlencode` ... [url.QueryEscape](https://pkg.go.dev/net/url#QueryEscape)
- `bool` ... [cast.ToBool](https://pkg.go.dev/github.com/spf13/cast#ToBool)
- `compare` ... Compare two values ( `func(x, y any, ignorePaths ...string) bool` ). The optional `ignorePaths` argument is a list of [jq syntax path expressions](https://jqlang.github.io/jq/manual/#path) to ignore when comparing two values.
- `diff` ... Difference between two values ( `func(x, y any, ignorePaths ...string) string` ). The optional `ignorePaths` argument is a list of [jq syntax path expressions](https://jqlang.github.io/jq/manual/#path) to ignore when comparing two values.
- `pick` ... Returns same map type filtered by given keys left [lo.PickByKeys](https://github.com/samber/lo?tab=readme-ov-file#pickbykeys).
- `omit` ... Returns same map type filtered by given keys excluded [lo.OmitByKeys](https://github.com/samber/lo?tab=readme-ov-file#omitbykeys).
- `merge` ... Merges multiple maps from left to right [lo.Assign](https://github.com/samber/lo?tab=readme-ov-file#assign).
- `input` ... [prompter.Prompt](https://pkg.go.dev/github.com/Songmu/prompter#Prompt)
- `intersect` ... Find the intersection of two iterable values ( `func(x, y any) any` ).
- `secret` ... [prompter.Password](https://pkg.go.dev/github.com/Songmu/prompter#Password)
- `select` ... Select from candidates. `func(message string, candidates []string, default string) string`
- `basename` ... [filepath.Base](https://pkg.go.dev/path/filepath#Base)
- `time` ... Converts the given string or number to `time.Time{}`.
- `faker.*` ... Generate fake data using [Faker](https://pkg.go.dev/github.com/k1LoW/runn/internal/builtin#Faker) ).
- `file` ... Read the file as a string. Returns nil if it does not exist.
- `hash.*` ... Compute hash values using secure algorithms. [hash.Sha256](https://pkg.go.dev/github.com/k1LoW/runn/internal/builtin#Hash.Sha256), [hash.Sha512](https://pkg.go.dev/github.com/k1LoW/runn/internal/builtin#Hash.Sha512).
- `jwt.*` ... Generate and parse JSON Web Tokens (JWT) using the specified claims and signature algorithm. [jwt.Sign](https://pkg.go.dev/github.com/k1LoW/runn/internal/builtin#Jwt.Sign), [jwt.Parse](https://pkg.go.dev/github.com/k1LoW/runn/internal/builtin#Jwt.Parse).  
See [testdata/book/http_bearer.yml](https://github.com/k1LoW/runn/blob/main/testdata/book/http_bearer.yml) for a complete example.  
Note: This function currently supports JWS (JSON Web Signature) only. JWE (JSON Web Encryption) is not supported.

## Option

See https://pkg.go.dev/github.com/k1LoW/runn#Option

### Example: Run as a test helper ( func `T` )

https://pkg.go.dev/github.com/k1LoW/runn#T

``` go
o, err := runn.Load("testdata/**/*.yml", runn.T(t))
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}
```

### Example: Add custom function ( func `Func` )

https://pkg.go.dev/github.com/k1LoW/runn#Func

``` yaml
desc: Test using GitHub
runners:
  req:
    endpoint: https://github.com
steps:
  -
    req:
      /search?l={{ urlencode('C++') }}&q=runn&type=Repositories:
        get:
          body:
            application/json:
              null
    test: 'steps[0].res.status == 200'
```

``` go
o, err := runn.Load("testdata/**/*.yml", runn.Func("urlencode", url.QueryEscape))
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}
```

## Scope

runn requires explicit specification of scope for some features.

runn has the following scopes.

| Scope | Description | Default |
| --- | --- | --- |
| `read:parent` | Required for reading files above the working directory. | `false` |
| `read:remote` | Required for reading remote files. | `false` |
| `run:exec` | Required for running Exec runner. | `false` |

To specify scopes, using the `--scopes` option or the environment variable `RUNN_SCOPES`.

```console
$ runn run path/to/**/*.yml --scopes read:parent,read:remote
```

```console
$ env RUNN_SCOPES=read:parent,read:remote runn run path/to/**/*.yml
```

Also, [runn.Scopes](https://pkg.go.dev/github.com/k1LoW/runn#Scopes) can be used in the code

``` go
o, err := runn.Load("path/to/**/*.yml", runn.Scopes(runn.AllowReadParent, runn.AllowReadRemote))
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}
```

To disable scope, can use `!read:*` instead of `read:*`

```console
$ runn run path/to/**/*.yml --scopes '!read:parent'
```

## Filter runbooks to be executed by the environment variable `RUNN_RUN`

Run only runbooks matching the filename "login".

``` console
$ env RUNN_RUN=login go test ./... -run TestRouter
```

## Measure elapsed time as profile

``` go
opts := []runn.Option{
	runn.T(t),
	runn.Book("testdata/books/login.yml"),
	runn.Profile(true),
}
o, err := runn.New(opts...)
if err != nil {
	t.Fatal(err)
}
if err := o.Run(ctx); err != nil {
	t.Fatal(err)
}
f, err := os.Open("profile.json")
if err != nil {
	t.Fatal(err)
}
if err := o.DumpProfile(f); err != nil {
	t.Fatal(err)
}
```

``` console
$ runn run testdata/books/login.yml --profile
```

The runbook run profile can be read with `runn rprof` command.

``` console
$ runn rprof runn.prof
  runbook[login site](t/b/login.yml)           2995.72ms
    steps[0].req                                747.67ms
    steps[1].req                                185.69ms
    steps[2].req                                192.65ms
    steps[3].req                                188.23ms
    steps[4].req                                569.53ms
    steps[5].req                                299.88ms
    steps[6].test                                 0.14ms
    steps[7].include                            620.88ms
      runbook[include](t/b/login_include.yml)   605.56ms
        steps[0].req                            605.54ms
    steps[8].req                                190.92ms
  [total]                                      2995.84ms
```

## Capture runbook runs

``` go
opts := []runn.Option{
	runn.T(t),
	runn.Capture(capture.Runbook("path/to/dir")),
}
o, err := runn.Load("testdata/books/**/*.yml", opts...)
if err != nil {
	t.Fatal(err)
}
if err := o.RunN(ctx); err != nil {
	t.Fatal(err)
}
```

``` console
$ runn run path/to/**/*.yml --capture path/to/dir
```

## Load test using runbooks

You can use the `runn loadt` command for load testing using runbooks.

``` console
$ runn loadt --load-concurrent 2 --max-rps 0 path/to/*.yml

Number of runbooks per RunN....: 15
Warm up time (--warm-up).......: 5s
Duration (--duration)..........: 10s
Concurrent (--load-concurrent).: 2
Max RunN per second (--max-rps): 0

Total..........................: 12
Succeeded......................: 12
Failed.........................: 0
Error rate.....................: 0%
RunN per seconds...............: 1.2
Latency .......................: max=1,835.1ms min=1,451.3ms avg=1,627.8ms med=1,619.8ms p(90)=1,741.5ms p(99)=1,788.4ms

```

It also checks the results of the load test with the `--threshold` option. If the condition is not met, it returns exit status 1.

``` console
$ runn loadt --load-concurrent 2 --max-rps 0 --threshold 'error_rate < 10' path/to/*.yml

Number of runbooks per RunN...: 15
Warm up time (--warm-up)......: 5s
Duration (--duration).........: 10s
Concurrent (--load-concurrent): 2

Total.........................: 13
Succeeded.....................: 12
Failed........................: 1
Error rate....................: 7.6%
RunN per seconds..............: 1.3
Latency ......................: max=1,790.2ms min=95.0ms avg=1,541.4ms med=1,640.4ms p(90)=1,749.7ms p(99)=1,786.5ms

Error: (error_rate < 10) is not true
error_rate < 10
├── error_rate => 14.285714285714285
└── 10 => 10
```

### Variables for threshold

| Variable name | Type | Description |
| --- | --- | --- |
| `total` | `int` | Total |
| `succeeded` | `int` | Succeeded |
| `failed` | `int` | Failed |
| `error_rate` | `float` | Error rate |
| `rps` | `float` | RunN per seconds |
| `max` | `float` | Latency max (ms) |
| `mid` | `float` | Latency mid (ms) |
| `min` | `float` | Latency min (ms) |
| `p90` | `float` | Latency p(90) (ms) |
| `p99` | `float` | Latency p(99) (ms) |
| `avg` | `float` | Latency avg (ms) |

## Install

### As a CLI tool

**deb:**

``` console
$ export RUNN_VERSION=X.X.X
$ curl -o runn.deb -L https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.deb
$ dpkg -i runn.deb
```

**RPM:**

``` console
$ export RUNN_VERSION=X.X.X
$ yum install https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.rpm
```

**apk:**

``` console
$ export RUNN_VERSION=X.X.X
$ curl -o runn.apk -L https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.apk
$ apk add runn.apk
```

**homebrew tap:**

```console
$ brew install k1LoW/tap/runn
```

**[aqua](https://aquaproj.github.io/):**

```console
$ aqua g -i k1LoW/runn
```

**manually:**

Download binary from [releases page](https://github.com/k1LoW/runn/releases)

**docker:**

```console
$ docker container run -it --rm --name runn -v $PWD:/books ghcr.io/k1low/runn:latest list /books/*.yml
```

**go install:**

```console
$ go install github.com/k1LoW/runn/cmd/runn@latest
```

### As a test helper

```console
$ go get github.com/k1LoW/runn
```

## Alternatives

- [zoncoen/scenarigo](https://github.com/zoncoen/scenarigo): An end-to-end scenario testing tool for HTTP/gRPC server.

## References

- [zoncoen/scenarigo](https://github.com/zoncoen/scenarigo): An end-to-end scenario testing tool for HTTP/gRPC server.
- [fullstorydev/grpcurl](https://github.com/fullstorydev/grpcurl): Like cURL, but for gRPC: Command-line tool for interacting with gRPC servers
- [ktr0731/evans](https://github.com/ktr0731/evans): Evans: more expressive universal gRPC client

## License

- [MIT License](LICENSE)
    - Include logo as well as source code.
    - Only logo license can be selected [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
    - Also, if there is no alteration to the logo and it is used for technical information about runn, I would not say anything if the copyright notice is omitted.

runn

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README