chef-load

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 33 Open GitHub ↗

chef-load - a tool for simulating load on a Chef Infra Server and/or a Chef Automate server

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Description

chef-load is a tool that simulates Chef Client API load on a [Chef Server](https://www.chef.io/chef/) and/or a [Chef Automate server](https://www.chef.io/automate/).

It is designed to be easy to use yet powerfully flexible and accurate in its simulation of the chef-client run.

It supports three load modes:

* **`policyfile`** (default) — all nodes follow the policyfile API flow (`GET policy_groups/<group>/policies/<name>` / `GET cookbook_artifacts/<name>/<identifier>`), matching Chef Infra Client runs with `PolicyName`/`PolicyGroup` set.
* **`legacy`** — all nodes follow the traditional run-list/roles/environments flow (original behaviour).
* **`mixed`** — a configurable fraction of nodes use the policyfile flow; the remainder use the legacy flow.

It works well at this point but there is always room for improvement so please provide feedback.

## Considerations when applying high load

chef-load will periodically resolve hostnames for the API requests. It is recommended that the
hostnames and their IP addresses are put in the system's `/etc/hosts` file so chef-load does not
need to make DNS requests.

Make sure the maximum number of open file descriptors is set to `unlimited`. This can either be done in chef-load's systemd service file as shown below or you can follow instructions in [this link](https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/).

## Installation & Setup

### Download prebuilt binary

Prebuilt chef-load binary files are available on chef-load's "Releases" page.

https://github.com/jeremiahsnapp/chef-load/releases

### Install via Habitat

Follow these steps to install the latest version of chef-load from the habitat depot in the `chef` origin:

1. Install Habitat: https://www.habitat.sh/docs/install-habitat/
2. Install and link chef-load binary
```
hab pkg install chef/chef-load
hab pkg binlink chef/chef-load chef-load
```

### Install onto AWS EC2 with Habitat

Follow these steps to install onto EC2, the latest version of chef-load from the habitat depot in the `chef` origin:

1. From the root of chef-load, run: `vagrant up; vagrant ssh;`
2. Install and link chef-load binary
```
hab pkg install chef/chef-load
hab pkg binlink chef/chef-load chef-load
```

### Generate a chef-load configuration file.

The configuration file uses [TOML syntax](https://github.com/toml-lang/toml) and documents a lot of the flexibility of chef-load so please read it.

```
chef-load init > chef-load.toml
```

chef-load logs all API requests in the file specified by the `log_file` setting in the config file. The default value is `/var/log/chef-load/chef-load.log`.

Make sure `chef-load.toml` has appropriate settings for applying load to your Chef Server,
Automate Server or both.

#### Policyfile settings

When `load_mode` is `"policyfile"` or `"mixed"`, set the target policy:

```toml
load_mode    = "policyfile"
policy_name  = "base"
policy_group = "production"
```

To simulate a fleet spread across multiple policies within the same policy group, use the `policyfiles` list instead of `policy_name`. Each simulated chef-client run randomly picks one name from the list — all nodes share the same `policy_group`. This mirrors how `run_lists` works for legacy nodes. All listed policies must exist in the configured `policy_group` on the Chef Server.

```toml
load_mode    = "policyfile"
policy_group = "production"
policyfiles  = [ "base", "webserver", "database" ]
```

When `policyfiles` is non-empty, `policy_name` is ignored.

For `"mixed"` mode, `policyfile_percentage` (default `0.5`) controls what fraction of simulated nodes use the policyfile API. Nodes are assigned deterministically by node number so the split is stable across runs.

```toml
load_mode             = "mixed"
policy_name           = "base"
policy_group          = "production"
policyfile_percentage = 0.3   # 30% policyfile, 70% legacy
```

`policyfiles` works in `"mixed"` mode too — policyfile-assigned nodes will pick a policy name randomly from the list while legacy-assigned nodes continue to use `run_list`/`run_lists`.

#### Compliance phase simulation

chef-load can simulate the built-in compliance phase introduced in Chef Infra Client 17+. When enabled, an `inspec_report` message is sent to the data collector at the end of each simulated chef-client run, linked to the run via `run_uuid` (matching actual chef-client behaviour).

```toml
enable_compliance_phase    = true
compliance_phase_percentage = 1.0  # fraction of nodes that report compliance
```

If `compliance_status_json_file` is set, that file's content is used as the report body. Otherwise a minimal synthetic report is sent representing a node with no InSpec profiles configured.

#### Chef API client

The client defined by "client_name" in the chef-load configuration file needs to be an admin user of the Chef Server organization.

## Continuous load of nodes, actions and reports

Run chef-load using only the configuration file.

```
chef-load start --config chef-load.toml
```

You can override the load mode and policyfile settings on the command line:

```
# policyfile mode (default)
chef-load start --config chef-load.toml --load_mode policyfile --policy_name base --policy_group production

# legacy (run-list/roles/environments) mode
chef-load start --config chef-load.toml --load_mode legacy

# mixed mode — 40% policyfile, 60% legacy
chef-load start --config chef-load.toml --load_mode mixed --policy_name base --policy_group production --policyfile_percentage 0.4
```

Enable compliance phase simulation:

```
chef-load start --config chef-load.toml --enable_compliance_phase --compliance_phase_percentage 0.5
```

You can use the `--node_name_prefix` command line option to set the prefix for the node names. This
enables easily running multiple instances of chef-load without affecting each others' nodes.
For example, a value of "chef-load" will result in nodes named "chef-load-1", "chef-load-2", etc.

```
chef-load start --config chef-load.toml --node_name_prefix chef-load-a

# in another terminal you can run the following to create another instance of chef-load
chef-load start --config chef-load.toml --node_name_prefix chef-load-b
```

You can set the number of nodes and/or actions using the `--nodes` and `--actions` command line options, as well as the interval using
the `--interval` command line option. This is useful for quickly adjusting the load without modifying the configuration file.

chef-load will evenly distribute the number of nodes and/or actions across the desired interval (minutes).

Examples:

* 1800 nodes & 600 actions / 30 minute interval = 60 chef-client runs & 20 actions per minute
* 1800 nodes & 600 actions / 60 minute interval = 30 chef-client runs & 10 actions per minute

```
chef-load start --config chef-load.toml --nodes 1800 --actions 500
```

```
chef-load start --config chef-load.toml --nodes 1800 --actions 500 --interval 60
```

_NOTE: Every chef-client run will automatically trigger a node update action, plus the specified actions._

### Example chef-load systemd service file

Here is a working example of a systemd service file for chef-load. Notice that it is able to set `LimitNOFILE` to unlimited to avoid running out of file descriptors.

```
[Unit]
Description=Chef load testing tool
After=network.target

[Service]
ExecStart=/home/centos/chef-load start --config /home/centos/chef_load.toml
Type=simple
PIDFile=/tmp/chef_load.pid
Restart=always
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
LimitNOFILE=infinity

[Install]
WantedBy=default.target
```

## API Request Profile

chef-load prints an API request profile when it receives a `USR1` signal and when it is terminated.

The URLs shown in the profile differ by load mode. In **policyfile** mode you will see requests like:

```
GET  /organizations/<org>/policy_groups/<group>/policies/<name>
GET  /organizations/<org>/cookbook_artifacts/<name>/<identifier>
POST /data-collector/v0/   (run_start, run_converge, and optionally inspec_report)
```

In **legacy** mode the profile looks like the example below (roles, environments, cookbook_versions).

```
root@ip-172-31-17-147:~# ./chef-load start --config chef-load.toml --nodes 60 --interval 1 --node_name_prefix foo
2017-08-30T20:25:37Z Starting chef-load with 60 nodes distributed evenly across a 1 minute interval
2017-08-30T20:25:37Z All API requests will be logged in /var/log/chef-load/chef-load.log
2017-08-30T20:25:44Z Received Signal: USR1
2017-08-30T20:25:44Z Printing profile of API requests
Total API Requests: 616
% of Total | Subtotal | Status | Method | URL
2.27         14         204      POST     https://automate.lxc/data-collector/v0/
1.14         7          409      POST     https://chef.lxc/organizations/demo/clients
1.14         7          200      GET      https://chef.lxc/organizations/demo/environments/_default
1.14         7          200      POST     https://chef.lxc/organizations/demo/environments/_default/cookbook_versions
1.14         7          200      GET      https://chef.lxc/organizations/demo/nodes/foo-<N>
1.14         7          200      PUT      https://chef.lxc/organizations/demo/nodes/foo-<N>
1.14         7          404      GET      https://chef.lxc/organizations/demo/roles/chef-client
90.91        560        200      GET      https://chef.lxc:443/bookshelf/<...>
^C
2017-08-30T20:25:50Z Received Signal: INT
2017-08-30T20:25:50Z Printing profile of API requests
Total API Requests: 1149
% of Total | Subtotal | Status | Method | URL
2.35         27         204      POST     https://automate.lxc/data-collector/v0/
1.22         14         409      POST     https://chef.lxc/organizations/demo/clients
1.22         14         200      GET      https://chef.lxc/organizations/demo/environments/_default
1.13         13         200      POST     https://chef.lxc/organizations/demo/environments/_default/cookbook_versions
1.22         14         200      GET      https://chef.lxc/organizations/demo/nodes/foo-<N>
1.13         13         200      PUT      https://chef.lxc/organizations/demo/nodes/foo-<N>
1.22         14         404      GET      https://chef.lxc/organizations/demo/roles/chef-client
90.51        1040       200      GET      https://chef.lxc:443/bookshelf/<...>
2017-08-30T20:25:50Z Stopping chef-load
```

The `-profile-logs` option will read the specified chef-load log files and print an API request profile. If chef-load receives a `USR1` signal or is terminated before it finishes reading the log files then it will print an API request profile for the data that it read up to that point in time.

```
root@ip-172-31-17-147:~# ./chef-load start --profile-logs  /var/log/chef-load/*
2017-08-31T15:04:42Z Reading log file /var/log/chef-load/chef-load.log
2017-08-31T15:04:43Z Reading log file /var/log/chef-load/chef-load.log.1
2017-08-31T15:04:43Z Reading log file /var/log/chef-load/chef-load.log.2
2017-08-31T15:04:43Z Reading log file /var/log/chef-load/chef-load.log.3
2017-08-31T15:04:43Z Received Signal: USR1
2017-08-31T15:04:43Z Printing profile of API requests
Total API Requests: 31478
% of Total | Subtotal | Status | Method | URL
2.55         804        204      POST     https://automate.lxc/data-collector/v0/
0.04         12         201      POST     https://chef.lxc/organizations/demo/clients
1.05         330        409      POST     https://chef.lxc/organizations/demo/clients
1.16         366        200      GET      https://chef.lxc/organizations/demo/environments/_default
1.16         365        200      POST     https://chef.lxc/organizations/demo/environments/_default/cookbook_versions
0.04         12         201      POST     https://chef.lxc/organizations/demo/nodes
1.12         354        200      GET      https://chef.lxc/organizations/demo/nodes/bar-12-<N>
0.04         12         404      GET      https://chef.lxc/organizations/demo/nodes/bar-12-<N>
1.11         348        200      PUT      https://chef.lxc/organizations/demo/nodes/bar-12-<N>
1.16         366        404      GET      https://chef.lxc/organizations/demo/roles/chef-client
90.57        28509      200      GET      https://chef.lxc:443/bookshelf/<...>
2017-08-31T15:04:43Z Reading log file /var/log/chef-load/chef-load.log.4
2017-08-31T15:04:44Z Reading log file /var/log/chef-load/chef-load.log.5
2017-08-31T15:04:44Z Reading log file /var/log/chef-load/chef-load.log.6
2017-08-31T15:04:44Z Printing profile of API requests
Total API Requests: 59128
% of Total | Subtotal | Status | Method | URL
2.56         1516       204      POST     https://automate.lxc/data-collector/v0/
0.03         20         201      POST     https://chef.lxc/organizations/demo/clients
1.08         641        409      POST     https://chef.lxc/organizations/demo/clients
1.16         685        200      GET      https://chef.lxc/organizations/demo/environments/_default
1.16         684        200      POST     https://chef.lxc/organizations/demo/environments/_default/cookbook_versions
0.03         20         201      POST     https://chef.lxc/organizations/demo/nodes
1.12         665        200      GET      https://chef.lxc/organizations/demo/nodes/bar-12-<N>
0.03         20         404      GET      https://chef.lxc/organizations/demo/nodes/bar-12-<N>
1.10         651        200      PUT      https://chef.lxc/organizations/demo/nodes/bar-12-<N>
1.16         685        404      GET      https://chef.lxc/organizations/demo/roles/chef-client
90.55        53541      200      GET      https://chef.lxc:443/bookshelf/<...>
```

## Using sample JSON data files

chef-load is able to use files containing ohai, converge status and compliance status data captured from real nodes. This helps by simulating more accurate API payloads.

chef-load's configuration file has the following settings to specify the path to sample data files.

* ohai_json_file
* converge_status_json_file
* compliance_status_json_file — also used as the body of `inspec_report` messages when `enable_compliance_phase = true`
* compliance_sample_reports_dir

The chef-load GitHub repo's ["sample-data" directory](https://github.com/jeremiahsnapp/chef-load/tree/master/sample-data) has a file for each type of data that can be used.

#### Get the compliance sample inspec reports
The samples are located in the chef-load GitHub repo listed above inside of a subdirectory named "inspec-reports".  These are currently used for the generate action (back filling dates)

#### Create your own sample ohai JSON file

An ohai JSON file can be created by running "ohai > example-ohai.json".

#### Create sample compliance status JSON file

A compliance status JSON file can be created by adding the "json-file" reporter to a node's "audit" cookbook's attribute. The next Chef Client run will save the compliance status report in the "/var/chef/cache/cookbooks/audit/" directory.

Alternatively, you can simply run the "inspec exec" command along with the "--format json" option to execute inspec profiles against a node. The resultant JSON file can be used with the compliance_status_json_file chef-load option.

#### Create sample converge status JSON file

You can copy the following code to a file named capture-converge-status.rb in any cookbook's "libraries" directory and it will cause the next Chef Client run to save the converge status JSON data to "/tmp/converge-status.json" on the node. Once you capture the data you can remove the code from the cookbook.

```
class Chef
  class DataCollector
    class Reporter < EventDispatch::Base
      private
      def send_to_data_collector(message)
        return unless data_collector_accessible?
        IO.write('/tmp/converge-status.json', JSON.generate(message))
        http.post(nil, message, headers)
      end
    end
  end
end
```

## Generation of nodes, actions and/or reports

Chef-load can also be used to generate a subset of data into a Chef Automate server.

### Generate Nodes (chef-client runs)

```
chef-load generate --nodes 60 --data_collector_url "https://automate.example.org/data-collector/v0/"
```

### Generate Chef Actions

```
chef-load generate --actions 60 --data_collector_url "https://automate.example.org/data-collector/v0/"
```

### Generate both Chef Actions and Nodes (chef-client runs)

```
chef-load generate -n 60 -a 60 --data_collector_url "https://automate.example.org/data-collector/v0/"
```

### Generate data using a config file

```
chef-load generate --config chef-load.toml
```

## Build chef-load from source

### Natively with Go

To build chef-load you must have Go installed.  
Ref: https://golang.org/dl/

Get and install chef-load

```
go get github.com/chef/chef-load
```

It is easy to cross-compile chef-load for other platforms.  
Options for $GOOS and $GOARCH are listed in the following link.  
Ref: https://golang.org/doc/install/source#environment

The following command will create a chef-load executable file for linux amd64 in `$PWD/bin`:

```
make bin
```

You can also set the OS and arch:
```
make bin GOOS=$GOOS GOARCH=$GOARCH
```

### Natively with Habitat

To build chef-load using habitat you have to first, install Habitat: https://www.habitat.sh/docs/install-habitat/

Get and install chef-load
```
$ git clone https://github.com/chef/chef-load
```

Enter the Habitat studio and build chef-load:
```
$ cd chef-load
$ hab studio enter
[1][default:/src:0]# build

-- truncated-output --

chef-load:
   chef-load: I love it when a plan.sh comes together.
   chef-load:
   chef-load: Build time: 0m35s
[2][default:/src:0]#
```

# License

chef-load - a tool that simulates Chef Client API load on a Chef Server and/or a Chef Automate server

|                      |                                          |
|:---------------------|:-----------------------------------------|
| **Author:**          | Jeremiah Snapp (<jeremiah@chef.io>)
| **Copyright:**       | Copyright 2016-2017, Chef Software, Inc.
| **License:**         | Apache License, Version 2.0

```
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```

chef-load

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README