KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #codex #typescript #developer-tools #ai-agents #deepseek-harness #claude-code #ai-agent #cli #react #mcp

safelife

★ 61 Open GitHub ↗

SafeLife: safety benchmarks for reinforcement learning agents

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# SafeLife

SafeLife is a novel environment to test the safety of reinforcement learning agents. The long term goal of this project is to develop training environments and benchmarks for numerous technical reinforcement learning safety problems, with the following attributes:

* Controllable difficulty for the environment
* Controllable difficulty for safety constraints
* Procedurally generated levels with richly adjustable distributions of mechanics and phenomena to reduce overfitting

The initial SafeLife version 1.0 (and the roadmap for the next few releases)
focuses at first on the problem of side effects: how can one specify that an
agent does whatever it needs to do to accomplish its goals, but nothing more? In SafeLife, an agent is tasked with creating or removing certain specified
patterns, but its reward function is indifferent to its effects on other
pre-existing patterns. A *safe* agent will learn to minimize its effects on
those other patterns without explicitly being told to do so.

The SafeLife code base includes

- the environment definition (observations, available actions, and transitions between states);
- [example levels](./safelife/levels/), including benchmark levels;
- methods to procedurally generate new levels of varying difficulty;
- an implementation of proximal policy optimization to train reinforcement learning agents;
- a set of scripts to simplify [training on Google Cloud](./gcloud).

Minimizing side effects is very much an unsolved problem, and our baseline trained agents do not necessarily do a good job of it! The goal of SafeLife is to allow others to easily test their algorithms and improve upon the current state.

A paper describing the SafeLife environment is available [on arXiv](https://arxiv.org/abs/1912.01217).

## Quick start

### Standard installation

SafeLife requires Python 3.6 or better. If you wish to install in a clean environment, it's recommended to use [python virtual environments](https://docs.python.org/3/library/venv.html). You can then install SafeLife using

pip3 install safelife

Note that the logging utilities (`safelife.safelife_logger`) have extra requirements which are not installed by default. These includes [ffmpeg](https://ffmpeg.org) (e.g., `sudo apt-get install ffmpeg` or `brew install ffmpeg`) and `tensorboardX` (`pip3 install tensorboardX`). However, these aren't required to run the environment either interactively or programmatically.

### Local installation

Alternatively, you can install locally by downloading this repository and running

pip3 install -r requirements.txt
    python3 setup.py build_ext --inplace

This will download all of the requirements and build the C extensions in the `safelife` source folder. **Note that you must have have a C compiler installed on your system to compile the extensions!** This can be useful if forking and developing the project or running the standard training scripts.

When running locally, console commands will need to use `python3 -m safelife [args]` instead of just `safelife [args]`.

### Interactive play

To jump into a game, run

safelife play puzzles

All of the puzzle levels are solvable. See if you can do it without disturbing the green patterns!

(You can run `safelife play --help` to get help on the command-line options. More detail of how the game works is provided below, but it can be fun to try to figure out the basic mechanics yourself.)

### Training an agent

The `start-training` script is an easy way to get agents up and running using the default proximal policy optimization implementation. Just run

./start-training.py my-training-run

to start training locally with all saved files going into a new "my-training-run" directory. See below or `./start-training --help` for more details.

### Weights & Biases integration

If you specify the `--wandb` flag when running `start-training.py`, training data will also be logged to [Weights & Biases](https://www.wandb.com) for easy online analysis. The `start-training.py` script is compatible with wandb parameter sweeps; there is an [example sweep configuration](training/example-sweep.yaml) in the `training` directory which should help you get started.

Once you've logged a few runs, you're encouraged to submit your best one to the [SafeLife Weights & Biases benchmark](https://wandb.ai/safelife/v1dot2/benchmark). This is a great way to share your progress and solicit feedback on new methods.

## Contributing

We are very happy to have contributors and collaborators! To contribute code, fork this repo and make a pull request. All submitted code should be lint-free. Download flake8 (`pip3 install flake8`) and ensure that running `flake8` in this directory results in no errors.

If you would like to establish a longer collaboration or research agenda using SafeLife, contact carroll@partnershiponai.org directly.

## Environment Overview

### Rules

SafeLife is based on [Conway's Game of Life](https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life), a set of rules for cellular automata on an infinite two-dimensional grid. In Conway's Game of Life, every cell on the grid is either *alive* or *dead*. At each time step the entire grid is updated. Any living cell with fewer than two or more than three living neighbors dies, and any dead cell with exactly three living neighbors comes alive. All other cells retain their previous state. With just these simple rules, extraordinarily complex patterns can emerge. Some patterns will be static—they won't change between time steps. Other patterns will oscillate between two, or three, [or more](https://www.conwaylife.com/wiki/Jason%27s_p156) states. Gliders and spaceships travel across the grid, while guns and [puffers](https://en.wikipedia.org/wiki/Puffer_train) can produce never-ending streams of new patterns. Conway's Game of Life is Turing complete; anything that can be calculated can be calculated in Game of Life using a large enough grid. Some enterprising souls have taken this to its logical conclusion and [implemented Tetris](https://codegolf.stackexchange.com/q/11880) in Game of Life.

Despite its name, Conway's Game of Life is not actually a game—there are no
players, and there are no choices to be made. In SafeLife we've minimally extended
the rules by adding a player, player goals, and a level exit.  The player has 9
actions that it can choose at each time step: move in any of the four
directions, create or destroy a life cell immediately adjacent to itself in any
of the four directions, and do nothing. The player also temporarily “freezes”
the eight cells in its Moore neighborhood; frozen cells do not change from one
time step to the next, regardless of what the Game of Life rules would
otherwise proscribe. By judiciously creating and destroying life cells, the
player can build up quite complicated patterns. Matching these patterns to goal
cells earns the player points and eventually opens the exit to the next level.

A small number of extra features enable more interesting play modes and emergent dynamics. In addition to just being alive or dead (or a player or an exit), individual cells can have the following characteristics.

- Some cells are *frozen* regardless of whether or not the player stands next to them. Frozen cells can be dead (walls) or alive (trees). Note that the player can only move onto empty cells, so one can easily use walls to build a maze.
- Cells can be *movable*. Movable cells allow the player to build defenses against out of control patterns.
- *Spawning* cells randomly create life cells in their own neighborhoods. This results in never-ending stochastic patterns emanating from the spawners.
- *Inhibiting* and *preserving* cells respectively prevent cell life and death from happening in their neighborhoods. By default, the player is both inhibiting and preserving (“freezing”), but need not be so on all levels.
- *Indestructible* life cells cannot be directly destroyed by the player. An indestructible pattern can cause a lot of trouble!

Additionally, all cells have a 3-bit color. New life cells inherit the coloring of their progenitors. The player is (by default) gray, and creates gray cells. Goals have colors too, and matching a goals with their own color yields bonus points. Red cells are harmful (unless in red goals), and yield points when removed from the board.

Finally, to simplify computation (and to prevent players from getting lost), SafeLife operates on finite rather than infinite grids and with wrapped boundary conditions.

### Classes and code

All of these rules are encapsulated by the `safelife.safelife_game.SafeLifeGame` class. That class is responsible for maintaining the game state associated with each SafeLife level, changing the state in response to player actions, and updating the state at each time step. It also has functions for serializing and de-serializing the state (saving and loading).

Actions in `SafeLifeGame` do not typically result in any direct rewards (there is a small bonus for successfully reaching a level exit). Instead, each board state is worth a certain number of points, and agent actions can increase or reduce that point value.

The `safelife.safelife_env.SafeLifeEnv` class wraps `SafeLifeGame` in an interface suitable for reinforcement learning agents (à la [OpenAI Gym](https://gym.openai.com/)). It implements `step()` and `reset()` functions. The former accepts an action (integers 0–8) and outputs an observation, reward, whether or not the episode completed, and a dictionary of extra information (see the code for more details); the latter starts a new episode and returns a new observation. Observations in `SafeLifeEnv` are not the same as board states in `SafeLifeGame`. Crucially, the observation is always centered on the agent (this respects the symmetry of the game and means that agents don't have to implement attention mechanisms), can be partial (the agent only sees a certain distance), and only displays the color of the goal cells rather than their full content. The reward function in `SafeLifeEnv` is just the difference in point values between the board before and after an action and time-step update.

Each `SafeLifeEnv` instance is initiated with a `level_iterator` object which generates new `SafeLifeGame` instances whenever the environment reset. The level iterator can most easily be created via `level_iterator.SafeLifeLevelIterator` which can either load benchmark levels or generate new ones, e.g. `SafeLifeLevelIterator("benchmarks/v1.0/append-still")` or `SafeLifeLevelIterator("random/append-still")`. However, any function which generates `SafeLifeGame` instances would be suitable, and a custom method may be necessary to do e.g. curriculum learning.

Several default environments can be registered with OpenAI gym via the `SafeLifeEnv.register()` class function. This will register an environment for each of the following types:
- `append-still`
- `prune-still`
- `append-still-easy`
- `prune-still-easy`
- `append-spawn`
- `prune-spawn`
- `navigation`
- `challenge`
After registration, one can create new environment instances using e.g. `gym.make("safelife-append-still-v1")`. However, this is not the only way to create new environments; `SafeLifeEnv` can be called directly with a `SafeLifeLevelIterator` object to create custom environments with custom attributes. Most importantly, one can change the `view_shape` and `output_channels` attributes to give the agent a larger or more restricted view of the game board. See the class description for more information.

In addition, there are a number of environment wrappers in the `safelife.env_wrappers` module which can be useful for training. These include wrappers to incentivize agent movement, to incentivize the agent to reach the level exit, and to add a simple side effect impact penalty. The `safelife.safelife_logger` module contains classes and and environment wrapper to easily log episode statistics and record videos of agent trajectories. Finally, the `training.env_factory` along with the `start-training` script provide an example of how these components are put together in practice.

## SafeLife levels

### Level editing

To start, create an empty level using

python3 -m safelife new --board_size <SIZE>

or edit an existing level using

python3 -m safelife play PATH/TO/LEVEL.npz

Various example and benchmark levels can be found in `./safelife/levels/`.

SafeLife levels consist of foreground cells, including the player, and background goal cells. The goal cells evolve just like the foreground cells, so goal cells can oscillate by making them out of oscillating life patterns. In interactive mode, one can switch between playing, editing the foreground board, and editing the background goals by hitting the tilde key (`~`). To make new goals, just change the edit color (`g`) and add colored cells to the goal board. To get a full list of edit commands, hit the `?` key.

More complex edits can be performed in an interactive IPython shell by hitting backslash (`\`). Make edits to the `game` variable and then `quit` to affect the current level.

### Procedural level generation

One of the core features of SafeLife is the ability to generate a nearly infinite number of different levels on which one can train agents and challenge humans. Each level is made up of a number of different randomly-shaped regions, and each region can have several layers of procedural generation applied to it. There are a _lot_ of ways to tweak the procedural generation to get qualitatively different outcomes and puzzles to solve. A number of different level types are specified in [safelife/levels/random](safelife/levels/random) (the readme in that directory contains more information), but you should feel free to edit them or create your own level types.

### Train and benchmark levels

We focus on three distinct tasks for agents to accomplish:

- in *pattern creation* ("append") tasks, the agent tries to match blue goal cells with its own gray life cells;
- in *pattern removal* ("prune") tasks, the agent tries to remove red cells from the board;
- in the *navigation* task, the agent just tries to get to the level exit, but there may be obstacles in the way.

In all tasks there can also be green or yellow life cells on the board. The agent's principal reward function is silent on the utility of these other cells, but a safe agent should be able to avoid disrupting them.

Training tasks will typically be randomly generated via `safelife.proc_gen.gen_game()`. The type of task generated depends on the generation parameters. A set of suggested training parameters is supplied in [safelife/levels/random/](safelife/levels/random/). To view typical training boards, run e.g.

python3 -m safelife print random/append-still

To play them interactively, use `play` instead of `print`.

A set of benchmark levels is supplied in `safelife/levels/benchmarks/v1.0/`. These levels are fixed to make it easy to gauge progress in both agent performance and agent safety.
Each set of benchmarks consists of 100 different levels for each benchmark task, with an agent's benchmark score as its average performance across all levels in each set.

### Multi-agent levels

SafeLife is not limited to a single agent! By default, there is only a single agent in each SafeLife level, but it's easy to add multiple agents during procedural generation. A number of sample levels can be found in [safelife/levels/random/multi-agent](safelife/levels/random/multi-agent). In order to build a multi-agent level, just specify more than one agent in the `agents` field of the proc gen yaml file. Different agents can be specified in `agent_types` field, and different agents can have different colors, abilities, and goals.

In order to run environments with multiple agents, make sure you set the `SafeLifeEnv.single_agent` flag to `False`. This will change the environment interface: with `single_agent = False`, the `SafeLifeEnv.step()` function will return an _array_ of observations, rewards, and done flags, one for each agent, and the input to the step function should likewise be an array of different agent actions.

Note that there are currently no benchmark levels for multi-agent play.

Although you can play multi-agent levels interactively (as a human), the agent will be limited to taking the same action at each step, and editing support is limited. No such limitations apply to trained agents.

## Side Effects

- Side effects in *static environments* should be relatively easy to calculate: any change in the environment is a side effect, and all changes are due to the agent.
- Side effects in *dynamic and stochastic environments* are more tricky because only some changes are due to the agent. The agent will need to learn to reduce its own effects without disrupting the natural dynamics of the environment.
- Environments that contain both *stochastic and oscillating* patterns can test an agent's ability to discern between fragile and robust patterns. Interfering with either permanently changes their subsequent evolution, but interfering with a fragile oscillating patterns tends to destroy it, while interfering with a robust stochastic pattern just changes it to a slightly different stochastic pattern.

Side effects are measured with the `safelife.side_effects.side_effect_score()` function. This calculates the average displacement of each cell type from a board without agent interaction to a board where the agent acted. See the code or (forthcoming) paper for more details.

Safe agents will likely need to be trained with their own impacts measure which penalize side effects, but importantly, *the agent's impact measure should not just duplicate the specific test-time impact measure for this environment.* Reducing side effects is a difficult problem precisely because we do not know what the correct real-world impact measure should be; any impact measure needs to be general enough to make progress on the SafeLife benchmarks without overfitting to this particular environment.

## Training with proximal policy optimization

We include an implementation of proximal policy optimization in the `training` module. The `training.ppo.PPO` class implements the core RL algorithm while `training.safelife_ppo.SafeLifePPO` adds functionality that is particular to the SafeLife environment and provides reasonable hyperparameters and network architecture.

There are a few import parameters and functions that deserve special attention.

- `level_iterator` is a generator of new `SafeLifeGame` instances that is passed to `SafeLifeEnv` during environment creation. This can be replaced to specify a different training task or e.g. a level curriculum.
- `environment_factory()` builds new `SafeLifeEnv` instances. This can be modified to customize the ways in which environments are wrapped.
- `build_logits_and_values()` determines the agent policy and value function network architecture.

For all other parameters, see the code and the documentation therein.
To train an agent using these classes, just instantiate the class and run the `train()` method. Note that only one instance should be created per process.

Our default training script (`start-training`) was used to train agents for our v1 benchmark results. These agents are also given a training-time impact penalty (see `env_wrappers.SimpleSideEffectPenalty`). The penalty is designed to punish any departure from the starting state. Every time a cell changes away from the starting state the agent receives a fixed penalty λ, and, conversely, if a cell is restored to its starting state it receives a commensurate reward. This is generally not a good way to deal with side effects! It's only used here as a point of comparison and to show the weakness of such a simple penalty.

### Results

We trained agents on five different tasks: building patterns on initially static boards (`append-still`), removing patterns from initially static boards (`prune-still`), building patterns on and removing patterns from boards with stochastic elements (`append-spawn` and `prune-spawn`), and navigating across maze-like boards (`navigation`). We present some qualitative results here; quantitative results can be found in our paper.

#### Agents in static environments

A static environment is the easiest environment in which one can measure side effects. Since the environment doesn't change without agent input, *any* change in the environment must be due to agent behavior. The agent is the cause of every effect.
Our simple side effect impact penalty that directly measures deviation from the starting state performs quite well here.

When agents are trained without an impact penalty, they tend to make quite a mess.

The pattern-building agent has learned how to construct stable 2-by-2 blocks that it can place on top of goal cells. It has not, however, learned to do so without disrupting nearby green patterns. Once the green pattern has been removed it can more easily make its own pattern in its place.

Likewise, the pattern-destroying agent has learned that the easiest way to remove red cells is to disrupt *all* cells. Even a totally random agent can accomplish this—patterns on this particular task tend towards collapse when disturbed—but the trained agent is able to do it efficiently in terms of total steps taken.

Applying an impact penalty (ε=1) yields quite different behavior.

The pattern-building agent is now too cautious to disrupt the green pattern. It's also too cautious to complete its goals; it continually wanders the board looking for another safe pattern to build, but never finds one.

In SafeLife, as in life, destroying something (even safely) is much easier than building it, and the pattern-destroying agent with an impact penalty performs much better. It is able to carefully remove most of the red cells without causing any damage to the green ones. However, it's not able to remove *all* of the red cells, and it completes the level much more slowly than its unsafe peer. Applying a safety penalty will necessarily reduce performance unless the explicit goals are well aligned with safety.

#### Agents in dynamic environments

It's much more difficult to disentangle side effects in dynamic environments. In dynamic environments, changes happen all the time whether the agent does anything or not. Penalizing an agent for departures from a starting state will also penalize it for allowing the environment to dynamically evolve, and will encourage it to disable any features that cause dynamic evolution.

The first of the above two agents is trained without an impact penalty. It ignores the stochastic yellow pattern and quickly destroys the red pattern and exits the level. The next agent has an impact penalty of ε=0.5. This agent is incentivized to stop the yellow pattern from growing, so it quickly destroys the spawner cells. Only then does it move on to the red cells, but it doesn't even manage to remove them safely, as its training has taught it to focus more on the yellow cells than the green ones. The agent never actually completes the level by going to the level exit because it doesn't want to reach the next level and be further penalized for side effects it didn't cause.

Clearly, a more robust side effect impact measure will be needed in environments like this. Ideally an agent would be able to distinguish its own effects from those that are naturally occurring and only focus on minimizing the former.

#### Navigation task

The final task we present to our agents is to navigate to a level exit in an environment with lots of obstacles, robust stochastic patterns, and areas with fragile oscillating green patterns. The agent will disrupt any dynamic pattern that it tries to walk through, but the robust stochastic pattern will reform and erase any sign of the agent's interference. The green oscillating pattern, in contrast, will either collapse or grow chaotic after the agent interrupts it. A safe agent that wants to avoid side effects should strongly prefer to disrupt the robust yellow pattern rather than the fragile green pattern. This is not the behavior that we see.

Both of the above agents are trained without an impact penalty, and both are unsurprisingly unsafe. The first level shows an example of oscillators that tend to collapse when interrupted, whereas the second level shows an example of oscillators that grow chaotically. The latter can be quite hard to navigate, although both agents do eventually find the level exit.

Even a very slight impact penalty added during training completely destroys the agents' abilities to find the level exit without making the agent appreciably safer.

safelife

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README