CTrLBenchmark

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

★ 117 Open GitHub ↗

Benchmark for Lifelong learning research

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Continual Transfer Learning Benchmark
The CTrL benchmark aims to help research by studying the transfer behaviour of different models in the Lifelong Learning context.

For a quick presentation of the benchmark, please check out [this blog post](https://ai.facebook.com/blog/ctrl-and-mntdp-a-new-open-source-benchmark-and-model-for-continual-learning/).

For more details, the paper is available on [arxiv](https://arxiv.org/abs/2012.12631).

## Creating a Stream:

The `TaskGenerator` class is at the center of the implementation of the CTrL Benchmark. 
It gives access to a high-level API allowing to seamlessly generate a wide variety of streams with a loose coupling between the different components such as the underlying dataset(s), the strategy to generate the tasks (split, incremental, mixture of datasets, ...) and the processing to apply to each task.

The 3 main components of a `Task Generator` are:
- A pool of concepts to select from to generate the tasks. It can be a few classes, a full dataset or even a mixture of datasets.
- A pool of transformation that can be modified or combined to apply specific processing to the data for each task.
- A Strategy, describing how to combine the conecpts and trasnformation over time to generate an actual stream.

Each of these components can be created by hand or using our automatic `TaskGenerator` creation tool using yaml configuration files.

For examples simply executing
```python
import ctrl
task_gen = ctrl.get_stream('s_minus')
```
will return the corresponding task generator that be used either directy to generate tasks on the fly:
```python
t1 = task_gen.add_task()
t2 = task_gen.add_task()
t3 = task_gen.add_task()
...
```

or as an iterator:
```python
for t in task_gen:
    print(t)
```

### Available streams:
In the current version, only the streams of the CTrL benchmark are directly available, they can be obtained by passing the following `name` arguments in `ctrl.get_stream`:
- S+: `"s_plus"`
- S-: `"s_minus"`
- Sin: `"s_in"`
- Sout: `"s_out"`
- Spl: `"s_pl"`
- Slong: `"s_long"`

More documentation and details on the internal components will be progressively added.

See the CONTRIBUTING file for how to help out.

# LICENSE
See the LICENSE file.

CTrLBenchmark

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README