Kratos-benchmark

KO | EN

gitlite — search

#ai #machine-learning #linux #docker #macos #nodejs #java #windows #cli #kubernetes #vue #security #pytorch #mcp

Kratos-benchmark

(★ 12)

Kratos: An FPGA Benchmark for Unrolled Deep Neural Networks with Fine-Grained Sparsity and Mixed Precision

Kratos-benchmark 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Kratos-explorer: an extension of Kratos-benchmark

This fork aims to create a more flexible test bench that supports easy exploration of multiple parameters, and automatic verification of final implementations.

## Kernel table

![kernel_table](https://raw.githubusercontent.com/abdelfattah-lab/Kratos-benchmark/main/<./img/kernel table.png>)

## Code structure

### `verilog`

Within this folder:
- Required SystemVerilog libraries
- `conv_1d`, `conv_2d`, `gemms`, `gemmt`: benchmark modules
- Other Verilog dependencies

### `structure`

Within this folder:
- `arch`: contains the `ArchFactory` class that must be implemented to generate a corresponding architecture file.
- `design`: contains the `Design` class that must be implemented to generate a corresponding SystemVerilog wrapper, as well as TCL and SDC files (for Quartus). Additional methods must be implemented for automatic verification:
    - `PluginDesign` allows insertion of a `Plugin` in parallel with the existing design, useful for stress testing concurrent circuits.
- `blif`: contains `BlifDesign`, which generates a .blif file intended for use with VPR-only flows.
- `plugin`: contains the `Plugin` class that can be passed into a `PluginDesign` to generate a wrapper with both the original design and the plugin in parallel.
- `exp` contains:
    - `Experiment`: instantiated with an `ArchFactory` and a `Design`, and then runs on a tool (e.g., VTR, Quartus) for generated architecture and design based on given parameters:
    - `ExperimentFactory`: parameters can be given as a list instead of a value (described under Usage/Parameters). This class can then generate a new `Experiment` for each unique combination of parameters.
- `run`: contains `Runner`, which takes in an `Experiment` subclass, as well as an `ArchFactory` and `Design`, and runs all possible combinations of experiments as generated by the `ExperimentFactory`.
- `test`: contains `VerilogImplTester`, intended for use with VTR to test a pre-VTR Verilog design against a post VTR synthesis/implementation netlist.
- `consts`: contains system-wide constants:
    - `keys`: recognized top-level parameter keys
    - `shared_defaults`: default values for certain parameters
    - `shared_requirements`: parameters that must be provided by users
    - `translation`: parameter keys to human label matching
- `util` provides several useful base classes:
    - `Abstract` provides a convenience function `raise_unimplemented()` to simulate interface behaviour of standard OOP languages, e.g., Java.
    - `ParamsChecker: Abstract` provides methods to check for required parameters and/or simultaneously fill in default values.
    - `DynamicallyNamed: Abstract` is an interface for objects that should have their names (i.e., a label) dynamically generated at runtime.
    - `Hashable` can be extended so the object can be used as a dictionary key.

### `impl`

This folder contains concrete implementations of the interfaces found in `structure`:
- Under `arch`:
    - `base` contains `BaseArchFactory`, the baseline FPGA architecture implementation from the Kratos paper.
- Under `design`:
    - `conv_1d`, `conv_2d`, `gemms`, `gemmt`: implementations for the various benchmarks
    - `vtr_full_benchmarks`: implementation that effectively copies an independent Verilog testbench with no dependencies, e.g., the [VTR benchmark suite](https://docs.verilogtorouting.org/en/latest/vtr/benchmarks/).
    - `simple_unrolled`: implementation of single-cycle registered multiplication with a constant
- Under `exp`, concrete implementations of `Experiment`:
    - `vtr` contains `VtrExperiment`, which will run the experiment on VTR.
    - `quartus` contains `QuartusExperiment`, which runs the experiment on Quartus.
    - `vpr` contains `VprExperiment`, which runs a VPR-only experiment, i.e., skips the synthesis stage (Yosys+Parmys, ABC) of VTR.
- Under `plugin`: `Plugin` implementations for various circuits of interest, e.g., the `sha` benchmark from the [VTR benchmark suite](https://docs.verilogtorouting.org/en/latest/vtr/benchmarks/).
- Under `test`:
    - `iverilog`: A concrete `VerilogImplTester` using [iverilog](https://github.com/steveicarus/iverilog).

### `runs`

This folder contains run sequences and benchmarks to use:
- Under `benchmarks`:
    - `kratos`: full-sized Kratos benchmarks.
    - `kratos_mini`: scaled-down Kratos benchmarks, for verification of flow correctness.
-There are the following run sequences:
    - `vtr_denoised_v1`:
        1. Runs all provided designs on v0 and v1 architecture on provided seeds.
        2. Averages results across all seeds for each architecture.
        3. Normalize v0 results as 1.0, and v1 results relative to v0. (Baseline normalization)
        4. Plots normalized results, and saves both baseline and normalized results to the default results folder, under the latest timestamp.
    - `vtr_denoised_same_arch`:
        1. Runs all provided designs on architecture provided on provided seeds.
        2. Averages results across all seeds for each possible parameter value in compare_param_key.
        3. Normalize results to base_param_value.
        4. Plots normalized results, and saves both baseline and normalized results to the default results folder, under the latest timestamp.
    - `vtr_all_designs`: Simply runs all provided designs, and reports if any failed.

## Usage

### Creating runs

Sample usage can be found in `sample.py`. Some key points to note:
- A run returns a a dictionary of `<experiment root directory>`: `pandas.DataFrame`, which then can be used to plot graphs, perform data analysis etc.
- You can provide `filter_params` and `filter_results` to the `run_all_threaded()` method to obtain a smaller DataFrame.

### Parameters

This test bench uses one dictionary passed into the runner. Class-specific parameters are then split using keys under `structure.consts.keys` (which we will shorten to `keys` here). A sample one (adapted from `sample.py`) is presented here with explanations:

```
{
    keys.KEY_EXP: {
        # Experiment parameters
        'root_dir': 'experiments/conv_2d_pw', # Base folder to create the experiment in
        'verilog_search_dir': os.path.join(os.path.dirname(os.path.realpath(__file__)), 'verilog') # Search path for SystemVerilog files
    },
    keys.KEY_ARCH: {
        # Architecture parameters
        'lut_size': 3 # LUT size used
    },
    keys.KEY_DESIGN: {
        # Design parameters
        'data_width': 4, # Data width
        'sparsity': 0.5, # Sparsity
        # ...and any other design-specific parameters
    }
}
```

#### Variable parameters

Provide all parameters that should be explored simultaneously as a **list**. The test bench will then automatically create and run all possible permutations.

If you would like to simultaneously explore LUT size and data width for example, then provide:
```
{
    # ...other parameters
    keys.KEY_ARCH: {
        # Architecture parameters
        'lut_size': [3, 4, 5] # Exploring sizes 3-5
        # ...other parameters
    },
    keys.KEY_DESIGN: {
        # Design parameters
        'data_width': [4, 8] # Exploring 4-bit, 8-bit
    }
}
```
In this example, the test bench will run experiments for:
- 3-LUT, 4-bit
- 4-LUT, 4-bit
- 5-LUT, 4-bit
- 3-LUT, 8-bit
- ...

In general, the resultant final number of experiments run will be `a * b * c * ...`, where `a, b, c, ...` are the lengths of each provided list.

### Background running

This section provides methods to ensure processes will continue running in the background with proper logging, so it can continue even when the terminal is closed, e.g., SSH connection terminated.

#### Using `tmux` (preferred)

`tmux` will work like a normal terminal, and you can disconnect and re-attach to sessions if required:
0. (If not installed) `sudo apt-get install tmux`
1. Make a new session with `tmux new -s <name>`.
2. Run whatever commands as required.
3. When the terminal is killed, you can resume the terminal with `tmux attach -t <name>`.

#### Using provided script (persistent log file)

`run_bg.sh` is provided to facilitate running of the Python scripts as background processes:
1. Ensure that `run_bg.sh` is executable, i.e., `chmod +x run_bg.sh`
2. Edit the parameters at the top of `run_bg.sh`:
    - `BASH_RUN_SCRIPT`: directs the script to run *another bash script* in the background.
    - It is *optional* to make this script executable, since `run_bg.sh` will modify the permissions for you.
    - A sample bash script to use, `run_with_venv.sh`, has been included.
    - Note the **known limitation in terminating the script**, as detailed at the top of the sample bash script.
3. To use the script:
    - Start execution: `./run_bg.sh start`
    - Stop execution: `./run_bg.sh stop`
    - Retart execution: `./run_bg.sh restart` (may not work as intended due to the limitation as detailed)

When the script is running, there will be a few files generated:
- `<script_name>.out`: This serves as the stdout of the script.
- `<script_name>.pid`: This logs the PID of the *parent bash script*.

### COFFE Maker

This repository contains `coffe_maker.py`, which is intended to be used with the organization-forked `COFFE`'s `coffe_drinker.py` in the following steps:
1. Use the `maker` to generate a `record.csv` file for all COFFE jobs to run, based on architecture and parameters.
2. Use the `drinker` to run all COFFE jobs in multiple processes, and generate an archive `.csv` file.
3. Insert this archive file at the appropriate location in `Kratos-explorer` so the `ArchFactory`s can look up historical COFFE estimates when required:
    - Inside a directory sharing the same root folder as the module file containing the `ArchFactory`. Directory name will depend on the `ArchFactory` (Default is `coffe_archives`).
    - Save the archive file with a filename specified by the `ArchFactory`, or else match the module filename: e.g., `arch_module.py` will look for `arch_module.csv` by default.
    - For example, for `GenExpArchFactory`, to save the archive file:
        - Directory is at `impl/arch/coffe_archives`.
        - Filename is `gen_exp.csv`.

Kratos-benchmark

Kratos-benchmark 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Kratos-benchmark

Kratos-benchmark 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README