IRFuzzer

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 22 Open GitHub ↗

[ICSE'25] Specialized Fuzzing for LLVM Backend Code Generation

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# IR Fuzzer

You can find the camera ready paper [here](https://github.com/user-attachments/files/19461085/IRFuzzer.pdf)

This repo hasn't been actively maintained, but I will review and accept any PR if that helps.

# Quick start

## Compile

You should be able to prepare everything by running `./build.sh`. 
It should compile everything for you.
If it failed for any reason, please send an issue to this repo.

The script will set some environment variables. 
You may want to leave these in your `.bashrc` for further fuzzing:

```sh
# Path to this directory
export FUZZING_HOME=$(pwd)
# The LLVM you want to fuzz
export LLVM=<Your LLVM>
export AFL=AFLplusplus
export PATH=$PATH:$HOME/clang+llvm/bin
# Tell AFL++ to only use our mutator
export AFL_CUSTOM_MUTATOR_ONLY=1
# Tell AFL++ Where our mutator is
export AFL_CUSTOM_MUTATOR_LIBRARY=$FUZZING_HOME/mutator/build/libAFLCustomIRMutator.so
# AFL instrumentation method
export AFL_LLVM_INSTRUMENT=CLASSIC
```

If you want to use dockerized environment, you can also do

```sh
docker build . -t irfuzzer
```

## Seed selection

Seed is the initial input we give fuzzers, they have a directly impact on fuzzing performance.
`seeds` provides a default seed start fuzzing, it is an empty module with some function signatures.
For better fuzzing performance, you are more than welcome to move modules in `$LLVM/llvm/test/CodeGen/<Arch>` into `seeds`.
Notice that `seeds` only accepts bytecode, not LLVM IR.

## Run

### Env vars

You can specify different arguments for the driver using environment variables.

**Required**

```sh
export TRIPLE=<Your triple>
export CPU=
export ATTR=
```
You can specify triples like `x86_64`, `aarch64`, `aie`, etc. 
If you don't know what triples you have, try `llc --version`, it will list all triples you have.
`CPU` and `ATTR` can be left empty, but it is a must have.
They are equivalent to `-mcpu` and `-mattr` you would normally put when using `llc`.

```sh
export MATCHER_TABLE_SIZE=13780
```
Matcher table size refers to the size of the matcher table generated by TableGen. 
The table is automatically generated as a static variable in  in `SelectCode(SDNode *N) <Target>GenDAGISel.inc`(For SelectionDAG) and in `<Target>InstructionSelector::getMatchTable() <Target>GenGlobalISel.inc`(For GlobalIsel). You have three ways to find its length:

1. every time AFL's compiler compiles the project, it counts the table size and pops a `[+] MatcherTable size: 22660`. You can look out for that.
2. If you missed it, you can delete the object file (`ISelDAGToDAG.cpp.o` or `InstructionSelector.cpp.o`) and force a re-compilation.
```sh
$ cd build-afl
$ rm lib/Target/AIE/CMakeFiles/LLVMAIECodeGen.dir/AIEISelDAGToDAG.cpp.o
$ ninja

[6/27] Building CXX object lib/Target/AIE/CMakeFiles/LLVMAIECodeGen.dir/AIEISelDAGToDAG.cpp.o
[+] MatcherTable size: 22660
```
3. You can also find this data in [`scripts/common.py`](./scripts/common.py). 
It may not be 100% accurate as the code gets updated.

**Optional**

```
export GLOBAL_ISEL=1;
```
By default, we are fuzzing SelectionDAG. If you want to fuzz GlobalIsel, attach this environment variable. Please make sure `MATCHER_TABLE_SIZE` matches with GlobalIsel's table size.

### Command line

**Once the environments are set**, the easiest way to start fuzzing is to do
```sh
./AFLplusplus/afl-fuzz -i <seed-dir> -o fuzzing llvm-isel-afl/build/isel-fuzzing
```
It would start a fuzzing instant to fuzz SelectionDAG.
Some useful argument you might give `afl-fuzz` includes:
- `-E <n>`: execute/mutate the input for `n` times and quit
- `-V <t>`: run the fuzzer for `t` seconds and quit

Fuzzing can take weeks, if not days.
I recommend using [`screen`](https://www.gnu.org/software/screen/) to run the fuzzing in the background.

AFL++ will give you a fancy UI to describe what's happening. 
You may check [this](https://github.com/mirrorer/afl/blob/master/docs/status_screen.txt) page to help you understand the stats.

### Archs and table size

Check `./script/common.py`.

## Scripts

### Dependencies

We prepared many scripts to automate the fuzzing process.
These scripts runs on Python 3.10+, as it supports type hints to make it look less messy.
Use `python3.10` explictly to avoid conflict with `python3.6`... suppose you are still using ubuntu 18.04 or order.
To install some dependencies you may want to:
```sh
# If your ubuntu is so old you don't have python3.10 in your apt I can't help you...
# `apt install -y python3.10 python3-pip wget`
wget https://bootstrap.pypa.io/get-pip.py
python3.10 get-pip.py

# You can install all the dependencies of the scripts with:
pip3.10 install -r scripts/requirements.txt
```

### Description and usage

- `common.py`: this is not intended to be directly called, yet it have many metadata inside, you are welcome to take a look.
- `fuzz.py`: this fuzzes a lot of triples using `docker` or `screen`. 
- `batch_classify.py`: this script runs all the crashed inputs and cluster the same ones together using the stack trace. You may want to run this after a fuzzing process.
- `combine-fuzzing-results.py`: this script combines multiple fuzzing directories into one. If you are not writing a paper and need massive data you probably don't need it.
- `process_data.py`: summarize the fuzzing result.

Using `fuzz.py` don't need you to set any environment variables, the script will take care of it.
You would most likely use the `fuzz.py` like this:

```sh
python3.10 scripts/fuzz.py -i seeds -o fuzzing -r 5 --set="  aie" --type=screen --isel=dagisel --fuzzer=irfuzzer --time=1w -j 80 --on_exist=force
```

It means: start fuzzing using input from `seeds` (`-i seed`), put the result in `fuzzing` (`-o fuzzing`), repeat the experiment for five times (`-r 5`), test aie without attribute and cpu setting (`--set="  aie"`), use screen to monitor the fuzzing (`--type=screen`), test SelectionDAG (`--isel=dagisel`), use our fuzzer (`--fuzzer=irfuzzer`), test for a week (`--time=1w`), start at most 80 jobs in parallel (`-j 80`) and if the output directory already exists, force remove it (`--on_exist=force`)

# How do we fuzz

See the details in our paper

# Trophies & Findings

(I think I will attach more links to keep track of these later)

## AI Engine
- AIE1 GlobalIsel lacks floating point support
    - G_FCONSTANT [fixed.](https://gitenterprise.xilinx.com/XRLabs/llvm-aie/pull/194)
- AIE1 GlobalIsel lacks vector support.
- AIE1 SelectionDAG has bugs in the memory store.
- AIE1 SelectionDAG has truncation errors. [Fixed.](https://gitenterprise.xilinx.com/XRLabs/llvm-aie/pull/161/)
- AIE1 `vst.spil` generates two stores to the same address. [PoC.](https://gitenterprise.xilinx.com/XRLabs/peano_usage/pull/15) [Fixed.](https://gitenterprise.xilinx.com/XRLabs/llvm-aie/pull/203)

## Open sourced architecture

See our [trophies repo](https://github.com/DataCorrupted/LLVM-fuzzing-trophies).

# FAQ

__Why build two versions of LLVM?__

One version is built by AFL's compiler, and another is built by LLVM14 and contains a new mutator we designed. 
AFL needs to inject some code to the AIE compiler to keep track of runtime info (Edge coverage, MatcherTable coverage, etc.)
Besides, the driver also depends on it.
The other version is the dependency for the mutator. You __can__ use AFL instrumented mutator, but it would slow down mutation speed and thus not recommended.

__Why fuzz a fork of AIE that is not up-to-date?__

Mainly because mutator also needs to understand the architecture we are fuzzing, although it only generates mid-end IR.
Therefore, until we merge mutator's code into AIE, all you can do is keep merging the code you want to test to mutator branch and compile everything.

__Are we fuzzing AIE2?__

Currently we are only fuzzing AIE1 since it is more complete than AIE2. 
But you can fuzz AIE2 if you want to. In principle fuzzing AIE1 is no different than AIE2. 
All you need to do is set `TRIPLE=aie2` and set `MATCHER_TABLE_SIZE` correctly.

__AIE compilation hangs__

It's an known issue that `Target/AIE/MCTargetDesc/AIEMCFormats.cpp` will take a long time (~10 minutes) to compile. A function in it `__cxx_global_var_init()` will cause the optimizer to run for a really long time. It is an interesting bug, but we haven't had time to fix it.

__What is a seed and what to use__

Seed is the initial file you give fuzzer to work on. 
Unfortunately, this is required for AFL. (libFuzzer can cold-start without seed).
In this repo, we included a minimal seed in `seeds/` so you can start fuzzing without really worrying about it.

However, academic research and industry practice have shown that a better seed can lead to better results. You may reach the same result faster or find behavior unseen before with different seeds.
So if you can manually craft some seeds to cover different codes you want to test, for example, if you want to focus on floating point, you can create seeds with floating point calculations in them.

To create a seed, you can write LLVM IR manually and convert it to bitcode using `llvm-as`. Or you can cast bitcode to IR using `llvm-dis` and change some of the instructions.

__Matcher table coverage is 0.0%__

Table coverage may be low but never 0.0% in any cases. Please make sure the matcher table is correctly instrumented.

1. Make sure your binary is linked against the library compiled by AFL.
2. Make sure AFL instrumented it. During compilation, there should be a line telling you `[+] Instrumenting matcher table.`

__What does the stats in AFL's UI mean?__

You may check [this](https://github.com/mirrorer/afl/blob/master/docs/status_screen.txt) page to help you understand the stats.

We introduced a new coverage, so `map density` shows two stats. The first one is edge coverage, which should reach 70~80% in a day or two, meaning that (almost) all control flow has been tested. 
The second stat is matcher table coverage. It shows how much the table has been referenced. The higher, the better.

__My fuzzer is running slow__

There are two reasons it could happen.
AFL has high file system interactions. Therefore, make sure your directory is not a nfs or any remotely mounted hard drive. If you want even faster speed, you can mount a tmpfs to do fuzzing in the memory.

Another reason is your seeds are taking a long time to execute. You may either choose smaller initial seeds or use shorter timeouts by adding `-t <timeout>` to AFL's arguments.

__Where are the crashes located?__

`$FUZZING_HOME/fuzzing_output/default/crashes`

__How to reproduce errors?__

One upside of fuzzing is it always gives you reproducible PoC. 
You can run `build-release/bin/llc <args> <crashing-input>`.

We have also find cases where `llc` won't reproduce. 
In that case try
```sh
export CPU=<YourCPU>
export ATTR=<YourAttr>
export TRIPLE=<YourTriple>
export MATCHER_TABLE_SIZE=<YourSize>
./llvm-isel-fuzzing/build/isel-fuzzing < <input>
```

We have noticed some setting difference between `llc` and our driver `isel-fuzzing`.
We haven't had time to deal with it. Will update this later.

If there are any input that can't be reproduced even using `isel-fuzzing`, there are two possibilities:
- Your matcher table size is set wrong.
- It may be a bug and please send us an issue.

__What if `MatcherTable` is not set or set incorrectly?__

To pass compilation and AFL's self-testing, `MATCHER_TABLE_SIZE` is defaulted to a small amount. You would most like to see `Shadow table size: 32 too small. Did you set it properly?` that means it is not set.
If `MATCHER_TABLE_SIZE` is not set correctly, you will have false positives where the seed is stored in `crashes` (Indicating the fuzzer finds the seed crashing), but you can't reproduce it with `llc`. 
That means the runtime code we injected is crashing, not the LLVM itself. Most likely, it's because `MATCHER_TABLE_SIZE` is set too small, and an OOB Write happened.

__My mutator aborted during fuzzing?__

This is a common issue, its not a bug in the mutator.
Most likely you didn't set the types correctly. 
If mutator can't find a typed value to complete an instruction generation, it aborts.
Therefore, it is important to write all types when creating the mutator.

Mutator is non-deterministic, debuging is hard. 
But here's a trick, the mutator is deterministic is the seed is the same.
If your fuzzer crashed, go find the `.cur_input` in your repo, this is the last input that mutator worked on before it crashed.
Use `./mutator/scripts/validate.sh .cur_input` to verify the mutator with this input.
The script will (hopefully) give you the seed that crashed the mutator.
You can then debug the mutator by providing it with a deterministic seed that validator just poped out: `./mutator/build/MutatorDriver .cur_input <seed>`.
If you can confirm that the last stack trace is `SourcePred.generate`, that's it, you didn't provide all the types required.
If you see any other reasons for crashing, contact me.

Also, when mutator dies, the fuzzer become a zombie process, don't forget to clean it up :)

IRFuzzer

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README