exLong

KO | EN

gitlite — search

#javascript #python #llm #golang #android #machine-learning #deep-learning #linux #macos #java #windows #awesome #kubernetes #pytorch

(★ 9)

exLong: Generating Exceptional Behavior Tests with Large Language Models

exLong 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🐲🔨 exLong: Generating Exceptional Behavior Tests with Large Language Models 
exLong is a large language model instruction-tuned from CodeLlama and embeds reasoning about 
- **traces** that lead to throw statements
- **conditional expressions** that guard throw statements
- **non-exceptional behavior tests** that execute similar traces

# About
This repo hosts the code and data for the following ICSE 2025 paper:

Title: [exLong: Generating Exceptional Behavior Tests with Large Language Models](https://arxiv.org/abs/2405.14619)

Authors: [Jiyang Zhang](https://jiyangzhang.github.io/), [Yu Liu](https://sweetstreet.github.io/), [Pengyu Nie](https://pengyunie.github.io/), [Junyi Jessy Li](https://jessyli.com/), [Milos Gligoric](http://users.ece.utexas.edu/~gligoric/)

```bibtex
@inproceedings{ZhangETAL25exLong,
  author = {Zhang, Jiyang and Liu, Yu and Nie, Pengyu and Li, Junyi Jessy and Gligoric, Milos},
  title = {exLong: Generating Exceptional Behavior Tests with Large Language Models},
  booktitle = {International Conference on Software Engineering},
  year = {2025},
}
```

We also include the implementation of the CLI tool described in our FSE 2025 Demo Paper:

Title: [A Tool for Generating Exceptional Behavior Tests With Large Language Models](https://arxiv.org/abs/2505.22818)

Authors: [Linghan Zhong](https://about.tongero.com), [Samuel Yuan](https://samuelyuan.com), [Jiyang Zhang](https://jiyangzhang.github.io/), [Yu Liu](https://sweetstreet.github.io/), [Pengyu Nie](https://pengyunie.github.io/), [Junyi Jessy Li](https://jessyli.com/), [Milos Gligoric](http://users.ece.utexas.edu/~gligoric/)

```bibtex
@inproceedings{ZhongETAL25exLongTool,
  author = {Zhong, Linghan and Yuan, Samuel and Zhang, Jiyang and Liu, Yu and Nie, Pengyu and Li, Junyi Jessy and Gligoric, Milos},
  title = {A Tool for Generating Exceptional Behavior Tests With Large Language Models},
  booktitle = {ACM International Conference on the Foundations of Software Engineering Demonstrations},
  year = {2025},
}
```

# Table of Contents
1. [Quick Start][sec-hf] 🤗
2. [Set Up][sec-setup] :rocket:
3. [Experiments][sec-exp] :construction_worker:
4. [Artifacts][sec-artifacts] :star:
5. [CLI][sec-cli] :computer:

# Quick Start
 
[sec-hf]: #quick-start

- The exLong dataset is on Hugging Face [🤗](https://huggingface.co/datasets/EngineeringSoftware/exLong-dataset)!
```bash
from datasets import load_dataset

with_name_ds = load_dataset("EngineeringSoftware/exLong-dataset", "with-EBT-name")
no_name_ds = load_dataset("EngineeringSoftware/exLong-dataset", "no-EBT-name")
```

- The exLong model is on Hugging Face [🤗](https://huggingface.co/EngineeringSoftware/exLong)!

```bash
pip install transformers accelerate bitsandbytes peft
```

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig

# Load the base model
base_model_name = "codellama/CodeLlama-7b-Instruct-hf"
base_model = AutoModelForCausalLM.from_pretrained(base_model_name)

# Load the LoRA configuration
peft_model_id = "EngineeringSoftware/exLong"
config = PeftConfig.from_pretrained(peft_model_id, revision="with-etest-name")  # set revision to "no-etest-name" for no EBT name

# Load the LoRA model
model = PeftModel.from_pretrained(base_model, peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(base_model_name)

prompt = """<s>[INST] <<SYS>>
You are a helpful programming assistant and an expert Java programmer. You are helping a user writing exceptional-behavior tests for their Java code.
<</SYS>>

Please complete an exceptional behavior test method in Java to test the method 'factorial' for the exception 'IllegalArgumentException'.
The method to be tested is defined as:
```java
public static long factorial(int n) {
    if (n < 0) {
        throw new IllegalArgumentException("Number must be non-negative.");
    }
    long result = 1;
    for (int i = 1; i <= n; i++) {
        result *= i;
    }
    return result;
}
` ` `
Please only give the new exceptional-behavior test method to complete the following test class. Do NOT use extra libraries or define new helper methods. Return **only** the code in the completion:
```java
public class FactorialTest {
}
` ` `
"""

input_ids = tokenizer(prompt, return_tensors="pt").input_ids

# Generate code
output = model.generate(
    input_ids=input_ids,
    max_new_tokens=100,
    temperature=0.2,      # Sampling temperature (lower is more deterministic)
    top_p=0.95,           # Top-p (nucleus) sampling
    do_sample=True        # Enable sampling
)

# Decode and print the generated code
generated_code = tokenizer.decode(output[0], skip_special_tokens=True)
print("Generated Code:")
print(generated_code)
```

# Set Up
[sec-setup]: #set-up

## Dependencies Set Up
[sec-dep]: #dependencies-set-up

1. Create conda environment
```bash
conda create -n exlong python=3.9
conda activate exlong
pip install -r requirements.txt
```

2. We used [axolotl](https://github.com/axolotl-ai-cloud/axolotl) to fine-tune the CodeLlama model. If you want to train your own model, install the extra dependencies
```bash
# we used an older version of axolotl to train the models
git clone git@github.com:JiyangZhang/axolotl-exlong.git
cd axolotl-exlong/
conda activate exlong
pip install packaging
# set CUDA_HOME
export CUDA_HOME=/opt/apps/cuda/12.0/
pip3 install -e '.[flash-attn,deepspeed]'
```
## Experiments Set Up
[sec-setupexp]: #experiments-set-up

1. Download raw dataset
```bash
mkdir -p _work/data/
mkdir -p _work/exp/
mkdir -p _work/setup/

wget -L https://utexas.box.com/shared/static/hfcp4za3j9vp8lh5u8iviadixuxu8080.gz -O raw-data.tar.gz
tar -xzf raw-data.tar.gz -C _work/data/
mv _work/data/etestgen-raw-data-12k _work/data/ne2e

wget -L https://utexas.box.com/shared/static/4m7mntp0ix18dkl1ikkspcmpuvybfs1f.gz -O ne2e-test.tar.gz
tar -xzf ne2e-test.tar.gz -C _work/data/

wget -L https://utexas.box.com/shared/static/y4e52k5x8vk8vcr59lg33gebcg2m1caw.gz -O rq2.tar.gz
tar -xzf rq2.tar.gz -C _work/data/

# netest-diversity
wget -L https://utexas.box.com/shared/static/j417e93j1rdvdqz2yobttygfhucfbkjm.gz -O netest-diversity.tar.gz
tar -xzf netest-diversity.tar.gz -C _work/data/

```
You should see  `_work/data/ne2e`, `_work/data/rq1-eval`, `_work/data/rq2` and `_work/data/netest-diversity`.

2. Prepare dataset and put them in the `_work/setup` directory
-  exLong && exlong sample (Table IV & V)
```bash
# exlong
inv -e data.setup-model-data --setup-name conditionnestack2e-with-name-ft
inv -e data.setup-model-data --setup-name conditionnestack2e-no-name-ft

# exlong sample
inv -e data.setup-model-data --setup-name conditionnestack2e-all-with-name-ft
inv -e data.setup-model-data --setup-name conditionnestack2e-all-no-name-ft
```
You should see `_work/setup/conditionnestack2e-with-name-ft/`, `_work/setup/conditionnestack2e-no-name-ft/`, `_work/setup/conditionnestack2e-all-with-name-ft/`, `_work/setup/conditionnestack2e-all-no-name-ft/` directories.

3. Construct prompts for exLong developer-view
- exLong
```bash
inv -e data.process-codellama-data --setup-name conditionnestack2e-with-name-ft
inv -e data.process-codellama-data --setup-name conditionnestack2e-no-name-ft
```

4. Construct prompts for exLong machine-view
```bash
mkdir _work/setup/conditionnestack2e-all-no-name-ft/eval/ -p
cp -r _work/data/rq2/ _work/setup/conditionnestack2e-all-no-name-ft/eval/
python -m etestgen.codellama.realDataProcessor --config_file configs/eval-codellama-7b-machine-view-conditionnestack2e-all-no-name.yaml process_test_data
```
You will see `_work/setup/conditionnestack2e-all-no-name-ft/eval/rq2/test-conditionnestack2e-all-no-name-ft.jsonl`.

# Experiments

[sec-exp]: #experiments

## Training

1. Training exLong w. EBT name

**Note**: conditionnestack2e is the setup name for exLong 
```bash
cd python/
accelerate launch -m axolotl.cli.train configs/axolotl/axolotl-conditionnestack2e-with-name-7b.yaml
```
You will see checkpoints in directory `_work/exp/conditionnestack2e-with-name-ft/lora-codellama-7b/`

2. Training exLong w.o. EBT name
```bash
cd python/
accelerate launch -m axolotl.cli.train configs/axolotl/axolotl-conditionnestack2e-no-name-7b.yaml
# script to run on TACC
sbatch axolotl-lora-codellama-7b-conditionnestack2e-no-name.sh
```
You will see checkpoints in directory `_work/exp/conditionnestack2e-no-name-ft/lora-codellama-7b/`

3. Running inference exLong for developer-view
```bash
cd python/
# Run evaluation on the selected 434 examples in the test set
python -m etestgen.codellama.CodeLLaMA --config_file configs/codellama-7b-conditionnestack2e-with-name-ft.yaml run_gen --split real-test
```
You will see checkpoints, model outputs in directory `_work/exp/conditionnestack2e-with-name-ft/lora-codellama-7b/real-test-set-model-outputs.jsonl`

4. Running inference exLong for machine-view
```bash
cd python/
python -m etestgen.codellama.CodeLLaMA --config_file configs/eval-codellama-7b-machine-view-conditionnestack2e-all-no-name.yaml run_gen
# Evaluation1: all covered projects
python -m etestgen.llm.eval --config_file configs/eval-codellama-7b-machine-view-conditionnestack2e-all-no-name.yaml eval_runtime_metrics
# You will see eval results in `results/model-results/conditionnestack2e-all-no-name-ft-lora-codellama-7b-eval-rq2-runtime-metrics.json`
# Evaluation2: intersection projects
python -m etestgen.llm.eval --eval_set rq2 --config_file configs/eval-codellama-7b-machine-view-conditionnestack2e-all-no-name.yaml eval_subset_llm_results --subset_id_file ../results/tool-results/intersect-ids.json
# You will see eval results in `results/model-results/conditionnestack2e-all-no-name-ft-lora-codellama-7b-eval-rq2-intersect-runtime-metrics.json`
```
You will see model generations in directory `_work/exp/conditionnestack2e-all-no-name-ft/lora-codellama-7b/rq2-model-outputs.jsonl`

## Evaluation: compute metrics

Given test cases generated by exLong, this step will evaluate them with metrics like BLEU, CodeBLEU, Test Coverage, etc.

- Input/Output info
    - Dataset used to evaluation is expected at `_work/{test_data}`
    - Processed LLM prediction is expected at `_work/exp/{setup}/{model_name}/test-results`
    - Similarity metrics result will be written to `_work/exp/{setup}/{model_name}/test-out/similarity_metrics_summary.json` and `results/model-results/{setup}-{exp}-{eval_set}-sim-metrics.json`
    - Runtime Metrics will be written to `results/model-results/{setup}-{exp}-{eval_set}-runtime-metrics.json` and individual result will be at `_work/exp/{setup}/{model_name}/test-results/metrics.jsonl`

- To run evaluation on similarity metrics
  - Run on an individual experiment

```bash
    python -m etestgen.llm.eval --eval_set test --config_file [/path/to/config/file] eval_llm_sim
    ```

- To run evaluation on runtime metrics
  - Run on an individual experiment

```bash
    python -m etestgen.llm.eval --eval_set test --config_file [/path/to/config/file] eval_runtime_metrics
    ```

## Ablations on exLong's context
#### Diversity of the nEBTs
1. Prepare Dataset
```bash
mkdir -p _work/setup/diversity-conditionnestack2e-sample-with-name-ft/real-eval/test/
mkdir -p _work/setup/diversity-conditionnestack2e-all-with-name-ft/real-eval/test/
cp -r _work/data/netest-diversity/* _work/setup/diversity-conditionnestack2e-sample-with-name-ft/real-eval/test/
cp -r _work/data/netest-diversity/* _work/setup/diversity-conditionnestack2e-all-with-name-ft/real-eval/test/
cd python/
python -m etestgen.codellama.DataProcessor --config_file configs/codellama-7b-diversity-conditionnestack2e-sample-with-name-ft.yaml process_real_test_data
python -m etestgen.codellama.DataProcessor --config_file configs/codellama-7b-diversity-conditionnestack2e-all-with-name-ft.yaml process_real_test_data
```
You will see processed data in `_work/setup/diversity-conditionnestack2e-all-with-name-ft/real-eval/test/` and `_work/setup/diversity-conditionnestack2e-sample-with-name-ft/real-eval/test/`.

2. Running Inference
```bash
# [2nd row] use the same exLong ckpt but try prompting with the same nEBT multiple times
python -m etestgen.codellama.CodeLLaMA --config_file configs/codellama-7b-diversity-conditionnestack2e-sample-with-name-ft.yaml  run_gen --split real-test --target_ckpt ../_work/exp/conditionnestack2e-with-name-ft/lora-codellama-7b/
# [3rd row] use the same exLong ckpt but try prompting with different nEBTs
python -m etestgen.codellama.CodeLLaMA --config_file configs/codellama-7b-diversity-conditionnestack2e-all-with-name-ft.yaml  run_gen --split real-test --target_ckpt ../_work/exp/conditionnestack2e-with-name-ft/lora-codellama-7b/
```
You will see model outputs in directory `_work/exp/diversity-conditionnestack2e-sample-with-name-ft/lora-codellama-7b/` and `_work/exp/diversity-conditionnestack2e-all-with-name-ft/lora-codellama-7b/`

#### exLong w.o. stack trace

1. Prepare Dataset
```bash
inv -e data.setup-model-data --setup-name conditionne2e-with-name-ft
inv -e data.process-codellama-data --setup-name conditionne2e-with-name-ft
```
2. Running Inference
```bash
python -m etestgen.codellama.CodeLLaMA --config_file python/configs/eval/codellama-7b-conditionne2e-with-name-ft.yaml run_gen
```

#### exLong w.o. stack trace & guard expression

1. Prepare Dataset
```bash
inv -e data.setup-model-data --setup-name ne2e-with-name-ft
inv -e data.process-codellama-data --setup-name ne2e-with-name-ft
```

2. Running Inference
```bash
python -m etestgen.codellama.CodeLLaMA --config_file configs/eval/codellama-7b-ne2e-with-name-ft.yaml run_gen
```

#### exLong w.o. stack trace & guard expression & nEBT

1. Prepare Dataset
```bash
inv -e data.setup-model-data --setup-name mut2e-with-name-ft
inv -e data.process-codellama-data --setup-name mut2e-with-name-ft
```

2. Running Inference
```bash
python -m etestgen.codellama.CodeLLaMA --config_file configs/eval/codellama-7b-mut2e-with-name-ft.yaml run_gen
```

# Artifacts:

[sec-artifacts]: #artifacts

### Model Checkpoints:
- [exLong-with-name (7B and 13B)](https://utexas.box.com/s/u20ya44oq8eon8aaot479iynpa90erog): exLong models in Table IV, Table VI and Table VIII.
- [exLong-no-name (7B)](https://utexas.box.com/s/9oo0fcbnhi8b6tggb273otjt5bzw8u0j): exLong models in Table V.
- [exLong-with-name w.o. stack trace (7B)](https://utexas.box.com/s/qikt46jxnf3g3pvqmznruf9bjd8yi17q): exLong no stack trace model in Table VI.
- [exLong-with-name w.o. stack trace & guard expr (7B)](https://utexas.box.com/shared/static/mxls9c8580igtbnbt2kw29lxl1nhdsv7.tar): exLong no stack trace & no guard expr model in Table VI.
- [exLong-with-name w.o. stack trace & guard expr & EBT (7B)](https://utexas.box.com/s/p7bcffw0vxelkrp5a2d70anzkekat2rv): exLong no stack trace & no guard expr & no EBT model in Table VI.
- [exLong-with-name w.o. stack trace & guard expr & EBT (13B)](https://utexas.box.com/s/uaunxdgzql5m6qqt8113ks288xmk0gh1): exLong 13B no stack trace & no guard expr & no EBT model in Table VIII.

### Dataset:
- [repos.tar.gz](https://utexas.box.com/s/5f9ogvbe3nnz2fijplu1zs2ohygmgsv2): The repository list from which we collected the dataset.
- [raw-data.tar.gz](https://utexas.box.com/shared/static/hfcp4za3j9vp8lh5u8iviadixuxu8080.gz): The raw collected data from the open-source repositories. `etestgen-raw-data-12k/`
- [ne2e-test.tar.gz](https://utexas.box.com/shared/static/4m7mntp0ix18dkl1ikkspcmpuvybfs1f.gz): The collected dataset for eval in **developer-view**. `rq1-eval/`
- [machine-view.tar.gz](https://utexas.box.com/shared/static/y4e52k5x8vk8vcr59lg33gebcg2m1caw.gz): The collected dataset for eval in **machine-view**. `rq2/`
- [netest-diversity.tar.gz](https://utexas.box.com/shared/static/j417e93j1rdvdqz2yobttygfhucfbkjm.gz): The collected dataset we use to study how the different nEBTs affect model's performance (Table VII). `netest-diversity/`
- [processed dataset](https://utexas.box.com/s/dwxneqvx1m1zw2t68tcugxmk1c7gitjm): The processed dataset (prompts) to train the exLong models.

# CLI
[sec-cli]: #cli

## User-View Use Case

In User View, Exlong generates an EBT for a user-specified target throw statement. Use the following command:

```bash
python -m etestgen.cli user_view [OPTIONS]
```

### Required Parameters

- `--repo_path`: Local path or remote link to the git repository
- `--mut_file_path`: Path to the file containing the MUT
- `--mut_line`: Line number of the beginning of the MUT's definition
- `--throw_file_path`: Path to the file containing the target throw statement
- `--throw_line`: Line number of the target throw statement
- `--test_context_path`: Path to the test file

### Optional Parameters

- `--sha`: Commit SHA (default: latest commit on the main branch)
- `--test_name`: Name of the test method to be generated (default: none)
- `--quant`: Whether to use quantized LLM (default: true)
- `--pick_best`: Whether to sample multiple candidate EBTs and select the best test based on runtime evaluation (default: false)
- `--output_file`: Output file path for the generated EBT, if not given the EBTs are added to the test object in `test_context_path`.
- `--regenerate_data`: Whether to run collect "stack trace", "guard condition", and etc. (default: true)

### Example

```bash
python -m etestgen.cli user_view \
    --repo_path=./Wisp \
    --mut_file_path=Scheduler.java \
    --mut_line=180 \
    --quant=true \
    --throw_file_path=Scheduler.java \
    --throw_line=340 \
    --test_context_path=SchedulerTest.java \
    --sha="ce1d9f3cb1944115ad98b4428ea24b24ab3faf56" \
    --test_name=testSchedulerError \
    --pick_best=True \
    --output_file=./ExlongTest.java
```

This command will generate an exception-based test for the throw statement at line 340 in `Scheduler.java`, targeting the method that begins at line 180, and output the result to `ExlongTest.java`.

## Machine View

The Machine View generates EBTs for the entire codebase to cover all throw statements automatically. Use the following command:

```bash
python -m etestgen.cli machine_view [OPTIONS]
```

#### Required Parameters

- `--repo_path` or `--repo_link`: Local path or remote link to the git repository
- `--test_context_path`: Path to the test file

#### Optional Parameters

- `--sha`: Commit SHA (default: latest commit on the main branch)
- `--pick_best`: Whether to sample multiple candidate EBTs for each throw statement and select the best test based on runtime evaluation (default: false)
- `--quant`: Whether to use quantized LLM (default: true)
- `--timeout`: Time budget for the tool to finish processing in seconds (default: infinity)
- `--output_file`: Output file path for the generated EBTs, if not given the EBTs are added to the test object in `test_context_path`.
- `--regenerate_data`: Whether to run collect "stack trace", "guard condition", and etc (default: true)
#### Example

```bash
python -m etestgen.cli machine_view \
    --repo_link="https://github.com/Coreoz/Wisp.git" \
    --sha="ce1d9f3cb1944115ad98b4428ea24b24ab3faf56" \
    --timeout=1000
```

This command will analyze the entire Wisp repository at the specified commit and generate EBTs for all throw statements within the given time budget of 1000 seconds.

exLong

exLong 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

exLong

exLong 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README