Quatrain

KO | EN

gitlite — search

#javascript #hacktoberfest #ai #llm #android #deep-learning #linux #nodejs #awesome #cli #awesome-list #security #mcp #open-source

(★ 9)

Quatrain(Question Answering for Patch Correctness Evaluation), a supervised learning approach that exploits a deep NLP model to classify the relatedness of a bug report with a patch description.

Quatrain Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![DOI](https://zenodo.org/badge/519884433.svg)](https://zenodo.org/badge/latestdoi/519884433)

# Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness
```bibtex
@inproceedings{tian2022change,
  title={Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness},
  author={Tian, Haoye and Tang, Xunzhu and Habib, Andrew and Wang, Shangwen and Liu, Kui and Xia, Xin and Klein, Jacques and Bissyand{\'E}, Tegawend{\'E} F},
  booktitle={37th IEEE/ACM International Conference on Automated Software Engineering},
  pages={1--13},
  url = {https://doi.org/10.1145/3551349.3556914}, 
  doi = {10.1145/3551349.3556914},
  year={2022}
}
```
Paper Link: https://dl.acm.org/doi/abs/10.1145/3551349.3556914

Quatrain
=======
Quatrain (Question Answering for Patch Correctness Evaluation), a supervised learning approach that exploits a deep NLP model to classify the
relatedness of a bug report with a patch description.

### Catalogue of Repository
```
artifact_detection_model: a model to detect codes in text.
data: processed and structured dataset.
experiment: scripts to obtain experimental results of paper. 
figure: saved figures for experiment
preprocess: scripts to extract bug reports and commit messages.
representation: embeddings representation model.
utils: scripts to deduplicate dataset.
---------------
INSTALL.md: installation instructions.
quatrain_model.h5: pre-trained QUATRAIN model for users' custom prediction.
requirements.txt: required dependencies.
run.py: entrance to conduct experiment.
```
## Ⅰ) Dataset
  1. bug report summary: title for bug issue.
  2. bug report description: detailed description for bug issue.
  3. patch description: CodeTrans-generated commit message for patch.

### A) Table 1: Datasets of labelled patches.
* [data/bugreport_patch.txt](https://github.com/Trustworthy-Software/Quatrain/blob/main/data/bugreport_patch.txt): 9135 (1591:7544) Pairs of Bug report & Commit message. Structured as `bug-id $$ bug report summary $$ bug report description $$ patchId $$ patch description $$ label`
* [data/bugreport_patch_json_bert.pickle](https://github.com/Trustworthy-Software/Quatrain/blob/main/data/bugreport_patch_json_bert.pickle): Bert embeddings of Pairs of Bug report & Commit message.
* data/bugreport_patch_array_bert.pickle: Bert embeddings of paris for 10-fold cross validation.

### B) Colleted elements
```
data/BugReport: Bug reports texts for Defects4j, Bugsjar, Bears. Structured as `bug-id $$ bug report summary $$ bug report description` in txt file.
data/CommitMessage: Commit messages written by developer or generated by CodeTrans in format of json and pickle. Structured as `bug-id: commit message` in json file.
---------------
BATS_RESULT_0.0.json: the prediction results of BATS with cut-off 0.0 on our dataset. 
BATS_RESULT_0.8.json: the prediction results of BATS with cut-off 0.8 on our dataset.
PATCHSIM_RESULT.json: the prediction results of Patch-Sim on our dataset.
PatchLabelsYe.csv: the original prediction results of ODS.
Bears_testinfo.txt: the stack failure information of test suites for Bears.   
bears_index_dict(inverse).json: dictionary of bug-id and commit-id. 
save_bugreport_patch.py: script to produce data/bugreport_patch.txt.
```

## Ⅱ) Requirements
### A) Environment 
  * python 3.7 (Anaconda recommended)
  * pip install -r requirements.txt

run `sudo apt-get install python3.7-dev` first if you don't have python3.7 dev package.

### B) Data elements 
  download _ASE2022withTextUnique.zip_ (need to be unzipped) and _ASE_features2_bert.pickle_ from [data in Zenodo](https://zenodo.org/record/7029808#.YwuKF2QzZb8 "Dataset for Quatrain"), 
  accordingly change the absolute path of these two files in **experiment/config.py** of this repository as below.
  1. self.path_patch ---> ASE2022withTextUnique.  Original dataset with patches text and commit messages text.
  2. self.path_ASE2020_feature ---> ASE_features2_bert.pickle. The feature from Tian et al.'s ASE2020 [paper](https://ieeexplore.ieee.org/abstract/document/9286101) for our RQ3 DL experiment. 
  
Simplified dataset: [ASE2022withText.](https://zenodo.org/record/7315302#.Y293X2RKhhE)

## Ⅲ) Experiment

To obtain the experimental results of our paper, execute `run.py` with the following parameters:

### A) Sec. 2.2 (Hypothesis validation)
  1. **Figure 2:** Distributions of Euclidean distances between bug and patch descriptions.
```
python run.py hypothesis
```

### B) Sec. 5.1 (RQ1: Effectiveness of Quatrain) 
  1. **Figure 5:** Distribution of Patches in Train and Test Data. 
  2. **Table 2:** Confusion matrix of Quatrain prediction.
```
python run.py RQ1
```
  3. **The improved F1:** a better F1 score of 0.793 by re-balancing the test data.
```
python run.py RQ1 balance
```

### C) Sec. 5.2 (RQ2: Analysis of the Impact of Input Quality on Quatrain)
#### RQ 2.1
  1. **Figure 6:**  Impact of length of patch description to prediction.
```
python run.py RQ2.1
```
#### RQ 2.2
  2. **Figure 7:**  The distribution of probability of patch correctness
on original and random bug report.
  3. **The dropped +Recall:**  22% (241/1073) of developer patches, which were previously predicted as correct, are no longer recalled by Quatrain after they have been associated to a random bug report.
```
python run.py RQ2.2
```
#### RQ 2.3
  4. **Figure 8:**   Impact of distance between generated patch descrip-
tion to ground truth on prediction performance.
  5. **The dropped +Recall:**  The metric (+Recall) drops by 37 percentage points to 45\% when the developer-written descriptions are replaced with CodeTrans-generated descriptions.
```
python run.py RQ2.3
```
  6. **The dropped AUC:**  we evaluated Quatrain in a setting where all developer commit messages were replaced with CodeTrans-generated descriptions: the AUC metric dropped by 11 percentage points to 0.774, confirming our findings.
```
python run.py RQ2.3 CodeTrans
```

### D) Sec. 5.3 (RQ3: Comparison Against the State of the Art)
#### Sec. 5.3.1 (Comparing against Static Approaches)
  1. **Table 3:** Quatrain vs a DL-based patch classifie.
  2. **New identification:**  Among 9135 patches, our approach identifies 7842 patches, of which 2735 patches cannot be identified by Tian et al.'s approach (RF).
```
python run.py RQ3 DL
```
  3. **Table 4:** Quatrain vs BATS.
  4. **New identification:**  180 out of 345 patches are exclusively identified by Quatrain.
```
python run.py RQ3 BATS
```

#### Sec. 5.3.2 (Comparing against Dynamic Approach)
  1. **Table 5:** Quatrain vs (execution-based) PATCH-SIM.
  2. **New identification:**  Most of the patches (1856/3149) that we identify are not correctly predicted by PATCH-SIM.
```
python run.py RQ3 PATCHSIM
```

### E) Sec. 6.1 (Experimental insights)
  1. **RF with 10-fold:** RandomForest (RF) on the embeddings of the bug report and the patch based on 10-fold cross validation.
  2. **RF with 10-group:** RandomForest (RF) on the embeddings of the bug report and the patch based on 10-group cross validation.
```
python run.py insights
```

## Ⅳ) Custom Prediction
To predict the correctness of your custom patches, you are welcome to use the prediction interface.
### A) Requirements for BERT
  * **BERT model client&server:** 24-layer, 1024-hidden, 16-heads, 340M parameters. download it [here](https://storage.googleapis.com/bert_models/2019_05_30/wwm_cased_L-24_H-1024_A-16.zip).
  * **Environment for BERT server** (different from reproduction)
    * python 3.7 
    * pip install tensorflow==1.14
    * pip install bert-serving-client==1.10.0
    * pip install bert-serving-server==1.10.0
    * pip install protobuf==3.20.1
    * Launch BERT server via `bert-serving-start -model_dir "Path2BertModel"/wwm_cased_L-24_H-1024_A-16 -num_worker=2 -max_seq_len=360 -port 8190`
    * switch the port in [BERT_Port](https://github.com/Trustworthy-Software/Quatrain/blob/main/representation/word2vec.py#L42) in case your port 8190 is occupied.
  * **Bug report text:** developer-written bug report.
  * **Patch description text:** generating patch description for your plausible patches with commit message generation tools, e.g. CodeTrans. [Github](https://github.com/agemagician/CodeTrans) and [API](https://huggingface.co/SEBIS/code_trans_t5_large_commit_generation_transfer_learning_finetune).

### B) Predict
Let's give it a try!
```
python run.py predict $bug_report_text $patch_description_text
```
For instance: `python run.py predict 'Missing type-checks for var_args notation' 'check var_args properly'`

## Ⅴ) Custom Train
To re-train QUATRAIN model on our or other dataset, execute the following steps.
  1. Structure your dataset as [data/bugreport_patch.txt](https://github.com/Trustworthy-Software/Quatrain/blob/main/data/bugreport_patch.txt). 
  2. Obtain Bert embeddings of your dataset via `experiment/save_bugreport_dataset_json.py`
  3. Accordingly, change self.dataset_json in experiment/config.py 
  4. Execute `python run.py RQ1`

## License
Quatrain is distributed under the terms of the MIT License, see [LICENSE](LICENSE).

[//]: # (### deduplicate.py)
[//]: # (deduplicating same patches.)

Quatrain

Quatrain Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Quatrain

Quatrain Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README