CodeJudge-Eval

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 12 Open GitHub ↗

[COLING25] CodeJudge Eval: Can Large Language Models be Good Judges in Code Understanding?

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h3 align="center"><a href="https://arxiv.org/abs/2408.10718" style="color:#9C276A">
CodeJudge-Eval:  Can Large Language Models be Good Judges in Code Understanding?</a></h3>
<h5 align="center"> 🎉 Our paper has been accepted by the 31st International Conference on Computational Linguistics (<a href="https://coling2025.org/">COLING 25</a>). </h2>
<h5 align="center"> If our project helps you, please give us a star ⭐ on GitHub to support us. 🙏🙏 </h2>

[![hf_data](https://img.shields.io/badge/🤗-Datasets-9C276A.svg)](https://huggingface.co/datasets/CodeResearch/CodeJudge-Eval)
[![arXiv](https://img.shields.io/badge/Arxiv-2408.10718-AD1C18.svg?logo=arXiv)](https://arxiv.org/abs/2408.10718)
[![License](https://img.shields.io/badge/License-MIT-yellow)](https://github.com/CodeLLM-Research/CodeJudge-Eval/LICENSE.txt)

</h5>

## Introduction

Recent advancements in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks. However, these benchmarks may not fully capture a model's code understanding abilities. We introduce **CodeJudge-Eval (CJ-Eval)**, a novel benchmark designed to assess LLMs' code understanding abilities from the perspective of code judging rather than code generation. **CJ-Eval** challenges models to determine the correctness of provided code solutions, encompassing various error types and compilation issues. By leveraging a diverse set of problems and a fine-grained judging system, **CJ-Eval** addresses the limitations of traditional benchmarks, including the potential memorization of solutions. Evaluation of 12 well-known LLMs on **CJ-Eval** reveals that even state-of-the-art models struggle, highlighting the benchmark's ability to probe deeper into models' code understanding abilities.

## Experiment Results

## More Details

More details can be found in our [paper](https://arxiv.org/abs/2408.10718).

## 📑 Citation

If you find **CodeJudge-Eval** useful for your research and applications, please cite using this BibTeX:
```bibtex
@misc{zhao2024codejudgeevallargelanguagemodels,
      title={CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?}, 
      author={Yuwei Zhao and Ziyang Luo and Yuchen Tian and Hongzhan Lin and Weixiang Yan and Annan Li and Jing Ma},
      year={2024},
      eprint={2408.10718},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2408.10718}, 
}
```

CodeJudge-Eval

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README