awesome-lm-evaluation

KO | EN

gitlite — search

#javascript #hacktoberfest #ai #typescript #golang #rust #linux #macos #java #windows #kubernetes #vue #pytorch #open-source

awesome-lm-evaluation

(★ 50)

🩺 A collection of ChatGPT evaluation reports on various bechmarks.

awesome-lm-evaluation 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🩺 Awesome Language Model Evaluations

👋 Hi, there. This is a collection of papers for Language Model evaluations, especially for the closed-source ChatGPT.

📫 Suggestions, PRs and all contributions are welcome. 
We really appreciate the lovely people who help us build a friendly community. 
Many thanks to the organizations and companies that provide amazing tools, model weights and inspiring ideas~

## 🛡️ Badges

- Model: ![ChatGPT](https://img.shields.io/badge/ChatGPT-black)
- Machine Translation: ![Translation](https://img.shields.io/badge/MT-orange)
- Natural Language Generation: ![Generation](https://img.shields.io/badge/NLG-green)
- Natural Language Understanding: ![Natural Language Understanding](https://img.shields.io/badge/NLU-red)
- Information Extraction: ![Information Extraction](https://img.shields.io/badge/IE-blue)
- Grammar Error Correction: ![Grammar Error Correction](https://img.shields.io/badge/GEC-b31b1b)

## 📑 Papers

[arXiv 2023] [**How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks**](https://arxiv.org/abs/2303.00293) 
_Xuanting Chen, Junjie Ye, Can Zu, Nuo Xu, Rui Zheng, Minlong Peng, Jie Zhou, Tao Gui, Qi Zhang, Xuanjing Huang_ 
![ChatGPT](https://img.shields.io/badge/ChatGPT-black) ![Natural Language Understanding](https://img.shields.io/badge/NLU-red)
> The GPT-3.5 models have demonstrated impressive performance in various Natural Language Processing (NLP) tasks, showcasing their strong understanding and reasoning capabilities. However, their robustness and abilities to handle various complexities of the open world have yet to be explored, which is especially crucial in assessing the stability of models and is a key aspect of trustworthy AI. In this study, we perform a comprehensive experimental analysis of GPT-3.5, exploring its robustness using 21 datasets (about 116K test samples) with 66 text transformations from TextFlint that cover 9 popular Natural Language Understanding (NLU) tasks. Our findings indicate that while GPT-3.5 outperforms existing fine-tuned models on some tasks, it still encounters significant robustness degradation, such as its average performance dropping by up to 35.74\% and 43.59\% in natural language inference and sentiment analysis tasks, respectively. We also show that GPT-3.5 faces some specific robustness challenges, including robustness instability, prompt sensitivity, and number sensitivity. These insights are valuable for understanding its limitations and guiding future research in addressing these challenges to enhance GPT-3.5's overall performance and generalization abilities.

[arXiv 2023] [**A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity**](https://arxiv.org/abs/2302.04023) 
_Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, Pascale Fung_ 
![ChatGPT](https://img.shields.io/badge/ChatGPT-black) ![Natural Language Understanding](https://img.shields.io/badge/NLU-red) ![Generation](https://img.shields.io/badge/NLG-green) ![Translation](https://img.shields.io/badge/MT-orange)

> This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an intermediate code generation step. Moreover, we find that ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning, hence making it an unreliable reasoner. It is, for example, better at deductive than inductive reasoning. ChatGPT suffers from hallucination problems like other LLMs and it generates more extrinsic hallucinations from its parametric memory as it does not have access to an external knowledge base. Finally, the interactive feature of ChatGPT enables human collaboration with the underlying LLM to improve its performance, i.e, 8% ROUGE-1 on summarization and 2% ChrF++ on machine translation, in a multi-turn "prompt engineering" fashion. We also release codebase for evaluation set extraction.

[arXiv 2023] [**Zero-Shot Information Extraction via Chatting with ChatGPT**](https://arxiv.org/abs/2302.10205) 
_Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, Wenjuan Han_ 
![ChatGPT](https://img.shields.io/badge/ChatGPT-black) ![Information Extraction](https://img.shields.io/badge/IE-blue)
> Zero-shot information extraction (IE) aims to build IE systems from the unannotated text. It is challenging due to involving little human intervention. Challenging but worthwhile, zero-shot IE reduces the time and effort that data labeling takes. Recent efforts on large language models (LLMs, e.g., GPT-3, ChatGPT) show promising performance on zero-shot settings, thus inspiring us to explore prompt-based methods. In this work, we ask whether strong IE models can be constructed by directly prompting LLMs. Specifically, we transform the zero-shot IE task into a multi-turn question-answering problem with a two-stage framework (ChatIE). With the power of ChatGPT, we extensively evaluate our framework on three IE tasks: entity-relation triple extract, named entity recognition, and event extraction. Empirical results on six datasets across two languages show that ChatIE achieves impressive performance and even surpasses some full-shot models on several datasets (e.g., NYT11-HRL). We believe that our work could shed light on building IE models with limited resources.

[arXiv 2023] [**Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT**](https://arxiv.org/abs/2302.10198) 
_Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, Dacheng Tao_ 
![ChatGPT](https://img.shields.io/badge/ChatGPT-black) ![Information Extraction](https://img.shields.io/badge/IE-blue) ![Natural Language Understanding](https://img.shields.io/badge/NLU-red)

> Recently, ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. Several prior studies have shown that ChatGPT attains remarkable generation ability compared with existing models. However, the quantitative analysis of ChatGPT’s understanding ability has been given little attention. In this report, we explore the understanding ability of ChatGPT by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models. We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question-answering tasks. Additionally, by combining some advanced prompting strategies, we show that the understanding ability of ChatGPT can be further improved. Scripts and data: [this https URL](https://github.com/WHU-ZQH/ChatGPT-vs.-BERT)

[arXiv 2023] [**Is ChatGPT A Good Translator? A Preliminary Study**](https://arxiv.org/abs/2301.08745v2) 
_Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Zhaopeng Tu_ 
![ChatGPT](https://img.shields.io/badge/ChatGPT-black) ![Translation](https://img.shields.io/badge/MT-orange)

> This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well and show minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. For distant languages, we explore an interesting strategy named pivot prompting that asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, which improves the translation performance significantly. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but is potentially a good translator for spoken language. Scripts and data: [this https URL](https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator)

[arXiv 2023] [**Is ChatGPT a General-Purpose Natural Language Processing Task Solver?**](https://arxiv.org/abs/2302.06476) 
_Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, Diyi Yang_ 
![ChatGPT](https://img.shields.io/badge/ChatGPT-black) ![Natural Language Understanding](https://img.shields.io/badge/NLU-red)![Generation](https://img.shields.io/badge/NLG-green)

> Spurred by advancements in scale, large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot -- i.e., without adaptation on downstream data. Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community due to the fact that it can generate high-quality responses to human input and self-correct previous mistakes based on subsequent conversations. However, it is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot. In this work, we empirically analyze the zero-shot learning ability of ChatGPT by evaluating it on 20 popular NLP datasets covering 7 representative task categories. With extensive empirical studies, we demonstrate both the effectiveness and limitations of the current version of ChatGPT. We find that ChatGPT performs well on many tasks favoring reasoning capabilities (e.g., arithmetic reasoning) while it still faces challenges when solving specific tasks such as sequence tagging. We additionally provide in-depth analysis through qualitative case studies.

[arXiv 2023] [**Exploring the Feasibility of ChatGPT for Event Extraction**](https://arxiv.org/abs/2303.03836) 
_Jun Gao, Huan Zhao, Changlong Yu, Ruifeng Xu_ 
![ChatGPT](https://img.shields.io/badge/ChatGPT-black) ![Information Extraction](https://img.shields.io/badge/IE-blue)

> Event extraction is a fundamental task in natural language processing that involves identifying and extracting information about events mentioned in text. However, it is a challenging task due to the lack of annotated data, which is expensive and time-consuming to obtain. The emergence of large language models (LLMs) such as ChatGPT provides an opportunity to solve language tasks with simple prompts without the need for task-specific datasets and fine-tuning. While ChatGPT has demonstrated impressive results in tasks like machine translation, text summarization, and question answering, it presents challenges when used for complex tasks like event extraction. Unlike other tasks, event extraction requires the model to be provided with a complex set of instructions defining all event types and their schemas. To explore the feasibility of ChatGPT for event extraction and the challenges it poses, we conducted a series of experiments. Our results show that ChatGPT has, on average, only 51.04% of the performance of a task-specific model such as EEQA in long-tail and complex scenarios. Our usability testing experiments indicate that ChatGPT is not robust enough, and continuous refinement of the prompt does not lead to stable performance improvements, which can result in a poor user experience. Besides, ChatGPT is highly sensitive to different prompt styles.

[arXiv 2023] [**ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark**](https://arxiv.org/abs/2303.13648) 
_Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, Michael Lyu_ 
![ChatGPT](https://img.shields.io/badge/ChatGPT-black) ![Grammar Error Correction](https://img.shields.io/badge/GEC-b31b1b)

> ChatGPT is a cutting-edge artificial intelligence language model developed by OpenAI, which has attracted a lot of attention due to its surprisingly strong ability in answering follow-up questions. In this report, we aim to evaluate ChatGPT on the Grammatical Error Correction(GEC) task, and compare it with commercial GEC product (e.g., Grammarly) and state-of-the-art models (e.g., GECToR). By testing on the CoNLL2014 benchmark dataset, we find that ChatGPT performs not as well as those baselines in terms of the automatic evaluation metrics (e.g., F0.5 score), particularly on long sentences. We inspect the outputs and find that ChatGPT goes beyond one-by-one corrections. Specifically, it prefers to change the surface expression of certain phrases or sentence structure while maintaining grammatical correctness. Human evaluation quantitatively confirms this and suggests that ChatGPT produces less under-correction or mis-correction issues but more over-corrections. These results demonstrate that ChatGPT is severely under-estimated by the automatic evaluation metrics and could be a promising tool for GEC.

## 🔑 License

<a rel="license" href="http://creativecommons.org/licenses/by-nc/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc/4.0/88x31.png" /></a> This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc/4.0/">Creative Commons Attribution-NonCommercial 4.0 International License</a>.

awesome-lm-evaluation

awesome-lm-evaluation 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

awesome-lm-evaluation

awesome-lm-evaluation 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README