leak-llm.github.io

KO | EN

gitlite — search

#python #hacktoberfest #go #android #machine-learning #rust #linux #docker #awesome #cli #kubernetes #awesome-list #vue #mcp

leak-llm.github.io

(★ 9)

A collaborative effort to track indirect data leaks to large language models.

leak-llm.github.io 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
Natural Language Processing (NLP) research is becoming increasingly focused on the use of Large Language Models (LLMs), with some of the most popular ones being either fully or partially closed-source. The lack of access to model details, especially regarding training data, has repeatedly raised concerns about data contamination among researchers. Several attempts have been made to address this issue, but they are limited to anecdotal evidence and trial and error. Additionally, they overlook the problem of indirect data leaking, where models are iteratively improved by using data coming from users. In this work, we conduct the first systematic review of work using OpenAI's ChatGPT and GPT-4, the most prominently used LLMs today, in the context of data contamination. By analysing 255 papers and considering OpenAI's data usage policy, we extensively document how much data has been leaked to ChatGPT in the first year after the model's release. At the same time, we document a number of evaluation malpractices emerging in the reviewed papers, including unfair or missing baseline comparisons, reproducibility issues, and authors' lack of awareness of the data usage policy. Our work provides the first quantification of the ChatGPT data leakage problem.

## Cite us
```
@inproceedings{balloccu-etal-2024-leak,
    title = "Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs",
    author = "Balloccu, Simone and  Schmidtová, Patrícia and Lango, Mateusz and Dušek, Ondřej",
    booktitle = "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics",
    year = "2024",
    publisher = "Association for Computational Linguistics",
}
```

## How to contribute?
If you would like to add your own work that has unfortunately leaked some data, or correct information that is already in a table, simply submit a pull request to modify this page directly at `https://github.com/leak-llm/leak-llm.github.io`.

## Results

We release our survey results as a table below. The table has the following structure (columns):

* `Title`: for each paper that we analysed
* `Applicable`: **Y** if the paper contains an actual evaluation of ChatGPT or GPT-4; **N/A** otherwise, in which case all the other columns have **N/A** or empty value.
* `Date (earliest draft)`: the date of the earliest version found online for the paper.
* `Running period`: containing the date in format **Month Year** (e.g. **December 2023**) if it was possible to derive when the experiment was ran; **-** otherwise.
* `Prompts available`: **Y** for those work that detail the used prompts; **N** otherwise.
* `Repository available`: **Y** for those work that publicly release experiment code in a repository detail; **N** otherwise.
* `Model`: used model for the experiment (**ChatGPT**; **GPT-4**; **Both**).
* `Model access`: if models were accessed through browser (**BROWSER**) or API (**API - POST MARCH 2023 OR OPT-OUT**).
* `Sampling etc.`: **Y** if the paper makes use of random sampling or any other practice such that it is not possible to exactly reconstrute the used data; **N** otherwise.
* `Comparison `: **Y** if the paper compares the considered model(s) with other baselines/approaches; **N** otherwise.
* `Different evaluation scale`: **Y** if the comparison scale between the considered model(s) and other baselines/approaches differs; **N** otherwise.
* `Custom datasets`: **Y** if the work adopts an ad-hoc/custom dataset that is not publicly available.
* `Datasets leaked`: Details of the leaked data, in the format "Name of dataset; Task name; Split (train/test/...); Number of instances; Fraction of the whole split".

| **Title** | **Applicable** | **Published** | **Date (earliest draft)** | **Running period** | **Prompts available** | **Repository available** | **Model** | **Model access** | **Sampling etc.** | **Comparison** | **Different evaluation scale** | **Custom datasets** | **Datasets leaked** |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | ------------- | ------------------------- | ------------------------------ | --------------------- | ------------------------ | --------- | -------------------------------- | ----------------- | -------------- | ------------------------------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **[LLM-contamination](https://hitz-zentroa.github.io/lm-contamination/)** | **N/A** | | \- | \- | N/A | N/A | \- | | N/A | N/A | N/A | | |
| **[Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://arxiv.org/abs/2308.08493)** | **Y** | Y | 16/08/2023 | June 2023 | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | Y | N | N | | |
| **[Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://arxiv.org/abs/2004.10964)** | **N/A** | | \- | \- | N/A | N/A | \- | | \- | \- | | | |
| **[Can we trust the evaluation on ChatGPT?](https://arxiv.org/abs/2303.12767)** | **Y** | Y | 22/03/2023 | January and February 2023 | Y | N | ChatGPT | BROWSER | Y | N | N | N |[SemEval-2016 Task 6](https://afshinrahimi.github.io/semeval2016-task6/) (68.8% of test set) [P-Stance](https://github.com/chuchun8/PStance) (99.1% of test set)|
| **[How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://arxiv.org/pdf/2301.07597.pdf?trk=public_post-text)** | **Y** | N | 18/01/2023 | \- | Y | Y | ChatGPT | BROWSER | Y | N | N | N |[HC3-English](https://github.com/Hello-SimpleAI/chatgpt-comparison-detection) (100.0% of entire set) [ELI5](https://facebookresearch.github.io/ELI5/) (19.0% of entire set) [WikiQA](https://www.microsoft.com/en-us/download/details.aspx?id=52419) (39.0% of entire set) wiki_csai (100.0% of entire set) [Medical Dialog](https://github.com/UCSD-AI4H/Medical-Dialogue-System) (0.3% of entire set) FiQA (59.1% of entire set) [HC3-Chinese](https://github.com/Hello-SimpleAI/chatgpt-comparison-detection) (100.0% of entire set) [WebTextQA + BaikeQA](https://github.com/brightmart/nlp_chinese_corpus) (0.1% of entire set) [BaiduBaike15](https://github.com/facebookarchive/bAbI-tasks/tree/master) (100.0% of entire set) [NLPCC-DBQA](http://tcci.ccf.org.cn/conference/2016/pages/page05_evadata.html) (7.0% of entire set) [Medical Dialog - zh](https://github.com/UCSD-AI4H/Medical-Dialogue-System) (0.1% of entire set) [FinanceZhidao](https://github.com/SophonPlus/ChineseNlpCorpus) (0.2% of entire set) [Chinese Psychological Question Answering Dataset](https://aistudio.baidu.com/datasetdetail/38489) (5.0% of entire set) [LegalQA](https://github.com/siatnlp/LegalQA) (2.3% of entire set)|
| **[GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://arxiv.org/abs/2308.12488)** | **N/A** | | 24/08/2023 | \- | N/A | N/A | Both | \- | \- | \- | | | |
| **[A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://arxiv.org/abs/2305.18486)** | **Y** | Y | 29/05/2023 | December 2022 | Y | Y | ChatGPT | BROWSER | Y | Y | Y | N |[EfficientQA](https://efficientqa.github.io/) (100.0% of dev set) custom-ethical (100.0% of entire set) [CB](https://super.gluebenchmark.com/) (100.0% of dev set) [RTE](https://paperswithcode.com/dataset/rte) (100.0% of dev set) [WiC](https://pilehvar.github.io/wic/) (100.0% of dev set) [AX-g](https://super.gluebenchmark.com/) (100.0% of dev set) [Inverse scaling challenge (11 datasets)](https://github.com/inverse-scaling/prize) (0.1% of entire set)|
| **[Quantifying Memorization Across Neural Language Models](https://arxiv.org/abs/2202.07646)** | **N/A** | | \- | \- | N/A | N/A | \- | | \- | \- | | | |
| **[Preventing Generation of Verbatim Memorization in Language Models Gives a False Sense of Privacy](https://arxiv.org/abs/2210.17546)** | **N/A** | | \- | \- | N/A | N/A | \- | | \- | \- | | | |
| **[List of various ChatGPT evaluations](https://github.com/THU-KEG/EvaluationPapers4ChatGPT#22-ethics-and-bias)** | **N/A** | | \- | \- | N/A | N/A | ChatGPT | | \- | \- | | | |
| **[GPT-4 Technical Report](https://arxiv.org/abs/2303.08774)** | **N/A** | | \- | \- | N/A | N/A | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | \- | \- | | | |
| **[Zeno Chatbot Report](https://github.com/zeno-ml/zeno-build/tree/main/examples/chatbot/report)** | **Y** | N | 18/05/2023 | \- | N | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://arxiv.org/abs/2302.04023)** | **Y** | N | 08/02/2023 | \- | Y | Y | ChatGPT | BROWSER | Y | Y | Y | N |[SAMSum](https://paperswithcode.com/dataset/samsum-corpus) (6.0% of test set) [CNN/DM](https://huggingface.co/datasets/cnn_dailymail) (0.4% of test set) [FLORES-200](https://github.com/facebookresearch/flores/tree/main/flores200) (53.0% of devtest set) [NusaX](https://github.com/IndoNLP/nusax) (25.0% of test set) [EntailmentBank](https://allenai.org/data/entailmentbank) (9.0% of test set) [bAbI-task15](https://github.com/facebookarchive/bAbI-tasks/tree/master) (12.0% of test set) [bAbI-task16](https://github.com/facebookarchive/bAbI-tasks/tree/master) (3.0% of test set) [CLUTRR](https://github.com/facebookresearch/clutrr) (1.0% of test set) αNLI (1.9% of dev set) [TimeDial](https://github.com/google-research-datasets/TimeDial) (2.0% of test set) [SpartQA](https://github.com/HLR/SpartQA_generation) (1.7% of test set) [StepGame](https://github.com/ZhengxiangShi/StepGame) (<0.1% of test set) Math (0.1% of test set) [CommonsenseQA](https://www.tau-nlp.sites.tau.ac.il/commonsenseqa) (2.6% of test set) [PiQA](https://github.com/ybisk/ybisk.github.io/tree/master/piqa) (1.6% of test set) [Pep-3k](https://github.com/ianporada/modeling_event_plausibility) (0.9% of entire set) [E-Care](https://github.com/Waste-Wood/e-CARE/) (1.4% of val set) [HotpotQA](https://hotpotqa.github.io/) (<0.1% of val set) [Letter string analogies - Webb](https://github.com/taylorwwebb/emergent_analogies_LLM/tree/main/letter_string) (1.6% of entire set) [COVID-19-Lee](https://github.com/HLTCHKUST/Perplexity-FactChecking/tree/main) (19.5% of test set) [OpenDialKG](https://github.com/facebookresearch/opendialkg) (0.3% of test set) [MultiWOZ 2.2](https://huggingface.co/datasets/multi_woz_v22) (5.0% of test set)|
| **[Don’t Stop Pretraining? Make Promp-based Fine-tuning Powerful Learner](https://arxiv.org/pdf/2305.01711.pdf)** | **N/A** | | \- | \- | N/A | N/A | \- | | \- | \- | | | |
| **[How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://arxiv.org/pdf/2302.09210.pdf)** | **Y** | N | 18/02/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://arxiv.org/abs/2301.08745)** | **Y** | N | 20/01/2023 | December 2022 | Y | Y | ChatGPT | BROWSER | Y | Y | N | N |[Flores-101](https://github.com/facebookresearch/flores/tree/main/previous_releases/flores101) (4.9% of test set) [WMT19 Biomedical Translation Task](https://github.com/biomedical-translation-corpora/corpora) (13.4% of test set) [WMT20 Robustness Task set 2 En->Ja](https://paperswithcode.com/dataset/wmt-2020) (3.6% of test set) [WMT20 Robustness Task set 2 Ja->En](https://paperswithcode.com/dataset/wmt-2020) (5.0% of test set) [WMT20 Robustness Task set 3](https://paperswithcode.com/dataset/wmt-2020) (0.9% of test set)|
| **[IS CHATGPT A GENERAL-PURPOSE NATURAL LANGUAGE PROCESSING TASK SOLVER?](https://arxiv.org/abs/2302.06476)** | **Y** | N | 08/02/2023 | January 2023 | Y | N | ChatGPT | BROWSER | Y | Y | Y | N |[MultiArith](https://huggingface.co/datasets/ChilleD/MultiArith) (100.0% of entire set) [GSM8K](https://github.com/openai/grade-school-math) (100.0% of test set) [AddSub](https://www.cs.washington.edu/nlp/arithmetic) (100.0% of entire set) [AQUA-RAT](https://huggingface.co/datasets/aqua_rat) (100.0% of test set) [SingleEq](https://gitlab.cs.washington.edu/ALGES/TACL2015/-/blob/master/questions.json?ref_type=heads) (100.0% of entire set) [SVAMP](https://github.com/arkilpatel/SVAMP) (100.0% of entire set) [CommonsenseQA](https://www.tau-nlp.sites.tau.ac.il/commonsenseqa) (100.0% of val set) [StrategyQA](https://allenai.org/data/strategyqa) (100.0% of entire set) [COPA](https://people.ict.usc.edu/~gordon/copa.html) (20.0% of val set) [Last Letter](https://github.com/RUCKBReasoning/CoT-KA) (100.0% of entire set) [Coin Flip](https://huggingface.co/datasets/skrishna/coin_flip) (16.0% of entire set) [Date](https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/date_understanding/README.md) (100.0% of entire set) [Object](https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/object_counting/README.md) (100.0% of entire set) [RTE](https://paperswithcode.com/dataset/rte) (4.8% of entire set) [CommitmentBank](https://github.com/mcdm/CommitmentBank) (4.6% of entire set) [BoolQ](https://huggingface.co/datasets/google/boolq) (100.0% of test set) MuTual (100.0% of test set) [SAMsum](https://paperswithcode.com/dataset/samsum-corpus) (100.0% of test set) [CoNLL03](https://huggingface.co/datasets/conll2003) (100.0% of test set) [SST2](https://github.com/YJiangcm/SST-2-sentiment-analysis) (100.0% of val set)|
| **[Extractive summarization via ChatGPT for faithful summary generation](https://arxiv.org/abs/2304.04193)** | **Y** | N | 09/04/2023 | \- | Y | N | ChatGPT | BROWSER | Y | Y | Y | N |[Reddit](https://github.com/ctr4si/MMN) (2.4% of test set) [XSum](https://github.com/EdinburghNLP/XSum/tree/master/XSum-Dataset) (8.8% of test set) [PubMed](https://github.com/armancohan/long-summarization/tree/master) (19.9% of test set) [CNN/DM](https://huggingface.co/datasets/cnn_dailymail) (8.7% of test set)|
| **[Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark](https://arxiv.org/abs/2303.13648)** | **Y** | N | 15/03/2023 | \- | Y | N | ChatGPT | BROWSER | Y | Y | N | N |[CoNLL2014](https://www.comp.nus.edu.sg/~nlp/conll14st.html) (7.6% of test set)|
| **[Will affective computing emerge from foundation models and general ai? A first evaluation on chatgpt](https://arxiv.org/abs/2303.03186)** | **Y** | Y | 03/03/2023 | February 2023 | Y | Y | ChatGPT | BROWSER | Y | N | N | N |[First Impressions](https://chalearnlap.cvc.uab.cat/dataset/24/description/) (100.0% of entire set) [Sentiment140](https://huggingface.co/datasets/sentiment140) (100.0% of entire set) [The Suicide and Depression dataset](https://www.kaggle.com/datasets/nikhileswarkomati/suicide-watch) (100.0% of entire set)|
| **[ChatGPT: Jack of all trades, master of none](https://arxiv.org/abs/2302.10724)** | **Y** | Y | 21/02/2023 | \- | Y | Y | Both | BROWSER | Y | Y | N | N |[WikiDetox Aggr. - Aggression](https://github.com/ewulczyn/wiki-detox/}{UnhealthyPer} (test or dev) \href{https://github.com/ewulczyn/wiki-detox/) (4.3% of test or dev set) [WikiDetox Aggr. - AggressionPer](https://github.com/ewulczyn/wiki-detox/}{UnhealthyPer} (test or dev) \href{https://github.com/ewulczyn/wiki-detox/) (0.2% of test or dev set) [CoLa](https://nyu-mll.github.io/CoLA/) (100.0% of dev set) [ColBERT](https://github.com/Moradnejad/ColBERT-Using-BERT-Sentence-Embedding-for-Humor-Detection/tree/master/Data) (2.5% of test or dev set) [Sarcasmania - Sarcasm](https://www.kaggle.com/datasets/rmsharks4/sarcasmania-dataset) (16.7% of test or dev set) [SMS Spam v.1](https://www.kaggle.com/datasets/uciml/sms-spam-collection-dataset) (100.0% of test or dev set) [WiC - WordContext](https://pilehvar.github.io/wic/) (100.0% of test or dev set) [RTE](https://paperswithcode.com/dataset/rte) (100.0% of dev set) [WNLI](https://huggingface.co/datasets/SetFit/wnli) (100.0% of dev set) [SQuAD v2](https://huggingface.co/datasets/squad_v2) (8.4% of dev set) [GSM8K - MathQA ](https://github.com/openai/grade-school-math) (75.8% of test set) ClarinEmo (100.0% of test or dev set) [GoEmotions - GoEmo](https://github.com/google-research/google-research/tree/master/goemotions) (18.4% of test or dev set) [GoEmotions - GoEmoPer0](https://github.com/google-research/google-research/tree/master/goemotions) (5.9% of test or dev set) [GoEmotions - GoEmoPer1](https://github.com/google-research/google-research/tree/master/goemotions) (5.9% of test or dev set) [GoEmotions - GoEmoPer2](https://github.com/google-research/google-research/tree/master/goemotions) (5.9% of test or dev set) [GoEmotions - GoEmoPer3](https://github.com/google-research/google-research/tree/master/goemotions) (5.9% of test or dev set) [Unhealthy Conv. - Unhealthy](https://github.com/conversationai/unhealthy-conversations) (2.2% of test or dev set) [Unhealthy Conv. - UnhealthyPer](https://github.com/conversationai/unhealthy-conversations) (0.4% of test or dev set) [PolEmo2 - PolEmo](https://clarin-pl.eu/dspace/handle/11321/710) (100.0% of test or dev set) [TweetEval - TweetEmoji](https://github.com/cardiffnlp/tweeteval/tree/main/datasets/emoji) (3.3% of test or dev set) [TweetEval - TweetSent](https://github.com/cardiffnlp/tweeteval/tree/main/datasets/sentiment) (41.8% of test or dev set) [TweetEval - TweetStance](https://github.com/cardiffnlp/tweeteval/tree/main/datasets/stance) (100.0% of test set) [RACE - ReAding](https://www.cs.cmu.edu/~glai1/data/race/) (20.4% of test or dev set) [Raganato - WSD](http://lcl.uniroma1.it/wsdeval/) (100.0% of test or dev set)|
| **[Zero-Shot Information Extraction via Chatting with ChatGPT](https://arxiv.org/abs/2302.10205)** | **Y** | N | 20/02/2023 | December 2022 | Y | Y | ChatGPT | BROWSER | N | Y | N | N |[NYT11-HRL](https://github.com/truthless11/HRL-RE/tree/master/data/NYT11) (100.0% of entire set) [DuIE2.0](https://github.com/zhoujx4/DuIE) (100.0% of entire set) [conll++](https://github.com/ZihanWangKi/CrossWeigh) (100.0% of entire set) [MSRA](https://github.com/OYE93/Chinese-NLP-Corpus/tree/master/NER/MSRA) (100.0% of entire set) [DuEE1.0](https://github.com/zhoujx4/DuEE) (100.0% of entire set) [ACE05](https://paperswithcode.com/dataset/ace-2005) (100.0% of entire set)|
| **[Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent](https://arxiv.org/pdf/2304.09542.pdf)** | **Y** | N | 19/04/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | Y | | |
| **[Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://arxiv.org/abs/2303.04048#:~:text=Experimental%20results%20show%20that%20compared,of%20the%20meta%2Devaluation%20datasets.)** | **Y** | N | 07/03/2023 | February and March 2023 | Y | Y | ChatGPT | BROWSER | N | Y | N | N |[SummEval](https://github.com/Yale-LILY/SummEval) (100.0% of entire set) [NewsRoom](https://huggingface.co/datasets/newsroom) (100.0% of entire set) [OpenMEVA-ROC](https://github.com/thu-coai/OpenMEVA) (100.0% of entire set) [RealSumm](https://github.com/neulab/REALSumm) (100.0% of entire set)|
| **[G-EVAL: NLG Evaluation using GPT-4 with Better Human Alignment](https://arxiv.org/abs/2303.16634)** | **Y** | N | 29/03/2023 | \- | Y | Y | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://arxiv.org/abs/2302.14520)** | **Y** | Y | 28/02/2023 | February 2023 | Y | Y | Both | BROWSER | N | Y | N | |[MQM-2022](https://github.com/google/wmt-mqm-human-evaluation/tree/main/generalMT2022) (100.0% of test set)|
| **[CHATGPT OUTPERFORMS CROWD-WORKERS FOR TEXT-ANNOTATION TASKS](https://arxiv.org/pdf/2303.15056.pdf)** | **Y** | N | 19/07/2023 | March and April 2023 | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N | | |
| **[SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark](https://arxiv.org/pdf/2307.15020.pdf)** | **Y** | N | 27/07/2023 | \- | Y | Y | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://arxiv.org/pdf/2305.08322.pdf)** | **Y** | N | 17/05/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning](https://arxiv.org/pdf/2304.05613.pdf)** | **Y** | N | 12/04/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[Zero-Shot Cross-Lingual Summarization via Large Language Models](https://arxiv.org/abs/2302.14229)** | **Y** | N | 28/02/2023 | \- | Y | N | Both | BROWSER | Y | Y | N | N |[CrossSum](https://github.com/csebuetnlp/CrossSum) (1.2% of test set) [WikiLingua](https://github.com/esdurmus/Wikilingua) (0.2% of test set) [XSAMSum](https://github.com/krystalan/ClidSum/tree/main#2-clidsum-benchmark-dataset) (<0.1% of test set)|
| **[A categorical archive of chatgpt failures](https://arxiv.org/abs/2302.03494)** | **N/A** | | 06/02/2023 | \- | Y | N | ChatGPT | BROWSER | \- | \- | | | |
| **[Benchmarks for automated commonsense reasoning: A survey](https://arxiv.org/abs/2302.04752)** | **Y** | N | 09/02/2023 | \- | Y | Y | ChatGPT | BROWSER | Y | N | N | Y |custom commonsense reasoning dataset (100.0% of entire set)|
| **[Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation.](https://arxiv.org/abs/2305.07375#:~:text=Experiments%20show%20that%20ChatGPT%20is,upgrading%20processes%2C%20such%20as%20RLHF.)** | **Y** | N | 12/05/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | Y | | |
| **[Causal reasoning and large language models: Opening a new frontier for causality](https://arxiv.org/pdf/2305.00050.pdf?trk=article-ssr-frontend-pulse_x-social-details_comments-action_comment-text)** | **Y** | N | 08/05/2023 | \- | Y | N | Both | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Theory of mind may have spontaneously emerged in large language models](https://arxiv.org/abs/2302.02083)** | **Y** | N | 04/02/2023 | March 2023 | Y | Y | Both | BROWSER | Y | Y | N | Y |theory-of-mind-custom 1/02 (100.0% of entire set)|
| **[Boosting Theory-of-Mind Performance in Large Language Models via Prompting](https://arxiv.org/ftp/arxiv/papers/2304/2304.11490.pdf)** | **Y** | N | 22/04/2023 | \- | Y | Y | Both | BROWSER | N | N | N | Y |theory-of-mind-custom 2 (100.0% of entire set)|
| **[Does ChatGPT have Theory of Mind?](https://arxiv.org/abs/2305.14020)** | **Y** | N | 23/05/2023 | March 2023 | Y | N | Both | BROWSER | N | N | N | Y |theory-of-mind-custom 3 (100.0% of entire set)|
| **[Can ChatGPT Defend the Truth? Automatic Dialectical Evaluation Elicits LLMs' Deficiencies in Reasoning](https://arxiv.org/abs/2305.13160)** | **Y** | N | 22/05/2023 | \- | N | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N | | |
| **[Evaluating the logical reasoning ability of chatgpt and gpt-4](https://arxiv.org/abs/2304.03439)** | **Y** | N | 07/04/2023 | \- | Y | Y | Both | BROWSER | N | Y | N | N |[LogiQA 2.0](https://github.com/csitfun/LogiQA2.0) (100.0% of test set) [LogiQA 2.0-zh](https://github.com/csitfun/LogiQA2.0) (100.0% of test set) MED (100.0% of test set) [ConjNLI](https://github.com/swarnaHub/ConjNLI) (100.0% of test set) [TaxiNLI](https://github.com/microsoft/TaxiNLI) (100.0% of test set) [AR-LSAT ](https://github.com/zhongwanjun/AR-LSAT) (100.0% of test set) [ReClor](https://whyu.me/reclor/) (100.0% of dev set) [ConTRoL "logical reasoning"](https://github.com/csitfun/ConTRoL-dataset) (100.0% of test set) [HELP](https://github.com/verypluming/HELP) (100.0% of test set) [MNLI](https://huggingface.co/datasets/multi_nli) (100.0% of test set) custom-LogiQA 2.0 (100.0% of entire set)|
| **[We're Afraid Language Models Aren't Modeling Ambiguity](https://arxiv.org/pdf/2304.14399.pdf?trk=public_post_comment-text)** | **Y** | N | 27/04/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Sparks of artificial general intelligence: Early experiments with gpt-4](https://arxiv.org/pdf/2303.12712.pdf?utm_source=webtekno)** | **N/A** | | \- | \- | \- | \- | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | \- | \- | | | |
| **[Chatgpt participates in a computer science exam](https://arxiv.org/abs/2303.09461)** | **Y** | N | 08/03/2023 | February 2023 | Y | Y | Both | BROWSER | N | N | N | Y |custom-CS exam (100.0% of entire set)|
| **[Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls](https://arxiv.org/pdf/2305.03111.pdf)** | **Y** | N | 30/05/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Mathematical capabilities of chatgpt](https://arxiv.org/pdf/2301.13867.pdf?fbclid=IwAR0ja5OIlhmUeP12YzbexspsTXkc4gkEZefCVPuYLuCMjMx1AgGauDST7K0)** | **Y** | N | 31/01/2023 | January 2023 | Y | Y | Both | BROWSER | N | N | N | N |[GHOSTS](https://github.com/friederrr/GHOSTS) (100.0% of entire set)|
| **[Could an artificial-intelligence agent pass an introductory physics course?](https://arxiv.org/abs/2301.12127)** | **Y** | Y | 28/01/2023 | January 2023 | Y | PART | ChatGPT | BROWSER | N | N | N | Y |custom-made physics exam&homeworks (100.0% of entire set)|
| **[Investigating the Use of an Artificial Intelligence Chatbot with General Chemistry Exam Questions](https://pubs.acs.org/doi/10.1021/acs.jchemed.3c00027)** | **Y** | Y | 10/01/2023 | March 2023 | Y | Y | ChatGPT | BROWSER | N | N | N | Y |custom-made chemistry exam (100.0% of entire set)|
| **[How do physics students evaluate artificial intelligence responses on comprehension questions? A study on the perceived scientific accuracy and linguistic quality of ChatGPT](https://arxiv.org/abs/2304.05906)** | **Y** | Y | 12/04/2023 | February 2023 | N/A | Y | ChatGPT | BROWSER | N | N | N | Y |custom-made physics exam (100.0% of entire set)|
| **[Chatgpt goes to law school](https://collimateur.uqam.ca/wp-content/uploads/sites/11/2023/01/SSRN-id4335905.pdf)** | **Y** | Y | 23/01/2023 | December 2022 | Y | N | ChatGPT | BROWSER | N | N | N | Y |custom-made law exam (100.0% of entire set)|
| **[ChatGPT has aced the test of understanding in college economics: Now what?](https://journals.sagepub.com/doi/pdf/10.1177/05694345231169654)** | **Y** | Y | 08/03/2023 | February 2023 | Y | N | ChatGPT | BROWSER | N | N | N/A | N |TUCE test (100.0% of entire set)|
| **[The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges](https://arxiv.org/abs/2304.05351)** | **Y** | N | 10/04/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment](https://mededu.jmir.org/2023/1/e45312/)** | **Y** | Y | 26/12/2022 | November 2022 | Y | N | ChatGPT | BROWSER | Y | N | N/A | Y |[NBME medical questions](https://jmir.org/api/download?alt_name=mededu_v9i1e45312_app1.xlsx&filename=3c2adca5ee88328073c589af108a5697.xlsx) (100.0% of entire set) [AMBOSS medial questions](https://jmir.org/api/download?alt_name=mededu_v9i1e45312_app1.xlsx&filename=3c2adca5ee88328073c589af108a5697.xlsx) (1.7% of entire set)|
| **[Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models](https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000198&fbclid=IwAR3U9R0A5QwAVMACFfgA79EFYWu32uFE8upittW5ZEb9qaNSZyWXpxdnJU4)** | **Y** | Y | 21/12/2022 | November 2022 | Y | Y | ChatGPT | BROWSER | Y | Y | Y | Y |[USMLE medial q](https://jmir.org/api/download?alt_name=mededu_v9i1e45312_app1.xlsx&filename=3c2adca5ee88328073c589af108a5697.xlsx) (93.0% of entire set)|
| **[Evaluating the performance of chatgpt in ophthalmology: An analysis of its successes and shortcomings](https://www.sciencedirect.com/science/article/pii/S2666914523000568)** | **Y** | Y | 26/01/2023 | \- | Y | N | ChatGPT | BROWSER | Y | N | N/A | Y |BCSC Self-Assessment Program (5.8% of entire set) OphthoQuestions (5.7% of entire set)|
| **[Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study](https://www.mdpi.com/1660-4601/20/4/3378)** | **Y** | Y | 15/02/2023 | January 2023 | Y | Y | ChatGPT | BROWSER | Y | N | N/A | Y |custom medical diagnosis dataset (100.0% of entire set)|
| **[Assessing the utility of ChatGPT throughout the entire clinical workflow](https://www.medrxiv.org/content/medrxiv/early/2023/02/26/2023.02.21.23285886.full.pdf)** | **Y** | Y | 26/02/2023 | January 2023 | Y | N | ChatGPT | BROWSER | Y | N | N/A | N |Merck Sharpe & Dohme (MSD) Clinical Manual (100.0% of entire set)|
| **[ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases](https://www.medrxiv.org/content/medrxiv/early/2023/04/27/2023.04.20.23288859.full.pdf)** | **Y** | N | 26/04/2023 | April 2023 | Y | N | Both | BROWSER | Y | N | N/A | Y |custom medical 1 (100.0% of entire set)|
| **[Can AI tell good stories? narrative transportation and persuasion with ChatGPT](https://psyarxiv.com/c3549/download?format=pdf)** | **Y** | N | 05/04/2023 | \- | N | N | ChatGPT | BROWSER | N | N | N/A | Y |custom narrative (100.0% of entire set)|
| **[Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://arxiv.org/abs/2306.04181)** | **Y** | N | 07/06/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[Chatgpt: A meta-analysis after 2.5 months](https://arxiv.org/pdf/2302.13795.pdf)** | **N/A** | | \- | \- | N/A | N/A | ChatGPT | | N/A | N/A | N/A | | |
| **[Causal-discovery performance of chatgpt in the context of neuropathic pain diagnosis](https://arxiv.org/abs/2301.13819)** | **Y** | N | 24/01/2023 | January 2023 | Y | Y | ChatGPT | BROWSER | Y | N | N/A | Y |custom medical 2 (100.0% of entire set)|
| **[Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://arxiv.org/pdf/2304.01852.pdf)** | **N/A** | | \- | \- | N/A | N/A | ChatGPT | | N/A | N/A | N/A | | |
| **[Harnessing the power of llms in practice: A survey on chatgpt and beyond](https://arxiv.org/pdf/2304.13712.pdf)** | **N/A** | | \- | \- | N/A | N/A | \- | | \- | \- | | | |
| **[Holistic evaluation of language models](https://crfm.stanford.edu/helm/latest/)** | **Y** | Y | 1/10/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[A Survey on Evaluation of Large Language Models](https://github.com/MLGroupJLU/LLM-eval-survey)** | **N/A** | | \- | \- | N/A | N/A | \- | | N/A | N/A | N/A | | |
| **[Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert](https://arxiv.org/abs/2302.10198)** | **Y** | N | 19/02/2023 | February 2023 | Y | Y | ChatGPT | BROWSER | Y | Y | N | N |[CoLa](https://nyu-mll.github.io/CoLA/) (2.5% of dev set) [SST-2](https://github.com/YJiangcm/SST-2-sentiment-analysis) (1.3% of dev set) [MRPC](https://www.microsoft.com/en-us/download/details.aspx?id=52398) (1.4% of dev set) [STS-B](https://adapterhub.ml/explore/sts/sts-b/) (3.5% of dev set) [QQP](https://gluebenchmark.com/) (<0.1% of dev set) [MNLI](https://huggingface.co/datasets/multi_nli) (0.1% of val set) [QNLI](https://paperswithcode.com/dataset/qnli) (0.4% of dev set) [RTE](https://paperswithcode.com/dataset/rte) (0.8% of dev set) [WNLI](https://huggingface.co/datasets/SetFit/wnli) (17.1% of dev set)|
| **[On the robustness of chatgpt: An adversarial and out-of-distribution perspective](https://arxiv.org/abs/2302.12095)** | **Y** | N | 22/02/2023 | February 2023 | Y | Y | ChatGPT | BROWSER | Y | Y | N | N |[AdvGLUE_SST-2](https://adversarialglue.github.io/instructions/) (100.0% of dev set) [AdvGLUE_QQP](https://github.com/AI-secure/adversarial-glue/tree/main) (100.0% of dev set) [AdvGLUE_MNLI](https://github.com/AI-secure/adversarial-glue/tree/main) (100.0% of dev set) [AdvGLUE_QNLI](https://github.com/AI-secure/adversarial-glue/tree/main) (100.0% of dev set) [AdvGLUE_RTE](https://github.com/AI-secure/adversarial-glue/tree/main) (100.0% of dev set) custom AdvGLUE-T (100.0% of dev set) [ANLI-R3](https://github.com/facebookresearch/anli?tab=readme-ov-file) (100.0% of test set) [Flipkart](https://www.kaggle.com/datasets/niraliivaghani/flipkart-product-customer-reviews-dataset) (0.1% of entire set) [DDXPlus (EN)](https://github.com/bruzwen/ddxplus) (<0.1% of test set)|
| **[An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)](https://arxiv.org/abs/2302.13814)** | **Y** | N | 23/02/2023 | January and February 2023 | Y | Y | ChatGPT | BROWSER | N | N | N/A | N |[DRAW-1K](https://www.microsoft.com/en-us/download/details.aspx?id=52628) (100.0% of entire set)|
| **[Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family](https://arxiv.org/abs/2303.07992)** | **Y** | N | 14/03/2023 | February and March 2023 | Y | Y | ChatGPT | BROWSER | Y | Y | N | N |[KQApro](https://thukeg.gitee.io/kqa-pro/) (90.0% of entire set) [LC-quad2.0](https://github.com/AskNowQA/LC-QuAD2.0) (100.0% of entire set) [WQSP](https://github.com/tan92hl/Complex-Question-Answering-Evaluation-of-GPT-family/tree/main/datasets/WQSP) (99.2% of entire set) [CWQ](https://allenai.org/data/complexwebquestions) (100.0% of entire set) [GrailQA](https://dki-lab.github.io/GrailQA/) (100.0% of test set) [GraphQuestions](https://github.com/ysu1989/GraphQuestions) (100.0% of entire set) [QALD-9](https://github.com/ag-sc/QALD/tree/master) (100.0% of entire set) [MKQA](https://github.com/apple/ml-mkqa) (2.3% of entire set)|
| **[Instruction tuning with gpt-4](https://arxiv.org/pdf/2304.03277.pdf)** | **Y** | N | 06/04/2023 | \- | Y | Y | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks](https://arxiv.org/ftp/arxiv/papers/2303/2303.00293.pdf)** | **N/A** | | 01/03/2023 | \- | N/A | N/A | \- | API - POST MARCH 2023 OR OPT-OUT | N/A | N/A | N/A | |custom-made psychological dataset (100.0% of entire set)|
| **[Consistency analysis of chatgpt](https://arxiv.org/abs/2303.06273)** | **Y** | N | 11/03/2023 | February 2023 | Y | N | ChatGPT | BROWSER | Y | Y | Y | N |[becel-SNLI](https://github.com/MJ-Jang/BECEL/tree/main) (4.9% of test set) [becel-RTE](https://github.com/MJ-Jang/BECEL/tree/main) (36.6% of test set) [becel-MRPC](https://github.com/MJ-Jang/BECEL/tree/main/data/mrpc) (16.6% of test set)|
| **[Does ChatGPT resemble humans in language use?](https://arxiv.org/abs/2303.08014)** | **Y** | N | 10/03/2023 | December 2022 and January 2023 | Y | Y | ChatGPT | BROWSER | N | N | N/A | Y | |
| **[A comprehensive capability analysis of gpt-3 and gpt-3.5 series models](https://arxiv.org/ftp/arxiv/papers/2303/2303.10420.pdf)** | **Y** | N | 18/03/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability](https://arxiv.org/pdf/2303.13547.pdf)** | **Y** | N | 12/03/2023 | \- | Y | Y | ChatGPT | BROWSER | N | Y | N | N |[Spider](https://drive.usercontent.google.com/download?id=1TqleXec_OykOYFREKKtschzY29dUcVAQ&export=download&authuser=0) (100.0% of dev set) [Spider-SYN](https://github.com/ygan/Spider-Syn) (100.0% of dev set) [Spider-DK](https://github.com/ygan/Spider-DK) (100.0% of dev set) [Spider-Realistic](https://zenodo.org/record/5205322) (100.0% of dev set) [Spider-CG(sub)](https://github.com/ygan/SpiderSS-SpiderCG) (100.0% of entire set) [Spider-CG(app)](https://github.com/ygan/SpiderSS-SpiderCG) (100.0% of entire set) [ADVETA(rpl)-spider](https://github.com/microsoft/ContextualSP) (100.0% of dev set) [ADVETA(add)-spider](https://github.com/microsoft/ContextualSP) (100.0% of dev set) [CSpider](https://taolusi.github.io/CSpider-explorer/) (100.0% of dev set) [DuSQL](https://github.com/luge-ai/luge-ai/tree/master/semantic-parsing) (100.0% of entire set) [SParC](https://github.com/taoyds/sparc) (100.0% of dev set) [CoSQL](https://yale-lily.github.io/cosql) (100.0% of dev set)|
| **[Is ChatGPT a good sentiment analyzer? A preliminary study](https://arxiv.org/pdf/2304.04339.pdf)** | **Y** | N | 10/04/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | Y | | |
| **[A preliminary evaluation of chatgpt for zero-shot dialogue understanding](https://arxiv.org/pdf/2304.04256.pdf)** | **Y** | N | 09/04/2023 | February 2023 | Y | N | ChatGPT | BROWSER | N | Y | N | N |[ATIS](https://github.com/howl-anderson/ATIS_dataset/tree/master) (100.0% of test set) [SNIPS](https://github.com/sonos/nlu-benchmark) (100.0% of test set) [MultiWOZ 2.1](https://huggingface.co/datasets/ConvLab/multiwoz21) (100.0% of test set) [MultiWOZ 2.4](https://github.com/smartyfh/MultiWOZ2.4) (100.0% of test set)|
| **[Zero-shot Temporal Relation Extraction with ChatGPT](https://arxiv.org/pdf/2304.05454.pdf)** | **Y** | Y | 11/04/2023 | \- | Y | N | ChatGPT | BROWSER | N | Y | N | N |[TimeBank-Dense](https://www.usna.edu/Users/cs/nchamber/caevo/) (100.0% of test set) [MATRES](https://github.com/qiangning/MATRES) (100.0% of test set) [TDDiscourse](https://github.com/aakanksha19/TDDiscourse) (100.0% of test set)|
| **[Can chatgpt reproduce human-generated labels? a study of social computing tasks](https://arxiv.org/pdf/2304.10145.pdf)** | **Y** | N | 22/04/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[Chatgraph: Interpretable text classification by converting chatgpt knowledge to graphs](https://arxiv.org/pdf/2305.03513.pdf)** | **Y** | Y | 19/09/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue: An Empirical Study](https://arxiv.org/pdf/2305.08391.pdf)** | **Y** | N | 15/05/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detection](https://arxiv.org/ftp/arxiv/papers/2305/2305.13276.pdf)** | **Y** | N | 22/05/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://arxiv.org/pdf/2305.15005.pdf)** | **Y** | N | 24/05/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[ChatGPT for Suicide Risk Assessment on Social Media: Quantitative Evaluation of Model Performance, Potentials and Limitations](https://arxiv.org/pdf/2306.09390.pdf)** | **Y** | Y | 15/06/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Metacognitive Prompting Improves Understanding in Large Language Models](https://arxiv.org/pdf/2308.05342.pdf)** | **Y** | N | 10/08/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Exploring ai ethics of chatgpt: A diagnostic analysis](https://arxiv.org/pdf/2301.12867.pdf)** | **Y** | N | 30/01/2023 | December 2022 -January 2023 | PART | N | ChatGPT | BROWSER | Y | Y | N | N |BBQ (80.0% of entire set) BOLD (80.0% of entire set) custom-made query to inspect gender occupation bias (100.0% of entire set) [IMDB](https://www.cs.cornell.edu/people/pabo/movie-review-data/) (1.9% of test set) [IMDB-contrast](https://github.com/allenai/contrast-sets/tree/main/IMDb) (100.0% of entire set) [BoolQ](https://huggingface.co/datasets/google/boolq) (10.3% of test set) [BoolQ-contrast](https://github.com/allenai/contrast-sets/tree/main/BoolQ) (100.0% of test set) [FLORES-200](https://github.com/facebookresearch/flores/tree/main/flores200) (80.0% of dev set) [ProsocialDialog](https://huggingface.co/datasets/allenai/prosocial-dialog) (0.2% of test set) custom jailbreaking (100.0% of entire set) [Open-BookQA](https://allenai.org/data/open-book-qa) (80.0% of test set) [TruthfulQA](https://github.com/sylinrl/TruthfulQA) (100.0% of test set) [TruthfulQA gen](https://github.com/sylinrl/TruthfulQA) (12.2% of test set) [RealToxicPrompts](https://github.com/allenai/real-toxicity-prompts) (80.0% of entire set) custom character role-play (100.0% of entire set)|
| **[Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech](https://arxiv.org/pdf/2302.07736.pdf)** | **Y** | Y | 11/02/2023 | January 2023 | Y | N | ChatGPT | BROWSER | Y | N | N/A | N |[LatentHatred](https://github.com/CLARIN-PL/chatgpt-evaluation-01-2023/}{GoEmoPer0-3} \href{https://github.com/SALT-NLP/implicit-hate) (12.0% of entire set)|
| **[Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media](https://arxiv.org/pdf/2304.03087.pdf)** | **Y** | N | 06/04/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[The Self-Perception and Political Biases of ChatGPT](https://arxiv.org/abs/2304.07333)** | **Y** | N | 14/04/2023 | March 2023 | N | N | ChatGPT | BROWSER | N | N | N/A | N |custom political compass test (100.0% of entire set)|
| **[Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://arxiv.org/pdf/2305.07004.pdf)** | **Y** | N | 11/05/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[BAD: BiAs Detection for Large Language Models in the context of candidate screening](https://arxiv.org/abs/2305.10407)** | **Y** | N | 17/05/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[DECODINGTRUST: A Comprehensive Assessment of Trustworthiness in GPT Models](https://arxiv.org/pdf/2306.11698.pdf)** | **Y** | N | 20/06/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[On Large Language Models' Selection Bias in Multi-Choice Questions](https://arxiv.org/pdf/2309.03882.pdf)** | **Y** | N | 08/09/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | Y | | |
| **[Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization](https://arxiv.org/abs/2302.08081)** | **Y** | N | 16/02/2023 | February 2023 | Y | N | ChatGPT | BROWSER | Y | Y | Y | N |[QMSum](https://github.com/Yale-LILY/QMSum) (35.5% of test set) [SQuALITY](https://github.com/nyu-mll/SQuALITY) (55.5% of test set) [CovidET](https://github.com/honglizhan/CovidET) (25.1% of test set) [NEWTS](https://github.com/ali-bahrainian/NEWTS) (16.6% of test set)|
| **[ChatGPT as a Factual Inconsistency Evaluator for Text Summarization](https://arxiv.org/pdf/2303.15621.pdf)** | **Y** | N | 13/04/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Uncovering ChatGPT's Capabilities in Recommender Systems](https://arxiv.org/abs/2305.02182)** | **Y** | N | 24/08/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://arxiv.org/pdf/2305.11747.pdf)** | **Y** | N | 22/05/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text](https://arxiv.org/pdf/2305.13304.pdf)** | **Y** | N | 22/05/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N/A | N | N/A | | |
| **[PokemonChat: Auditing ChatGPT for Pokémon Universe Knowledge](https://arxiv.org/abs/2306.03024)** | **Y** | N | 05/06/2023 | December 2022 | Y | N | ChatGPT | BROWSER | N/A | N | N/A | Y |custom pokemon dataset (100.0% of entire set)|
| **[Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Study](https://arxiv.org/abs/2306.01169)** | **Y** | N | 01/06/2023 | \- | Y | N | ChatGPT | BROWSER | Y | N | N | N |custom book and article summarization dataset (100.0% of entire set)|
| **[Generative Job Recommendations with Large Language Model](https://arxiv.org/abs/2307.02157)** | **Y** | N | 05/06/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N/A | N/A | N/A | | |
| **[Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviews](https://arxiv.org/pdf/2307.06464.pdf)** | **Y** | N | 12/06/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://arxiv.org/abs/2307.11088)** | **Y** | N | 31/07/2023 | \- | Y | Y | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[LLM-Rec: Personalized Recommendation via Prompting Large Language Models](https://arxiv.org/pdf/2307.15780.pdf)** | **N/A** | | 2023 | \- | \- | \- | \- | | \- | \- | | | |
| **[LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://arxiv.org/pdf/2308.14508.pdf)** | **Y** | N | 28/08/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Can Large Language Models Understand Real-World Complex Instructions](https://arxiv.org/pdf/2309.09150.pdf)** | **Y** | N | 27/09/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Mind meets machine: Unravelling GPT-4's cognitive psychology](https://arxiv.org/abs/2303.11436)** | **Y** | N | 20/03/2023 | \- | Y | N | GPT-4 | BROWSER | N | N | N/A | N | |
| **[Capabilities of GPT-4 on Medical Challenge Problems](https://arxiv.org/pdf/2303.13375.pdf)** | **Y** | N | 11/04/2023 | \- | Y | N | Both | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[GPT is becoming a Turing machine: Here are some ways to program it](https://arxiv.org/pdf/2303.14310.pdf)** | **Y** | N | 25/03/2023 | \- | Y | N | GPT-4 | BROWSER | Y | N | N/A | Y |custom-made dataset of coding problems (100.0% of entire set)|
| **[ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://arxiv.org/pdf/2303.16421.pdf)** | **Y** | N | 29/03/2023 | March 2023 | Y | N | ChatGPT | BROWSER | Y | N | N/A | N |[CommonsenseQA](https://www.tau-nlp.sites.tau.ac.il/commonsenseqa) (9.8% of dev set) [OpenBookQA](https://allenai.org/data/open-book-qa) (24.0% of dev set) [WSC](https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html) (36.6% of dev set) [PIQA](https://github.com/ybisk/ybisk.github.io/tree/master/piqa) (6.5% of dev set) [Social IQA](https://allenai.org/data/socialiqa) (6.1% of dev set) [ARC](https://allenai.org/data/arc) (33.4% of dev set) [QASC](https://allenai.org/data/qasc) (12.9% of dev set) [HellaSWAG](https://rowanzellers.com/hellaswag/) (1.2% of dev set) [NumerSense](https://inklab.usc.edu/NumerSense/) (10.6% of dev set) [MC-TACO](https://github.com/CogComp/MCTACO) (3.1% of dev set) [ProtoQA](https://github.com/iesl/protoqa-data) (100.0% of dev set)|
| **[Humans in Humans Out: On GPT Converging Toward Common Sense in both Success and Failure](https://arxiv.org/pdf/2303.17276.pdf)** | **Y** | N | 30/03/2023 | | Y | N | Both | BROWSER | N | N | N/A | N |custom-made theory of reason (100.0% of entire set)|
| **[Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?](https://arxiv.org/abs/2308.00189)** | **N/A** | | 31/07/2023 | \- | \- | \- | \- | | \- | \- | | | |
| **[How well do Large Language Models perform in Arithmetic tasks?](https://arxiv.org/abs/2304.02015)** | **Y** | N | 16/03/2023 | \- | Y | Y | Both | BROWSER | Y | Y | Y | N |custom mathematical (100.0% of entire set)|
| **[Uncertainty in Natural Language Generation: From Theory to Applications](https://arxiv.org/abs/2307.15703)** | **N/A** | | 28/07/2023 | \- | N/A | N/A | \- | | N/A | N/A | N/A | | |
| **[ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking about](https://arxiv.org/abs/2304.03325)** | **Y** | N | 06/04/2023 | \- | N | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[How is ChatGPT's behavior changing over time?](https://arxiv.org/abs/2307.09009v1)** | **Y** | N | 18/07/2023 | \- | N | Y | Both | API - POST MARCH 2023 OR OPT-OUT | Y | N/A | N/A | | |
| **[ChatABL: Abductive Learning via Natural Language Interaction with ChatGPT](https://arxiv.org/pdf/2304.11107.pdf)** | **Y** | N | 21/04/2023 | \- | PART | N | Both | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models](https://arxiv.org/abs/2307.10236)** | **Y** | N | 16/07/2023 | \- | PART | N | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | Y | | |
| **[StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://arxiv.org/pdf/2305.09645.pdf)** | **Y** | N | 16/05/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N | | |
| **[Chain-of-Symbol Prompting Elicits Planning in Large Langauge Models](https://arxiv.org/pdf/2305.10276.pdf)** | **Y** | N | 26/06/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/pdf/2305.10601.pdf)** | **Y** | N | 17/05/2023 | \- | Y | Y | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[ChatLog: Recording and Analyzing ChatGPT Across Time](https://arxiv.org/abs/2304.14106)** | **Y** | N | 27/04/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[Chain of Knowledge: A Framework for Grounding Large Language Models with Structured Knowledge Bases](https://arxiv.org/pdf/2305.13269.pdf)** | **Y** | N | 3/10/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[ChatHaruhi: Reviving Anime Character in Reality via Large Language Model.](https://arxiv.org/abs/2308.09597)** | **Y** | N | 18/08/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N/A | Y | N/A | | |
| **[Adaptive Chameleon or Stubborn Sloth: Unraveling the Behavior of Large Language Models in Knowledge Conflicts](https://arxiv.org/abs/2305.13300)** | **Y** | N | 3/10/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[GPT-3.5 vs GPT-4: Evaluating ChatGPT's Reasoning Performance in Zero-shot Learning](https://arxiv.org/pdf/2305.12477v2.pdf)** | **Y** | N | 21/05/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | N | N | N | | |
| **[LogiCoT: Logical Chain-of-Thought Instruction-Tuning Data Collection with GPT-4](https://arxiv.org/pdf/2305.12147.pdf)** | **Y** | N | 20/05/2023 | \- | Y | Y | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | N/A | N/A | N/A | | |
| **[Enabling Large Language Models to Generate Text with Citations](https://arxiv.org/pdf/2305.14627.pdf)** | **Y** | N | 24/05/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[Why Does ChatGPT Fall Short in Providing Truthful Answers?](https://arxiv.org/pdf/2304.10513.pdf)** | **Y** | N | 24/05/2023 | \- | Y | N | Both | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation From Deductive, Inductive and Abductive Views](https://arxiv.org/pdf/2306.09841.pdf)** | **Y** | N | 08/08/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://arxiv.org/abs/2307.11019)** | **Y** | N | 23/07/2023 | late May and early June 2023 | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://arxiv.org/pdf/2307.10635.pdf)** | **Y** | N | 20/07/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[Think-on-Graph: Deep and Responsible Reasoning of Large Language Model with Knowledge Graph](https://arxiv.org/pdf/2307.07697v1.pdf)** | **Y** | N | 15/07/2023 | \- | N | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models](https://arxiv.org/pdf/2308.00304.pdf)** | **Y** | N | 14/08/2023 | \- | Y | N | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://arxiv.org/pdf/2308.11483.pdf)** | **Y** | N | 22/08/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[Large Language Models on the Chessboard: A Study on ChatGPT's Formal Language Comprehension and Complex Reasoning Skills](https://arxiv.org/pdf/2308.15118.pdf)** | **Y** | N | 29/08/2023 | \- | PART | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N/A | N | N/A | | |
| **[Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysis](https://arxiv.org/abs/2308.11224)** | **Y** | N | 22/08/2023 | June 2023 | Y | Y | Both | BROWSER | N/A | Y | N | N |custom graph (100.0% of entire set)|
| **[Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applications](https://arxiv.org/abs/2306.11892)** | **Y** | N | 20/06/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://arxiv.org/pdf/2309.10691.pdf)** | **Y** | N | 19/09/2023 | \- | Y | Y | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[ChatGPT for Robotics: Design Principles and Model Abilities](https://arxiv.org/abs/2306.17582v2)** | **N/A** | | 20/02/2023 | \- | Y | Y | ChatGPT | UNCLEAR | \- | N/A | | | |
| **[Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models: A Case Study on ChatGPT](https://arxiv.org/abs/2303.13809)** | **Y** | N | 24/03/2023 | March 2023 | Y | Y | ChatGPT | BROWSER | Y | N | N/A | N |[WMT20 EN-DE](https://paperswithcode.com/dataset/wmt-2020) (4.0% of test set) [WMT20 ZH-EN](https://paperswithcode.com/dataset/wmt-2020) (4.0% of test set)|
| **[Towards Making the Most of ChatGPT for Machine Translation](https://arxiv.org/abs/2303.13780)** | **Y** | N | 24/03/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[Linguistically Informed ChatGPT Prompts to Enhance Japanese-Chinese Machine Translation: A Case Study on Attributive Clauses](https://arxiv.org/abs/2303.15587)** | **Y** | N | 27/03/2023 | \- | Y | N | ChatGPT | UNCLEAR | N/A | N | N/A | N | |
| **[How to Design Translation Prompts for ChatGPT: An Empirical Study](https://arxiv.org/abs/2304.02182)** | **Y** | N | 21/04/2023 | \- | Y | N | ChatGPT | BROWSER | Y | N | N/A | N |[Flores-101](https://github.com/facebookresearch/flores/tree/main/previous_releases/flores101) (4.9% of test set) [WMT2014 Ott et al 2018](https://paperswithcode.com/dataset/wmt-2014) (10.0% of test set)|
| **[Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://arxiv.org/abs/2305.01210)** | **Y** | N | 10/06/2023 | \- | N/A | Y | Both | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness](https://arxiv.org/abs/2304.11633)** | **Y** | N | 23/04/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[Can ChatGPT Assess Human Personalities? A General Evaluation Framework](https://arxiv.org/abs/2303.01248)** | **Y** | N | 07/03/2023 | \- | Y | Y | ChatGPT | BROWSER | N | N | N/A | N |[Myers-Briggs Type Indicator](https://github.com/Kali-Hac/ChatGPT-MBTI) (100.0% of entire set)|
| **[Is ChatGPT Equipped with Emotional Dialogue Capabilities?](https://arxiv.org/pdf/2304.09582.pdf)** | **Y** | N | 19/4/2023 | \- | N | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://arxiv.org/pdf/2302.13007.pdf)** | **Y** | N | 20/03/2023 | \- | Y | N | ChatGPT | BROWSER | N | N | N/A | N |Amazon (customer reviews) (<0.1% of train set) [Symptoms (on Kaggle)](https://www.kaggle.com/datasets/paultimothymooney/medical-speech-transcription-and-intent) (100.0% of train set) [PubMed20K](https://huggingface.co/datasets/ml4pubmed/pubmed-classification-20k) (100.0% of train set)|
| **[Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations](https://arxiv.org/abs/2303.18027)** | **Y** | N | 5/4/2023 | 31/03/2023 | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[To ChatGPT, or not to ChatGPT: That is the question!](https://arxiv.org/abs/2304.01487)** | **Y** | N | 4/4/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N/A | N/A | | |
| **[In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT](https://arxiv.org/abs/2304.08979)** | **Y** | N | 18/08/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[Analyzing ChatGPT's Aptitude in an Introductory Computer Engineering Course](https://arxiv.org/abs/2304.06122)** | **Y** | N | 13/03/2023 | \- | Y | Y | ChatGPT | BROWSER | Y | N | N/A | Y |custom CS exams/homeworks (100.0% of entire set)|
| **[Linguistic ambiguity analysis in ChatGPT](https://arxiv.org/abs/2302.06426)** | **Y** | N | 13/02/2023 | \- | Y | Y | ChatGPT | BROWSER | N | N | N/A | Y |custom linguistic ambiguity (100.0% of entire set) custom (100.0% of entire set)|
| **[Using ChatGPT for Entity Matching](https://arxiv.org/abs/2305.03423)** | **Y** | N | 05/05/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[MEGA: Multilingual Evaluation of Generative AII](https://arxiv.org/abs/2303.12528)** | **Y** | Y | 22/10/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | Y | | |
| **[Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers](https://aclanthology.org/2023.bionlp-1.30/)** | **Y** | Y | 07/06/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Zero-shot Clinical Entity Recognition using ChatGPT](https://arxiv.org/abs/2303.16416v2)** | **Y** | N | 29/03/2023 | 22/02/2023 | Y | N | ChatGPT | BROWSER | Y | Y | N | N |[i2b2 2010](https://www.i2b2.org/NLP/Relations/) (33.5% of entire set) [MTSamples](https://mtsamples.com/) (1.9% of entire set)|
| **[Human-like Summarization Evaluation with ChatGPT](https://arxiv.org/abs/2304.02554)** | **Y** | N | 05/04/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **["HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media](https://arxiv.org/abs/2304.10619)** | **Y** | N | 20/04/2023 | \- | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | | |
| **[Large language models effectively leverage document-level context for literary translation, but critical errors persist](https://arxiv.org/abs/2304.03245)** | **Y** | N | 06/04/2023 | \- | Y | Y | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | N/A | N/A | N/A | | |
| **[BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models](https://arxiv.org/abs/2306.10968v1)** | **Y** | N | 19/06/2023 | \- | Y | Y | ChatGPT | BROWSER | Y | Y | N | N |[WMT22](https://www.statmt.org/wmt22/translation-task.html) (0.1% of test set)|
| **[Neural Machine Translation Data Generation and Augmentation using ChatGPT](https://arxiv.org/abs/2307.05779)** | **N/A** | | 11/07/2023 | \- | N/A | N/A | \- | \- | N/A | N/A | N/A | | |
| **[ChatGPT MT: Competitive for High- (but not Low-) Resource Languages](https://arxiv.org/abs/2309.07423)** | **Y** | N | 14/09/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | Y | | |
| **[How would Stance Detection Techniques Evolve after the Launch of ChatGPT?](https://arxiv.org/abs/2212.14548)** | **Y** | N | 30/12/2022 | \- | Y | N | ChatGPT | BROWSER | N | Y | N | N |[SemEval-2016 Task 6](https://afshinrahimi.github.io/semeval2016-task6/) (100.0% of test set) [P-Stance](https://github.com/chuchun8/PStance) (100.0% of test set)|
| **[BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models](https://arxiv.org/abs/2309.06085)** | **Y** | N | 12/09/2023 | \- | Y | N | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | | |
| **[Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts](https://aclanthology.org/2023.acl-long.313/)** | **N/A** | Y | 25/05/2023 | \- | Y | Y | \- | | \- | | | | |
| **[UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction](https://arxiv.org/abs/2303.01194)** | **Y** | Y | 02/03/2023 | December 2022 | Y | N | ChatGPT | BROWSER | Y | N/A | N/A | N |[SemEval-2023 Task 9](https://codalab.lisn.upsaclay.fr/competitions/7096#learn_the_details) (0.001% of training set)|
| **[Exploring the Feasibility of ChatGPT for Event Extraction](https://arxiv.org/abs/2303.03836)** | **Y** | N | 07/03/2023 | | Y | N | ChatGPT | BROWSER | Y | Y | N | N |[ACE05](https://paperswithcode.com/dataset/ace-2005) (0.3% of test set)|
| **[Does Synthetic Data Generation of LLMs Help Clinical Text Mining?](https://arxiv.org/abs/2303.04360)** | **Y** | N | 08/03/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction](https://arxiv.org/abs/2303.05063)** | **Y** | N | 09/03/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | \- | \- | \- | | |
| **[ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://arxiv.org/abs/2303.07839)** | **Y** | N | 11/03/2023 | January - February 2023 | Y | N/A | ChatGPT | BROWSER | N/A | N/A | N/A | Y | |
| **[SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://arxiv.org/abs/2303.08896)** | **Y** | N | 15/03/2023 | \- | Y | Y | \- | | N/A | N/A | N/A | | |
| **[Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potential](https://arxiv.org/abs/2303.09038)** | **Y** | Y | 16/03/2023 | | Y | N | Both | API - POST MARCH 2023 OR OPT-OUT | \- | N | N/A | | |
| **[An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answering](https://arxiv.org/abs/2303.10368)** | **Y** | Y | 18/03/2023 | January 2023 | Y | Y | ChatGPT | BROWSER | Y | Y | Y | N |[SimpleQuestions](https://github.com/davidgolub/SimpleQA/tree/master/datasets/SimpleQuestions) (0.3% of entire set) [WebQuestionsSP](https://www.microsoft.com/en-us/download/details.aspx?id=52763) (4.5% of entire set) [FreebaseQA](https://github.com/kelvin-jiang/FreebaseQA) (1.0% of entire set)|
| **[DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](https://arxiv.org/abs/2303.11032)** | **Y** | N | 20/03/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | Y | | |
| **[MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://arxiv.org/abs/2303.11381)** | **Y** | N | 20/03/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N/A | N/A | | |
| **[Is ChatGPT A Good Keyphrase Generator? A Preliminary Study](https://arxiv.org/abs/2303.13001)** | **Y** | N | 23/03/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | | |
| **[Beyond Black Box AI-Generated Plagiarism Detection: From Sentence to Document Level](https://aclanthology.org/2023.bea-1.58/)** | **Y** | Y | 13/06/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | N | |
| **[Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithms](https://arxiv.org/abs/2303.17650)** | **Y** | N | 30/03/2023 | February 2023 | Y | Y | ChatGPT | BROWSER | Y | N | N/A | N |[CNN DailyMail](https://huggingface.co/datasets/cnn_dailymail) (0.4% of test set)|
| **[MGTBench: Benchmarking Machine-Generated Text Detection](https://arxiv.org/abs/2303.14822)** | **N/A** | | 26/03/2023 | | N/A | N/A | \- | \- | N/A | N/A | N/A | | |
| **[Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://arxiv.org/abs/2210.13312)** | **Y** | N | 30/3/2023 | | N | N | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | Y | | |
| **[What would Harry say? Building Dialogue Agents for Characters in a Story](https://arxiv.org/abs/2211.06869)** | **Y** | N | 13/11/2022 | | N | N | ChatGPT | BROWSER | Y | Y | Y | N |custom Harry Potter dialogues (100.0% of entire set)|
| **[Reliable Natural Language Understanding with Large Language Models and Answer Set Programming](https://arxiv.org/abs/2302.03780)** | **N/A** | | | | N/A | N/A | \- | | N/A | N/A | N/A | | |
| **[ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models](https://arxiv.org/abs/2212.10474)** | **Y** | Y | 20/12/2022 | \- | Y | Y | ChatGPT | BROWSER | Y | Y | Y | Y | |
| **[Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End](https://arxiv.org/abs/2212.10522)** | **Y** | N | 20/12/2022 | | Y | N | ChatGPT | BROWSER | Y | Y | Y | Y |custom EMNLP 2022 abstracts (4.8% of entire set)|
| **[Modeling Label Semantics Improves Activity Recognition](https://arxiv.org/abs/2301.03462)** | **N/A** | | 01/01/2023 | | N/A | N/A | \- | | N/A | N/A | N/A | | |
| **[An Analysis of the Automatic Bug Fixing Performance of ChatGPT](https://arxiv.org/abs/2301.08653)** | **Y** | N | 20/01/2023 | December 2022 - January 2023 | Y | Y | ChatGPT | BROWSER | N | Y | N | N |[QuixBugs](https://github.com/jkoppel/QuixBugs) (100.0% of entire set)|
| **[Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Models](https://arxiv.org/abs/2302.02094)** | **Y** | Y | 04/02/2023 | | Y | N | ChatGPT | BROWSER | N | Y | N | N | |
| **[ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots](https://arxiv.org/abs/2302.06466)** | **Y** | N | 08/02/2023 | | \- | N | ChatGPT | BROWSER | Y | Y | N | N |[QALD-9](https://github.com/ag-sc/QALD/tree/master) (100.0% of test set) [LCQuAD-2.0](http://lc-quad.sda.tech/) (0.3% of entire set) [YAGO-Bench ](https://yago-knowledge.org/downloads/yago-4) (100.0% of entire set) [DBLP-Bench](https://dblp.org/rdf/release/dblp-2022-06-01.nt.gz) (100.0% of entire set) [MAG-Bench](https://zenodo.org/records/4617285#.YrNszNLMJhH) (100.0% of entire set)|
| **[ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models](https://arxiv.org/abs/2302.07257)** | **Y** | N | 14/02/2023 | January 2023 | Y | N | ChatGPT | BROWSER | Y | Y | Y | N |[MIMIC](https://physionet.org/content/mimic-cxr/2.0.0/) (0.5% of test set)|
| **[Guiding Large Language Mdels via Directional Stimulus Prompting](https://arxiv.org/abs/2302.11520)** | **Y** | N | 07/07/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[Utilizing ChatGPT Generated Data to Retrieve Depression Symptoms from Social Media](https://arxiv.org/abs/2307.02313)** | **N/A** | | 05/07/2023 | \- | N/A | \- | \- | \- | N/A | N/A | N/A | | |
| **[Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://arxiv.org/abs/2302.12813)** | **Y** | N | 24/02/2023 | | N/A | N | ChatGPT | BROWSER | Y | N | N/A | N |[DSTC7 Track 2](https://paperswithcode.com/dataset/dstc7-task-2) (100.0% of entire set) [DSTC11 Track 5](https://github.com/alexa/dstc11-track5) (100.0% of val set) [OTT-QA](https://ott-qa.github.io/) (100.0% of entire set)|
| **[ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity?](https://aclanthology.org/2023.acl-short.81/)** | **Y** | Y | 02/06/2023 | January 2023 | Y | Y | ChatGPT | BROWSER | Y | Y | N | N |[MultiWOZ 2.1](https://huggingface.co/datasets/ConvLab/multiwoz21) (13.5% of test set)|
| **[ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://aclanthology.org/2023.acl-srw.1/)** | **Y** | Y | 01/07/2023 | February to April 2023 | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | N | |
| **[Not The End of Story: An Evaluation of ChatGPT-Driven Vulnerability Description Mappings](https://aclanthology.org/2023.findings-acl.229/)** | **Y** | Y | 01/07/2023 | \- | Y | Y | ChatGPT | BROWSER | N | Y | N | N |[CVE-2021](https://cve.mitre.org/) (100.0% of entire set) [CVE-ATT&CK](https://cve.mitre.org/) (100.0% of entire set) [BRON (preprocessed)](https://github.com/ALFA-group/BRON) (100.0% of entire set)|
| **[ChatGPT is not a good indigenous translator](https://aclanthology.org/2023.americasnlp-1.17/)** | **Y** | Y | 01/07/2023 | | N | N | GPT-4 | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | N | |
| **[You’ve Got a Friend in ... a Language Model? A Comparison of Explanations of Multiple-Choice Items of Reading Comprehension between ChatGPT and Humans](https://aclanthology.org/2023.bea-1.30/)** | **Y** | Y | 01/07/2023 | December 2022 and January 2023 | Y | Y | ChatGPT | BROWSER | N | N | N/A | Y |custom - Evaluar para Avanzar (EpA) exams (100.0% of entire set)|
| **[Language-Agnostic Transformers and Assessing ChatGPT-Based Query Rewriting for Multilingual Document-Grounded QA](https://aclanthology.org/2023.dialdoc-1.11/)** | **Y** | Y | 03/05/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | N | |
| **[HW-TSC at SemEval-2023 Task 7: Exploring the Natural Language Inference Capabilities of ChatGPT and Pre-trained Language Model for Clinical Trial](https://aclanthology.org/2023.semeval-1.221/)** | **Y** | Y | 01/07/2023 | January 2023 | Y | N | ChatGPT | BROWSER | N | Y | N | Y |[SemEval-2023 Task 7](https://github.com/HKUST-KnowComp/NLI4CT) (100.0% of entire set)|
| **[Can ChatGPT Understand Causal Language in Science Claims?](https://aclanthology.org/2023.wassa-1.33/)** | **Y** | Y | 01/07/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | N | |
| **[ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models](https://aclanthology.org/2023.wassa-1.29/)** | **N/A** | Y | 02/06/2023 | January to March 2023 | Y | Y | ChatGPT | BROWSER | N/A | N/A | N/A | N | |
| **[Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://aclanthology.org/2023.c3nlp-1.7/)** | **Y** | Y | 30/03/2023 | \- | Y | Y | ChatGPT | BROWSER | Y | N | N/A | Y |[Hofstede Cultural Survey](https://github.com/yongcaoplus/ProbingChatGPT) (100.0% of entire set)|
| **[Evaluating Reading Comprehension Exercises Generated by LLMs: A Showcase of ChatGPT in Education Applications](https://aclanthology.org/2023.bea-1.52/)** | **Y** | Y | 01/07/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | N | |
| **[ChatGPT vs. Crowdsourcing vs. Experts: Annotating Open-Domain Conversations with Speech Functions](https://aclanthology.org/2023.sigdial-1.23/)** | **Y** | Y | 01/09/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | N | |
| **[Leveraging Large Language Models for Automated Dialogue Analysis](https://aclanthology.org/2023.sigdial-1.20/)** | **Y** | Y | 01/09/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | N | |
| **[Breaking the Bank with ChatGPT: Few-Shot Text Classification for Finance](https://aclanthology.org/2023.finnlp-1.7/)** | **Y** | Y | 28/08/2023 | | Y | N | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | N | |
| **[Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction](https://aclanthology.org/2023.bea-1.53.pdf)** | **Y** | Y | 05/06/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | N | N/A | N | |
| **[Credible Without Credit: Domain Experts Assess Generative Language Models](https://aclanthology.org/2023.acl-short.37/)** | **Y** | Y | 01/07/2023 | \- | Y | Y | ChatGPT | BROWSER | Y | N | N/A | Y |custom qa (100.0% of entire set)|
| **[SUT at SemEval-2023 Task 1: Prompt Generation for Visual Word Sense Disambiguation](https://aclanthology.org/2023.semeval-1.298/)** | **N/A** | Y | 01/07/2023 | | N/A | N/A | | \- | N/A | N/A | N/A | N/A | |
| **[How Well Do Large Language Models Perform on Faux Pas Tests?](https://aclanthology.org/2023.findings-acl.663/)** | **Y** | Y | 24/05/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | | |
| **[Zero-Shot Information Extraction for Clinical Meta-Analysis using Large Language Models](https://aclanthology.org/2023.bionlp-1.37/)** | **Y** | Y | 01/07/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | N | |
| **[Can Large Language Models Be an Alternative to Human Evaluation?](https://aclanthology.org/2023.acl-long.870.pdf)** | **Y** | Y | 03/05/2023 | December 2022 - January 2023 | Y | Y | ChatGPT | BROWSER | Y | Y | N | N |[WritingPrompts](https://www.kaggle.com/datasets/ratthachat/writing-prompts) (1.3% of test set)|
| **[RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs](https://aclanthology.org/2023.acl-long.427/)** | **N/A** | Y | 15/05/2023 | | N/A | N/A | | | N/A | N/A | N/A | N/A | |
| **[Toward Human-Like Evaluation for Natural Language Generation with Error Analysis](https://aclanthology.org/2023.acl-long.324.pdf)** | **N/A** | Y | 20/12/2022 | | N/A | N/A | | | N/A | N/A | N/A | N/A | |
| **[Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text](https://aclanthology.org/2023.acl-long.403.pdf)** | **N/A** | Y | 21/11/2022 | | N/A | N/A | | | N/A | N/A | N/A | N/A | |
| **[MolXPT: Wrapping Molecules with Text for Generative Pre-training](https://aclanthology.org/2023.acl-short.138/)** | **N/A** | Y | 18/05/2023 | | N/A | N/A | | | N/A | N/A | N/A | N/A | |
| **[MUX-PLMs: Pre-training Language Models with Data Multiplexing](https://aclanthology.org/2023.repl4nlp-1.17/)** | **N/A** | Y | 24/02/2023 | | N/A | N/A | | | N/A | N/A | N/A | N/A | |
| **[Is GPT-3 a Good Data Annotator?](https://aclanthology.org/2023.acl-long.626.pdf)** | ** ** | Y | 20/12/2022 | | Y | N | Both | BROWSER | \- | | | | |
| **[Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://aclanthology.org/2023.eacl-main.241/)** | **N/A** | Y | 14/10/2022 | | N/A | N/A | | | N/A | N/A | N/A | N/A | |
| **[Detoxifying Online Discourse: A Guided Response Generation Approach for Reducing Toxicity in User-Generated Text](https://aclanthology.org/2023.sicon-1.2/)** | **Y** | Y | 01/07/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | Y | |
| **[Faithful Question Answering with Monte-Carlo Planning](https://aclanthology.org/2023.acl-long.218/)** | **Y** | Y | 04/05/2023 | December 2022 | Y | Y | ChatGPT | BROWSER | Y | Y | Y | N |[EntailmentBank](https://allenai.org/data/entailmentbank) (15.0% of test set)|
| **[Nut-cracking Sledgehammers: Prioritizing Target Language Data over Bigger Language Models for Cross-Lingual Metaphor Detection](https://aclanthology.org/2023.clasp-1.12/)** | **Y** | Y | 01/09/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | N | |
| **[Generating Faithful Text From a Knowledge Graph with Noisy Reference Text](https://aclanthology.org/2023.inlg-main.8/)** | **Y** | Y | 12/8/2023 | | Y | | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | Y | |
| **[Empowering Conversational Agents using Semantic In-Context Learning](https://aclanthology.org/2023.bea-1.62/)** | **Y** | Y | 01/07/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | N | |
| **[Can Large Language Models Safely Address Patient Questions Following Cataract Surgery?](https://aclanthology.org/2023.clinicalnlp-1.17/)** | **Y** | Y | 01/07/2023 | December 2022 | Y | N | ChatGPT | BROWSER | N | N | N | Y |custom-sourced post-operations queries from patients (100.0% of entire set)|
| **[Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models](https://aclanthology.org/2023.acl-long.828/)** | **Y** | Y | 15/06/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | Y | N | |
| **[Generative Pretrained Transformers for Emotion Detection in a Code-Switching Setting](https://aclanthology.org/2023.wassa-1.61/)** | **Y** | Y | 01/07/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | N | |
| **[On the Underspecification of Situations in Open-domain Conversational Datasets](https://aclanthology.org/2023.nlp4convai-1.2/)** | **Y** | Y | 01/07/2023 | | Y | N | ChatGPT | BROWSER | \- | N/A | N/A | | |
| **[Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors](https://aclanthology.org/2023.acl-long.650/)** | **Y** | Y | 25/05/2023 | \- | Y | Y | ChatGPT | UNCLEAR | \- | | | | |
| **[Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://aclanthology.org/2023.wassa-1.19/)** | **N/A** | Y | 03/06/2023 | | | | | | \- | | | | |
| **[Good Data, Large Data, or No Data? Comparing Three Approaches in Developing Research Aspect Classifiers for Biomedical Papers](https://aclanthology.org/2023.bionlp-1.8/)** | **Y** | Y | 07/06/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | N | |
| **[IDOL: Indicator-oriented Logic Pre-training for Logical Reasoning](https://aclanthology.org/2023.findings-acl.513/)** | **Y** | Y | 27/06/2023 | February 2023 | Y | Y | ChatGPT | BROWSER | Y | Y | N | N |[ReClor](https://whyu.me/reclor/) (0.4% of entire set) [LogiQA](https://github.com/lgw863/LogiQA-dataset) (0.3% of entire set)|
| **[PrecogIIITH@WASSA2023: Emotion Detection for Urdu-English Code-mixed Text](https://aclanthology.org/2023.wassa-1.58/)** | **Y** | Y | 01/07/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | N | |
| **[Do PLMs Know and Understand Ontological Knowledge?](https://aclanthology.org/2023.acl-long.173/)** | **Y** | Y | 01/07/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | Y | |
| **[Evaluation of Question Generation Needs More References](https://aclanthology.org/2023.findings-acl.396/)** | **Y** | Y | 26/05/2023 | | Y | N | ChatGPT | BROWSER | N | Y | N | N |[Quiz_Design](https://github.com/salesforce/QGen/tree/main/Quiz_Design) (100.0% of entire set)|
| **[A System for Answering Simple Questions in Multiple Languages](https://aclanthology.org/2023.acl-demo.51/)** | **Y** | Y | 01/07/2023 | March 2023 version of ChatGPT | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | N | |
| **[HIT-SCIR at WASSA 2023: Empathy and Emotion Analysis at the Utterance-Level and the Essay-Level](https://aclanthology.org/2023.wassa-1.54/)** | **Y** | Y | 01/07/2023 | | Y | N | ChatGPT | UNCLEAR | \- | N/A | N/A | N | |
| **[Pre-trained Language Models Can be Fully Zero-Shot Learners](https://aclanthology.org/2023.acl-long.869/)** | **Y** | Y | 26/05/2023 | March 2023 | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | N | |
| **[Predicting the Quality of Revisions in Argumentative Writing](https://aclanthology.org/2023.bea-1.24/)** | **Y** | Y | 01/06/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | N | N/A | Y | |
| **[Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods](https://aclanthology.org/2023.bea-1.18/)** | **N/A** | Y | 29/05/2023 | | | | | | \- | | | | |
| **[When Truth Matters – Addressing Pragmatic Categories in Natural Language Inference (NLI) by Large Language Models (LLMs)](https://aclanthology.org/2023.starsem-1.4/)** | **Y** | Y | 01/07/2023 | December 2022 | Y | Y | ChatGPT | BROWSER | Y | N | N/A | N |[MNLI](https://huggingface.co/datasets/multi_nli) (0.4% of val set)|
| **[Debiasing should be Good and Bad: Measuring the Consistency of Debiasing Techniques in Language Models](https://aclanthology.org/2023.findings-acl.280/)** | **N/A** | Y | 23/05/2023 | | | | | | \- | | | | |
| **[GPoeT: a Language Model Trained for Rhyme Generation on Synthetic Data](https://aclanthology.org/2023.latechclfl-1.2/)** | **N/A** | Y | 01/05/2023 | | | | | | \- | | | | |
| **[Examining Bias in Opinion Summarisation Through the Perspective of Opinion Diversity](https://aclanthology.org/2023.wassa-1.14/)** | **Y** | Y | 01/07/2023 | March 2023 version of ChatGPT | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | N | |
| **[Unsupervised Summarization Re-ranking](https://aclanthology.org/2023.findings-acl.529/)** | **Y** | Y | 14/05/2023 | \- | N | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | Y | Y | N | N | |
| **[Improving Dutch Vaccine Hesitancy Monitoring via Multi-Label Data Augmentation with GPT-3.5](https://aclanthology.org/2023.wassa-1.23/)** | **Y** | Y | 01/07/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N/A | N | N/A | Y | |
| **[What Makes a Good Counter-Stereotype? Evaluating Strategies for Automated Responses to Stereotypical Text](https://aclanthology.org/2023.sicon-1.4/)** | **Y** | Y | 01/07/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N/A | N | N/A | Y | |
| **[What Makes Good Counterspeech? A Comparison of Generation Approaches and Evaluation Metrics](https://aclanthology.org/2023.cs4oa-1.5/)** | **Y** | Y | 01/09/2023 | May 2023 | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | Y | |
| **[DAMO-NLP at SemEval-2023 Task 2: A Unified Retrieval-augmented System for Multilingual Named Entity Recognition](https://aclanthology.org/2023.semeval-1.277/)** | **Y** | Y | 05/05/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | N | |
| **[UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?](https://aclanthology.org/2023.clinicalnlp-1.49/)** | **Y** | Y | 29/06/2023 | \- | Y | Y | Both | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | N | |
| **[Frontier Review of Multimodal AI](https://aclanthology.org/2023.ccl-2.9.pdf)** | **N/A** | Y | 03/08/2023 | | N/A | N/A | \- | \- | | | | | |
| **[Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation](https://arxiv.org/abs/2304.01746)** | **Y** | N | 04/04/2023 | | Y | N | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | N | Y | N | N | |
| **[DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI](https://arxiv.org/abs/2307.10172)** | **Y** | N | 19/07/2023 | \- | Y | Y | ChatGPT | API - POST MARCH 2023 OR OPT-OUT | \- | N/A | N/A | N | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |
| ** ** | ** ** | | | | | | | | \- | | | | |

## Acknowledgements
This work was co-funded by the European Union (ERC, NG-NLG, 101039303).

leak-llm.github.io

leak-llm.github.io 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

leak-llm.github.io

leak-llm.github.io 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README