KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #codex #typescript #ai-agents #ai-agent #cli #claude-code #open-source #deepseek-harness #dsh-plugin #python #mcp

COLING-2025-Workshop-on-MGT-Detection-Task-3

★ 10 Open GitHub ↗

Repository for submission to the COLING GenAI Content Detection Shared Task 3: "Cross-domain Machine-Generated Text Detection"

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Task 3: Cross-Domain Machine-Generated Text Detection

When applying detectors to machine-generated text in the wild, the dominant emerging paradigm is to use an **open-domain API-based** detector. However, many commonly used detectors exhibit poor cross-domain and cross-model robustness. Thus, it is critical to train our detectors to be able to handle text from many domains with both high accuracy and low false positive rates.

In the COLING Workshop on MGT Detection Task 3, we focus on **cross-domain robustness** of detectors by testing submissions on the [RAID benchmark](https://raid-bench.xyz/). We adopt the same straightforward binary problem formulation as Task 1, that is: given a piece of text, determine whether it is generated by a machine or authored by a human.

However, in this task the texts will **not be limited to any one domain** and may come from any one of 8 different domains, 11 generative models, and 4 decoding strategies. Your goal will be to create a detector that exhibits high levels of robustness across all of these models, domains, and decoding strategies while maintaining a low false positive rate.

Our domains are:

There are two subtasks:
- Subtask A: Non-Adversarial Cross-Domain MGT detection.
- Subtask B: Adversarial Cross-Domain MGT detection.

## <a name="news"></a>NEWS

### 18 Sep 2024

We have released our instructions and training set.

### 24 Sep 2024

We have released our format checking script.

### 22 Oct 2024

The deadline for the competition has been extended to Nov 2nd 2024.

### 2 Nov 2024

The competition phase has ended and the evaluation phase has begun.

### 6 Nov 2024

The evaluation phase has ended and the [leaderboard](https://raid-bench.xyz/shared-task) is now live!

### 15 Jan 2025

The summary [paper](https://arxiv.org/abs/2501.08913) for the shared task is complete!

## <a name="competition"></a>Competition

The competition will be held on the [RAID Website](https://raid-bench.xyz/). We will be releasing a separate leaderboard specifically for the shared task that will exist alongside the main RAID leaderboard and will be populated with results after the task finishes.

To submit to the shared task, you must first get predictions for your detector on the test set. Please consult the [RAID Leaderboard Submission Instructions](https://github.com/liamdugan/raid?tab=readme-ov-file#leaderboard-submission) for more details on how to get the `predictions.json` file for your detector.

After you have the `predictions.json` file you must then write a metadata file for your submission. Your metadata file should use the template found in this repository at `submissions/template-metadata.json`.

Finally, fork this repository. Add your files to `submissions/YOUR-DETECTOR-NAME/predictions.json` and your metadata file to `submissions/YOUR-DETECTOR-NAME/metadata.json` and make a pull request to this repository. We have provided an example submission of the OpenAI Roberta Large classifier under `submissions/openai-roberta-large`.

> [!NOTE]
> Please **DO NOT SUBMIT** to the main RAID leaderboard during the duration of the shared task. If you do so, **you will be disqualified.**

## <a name="dataset"></a>Dataset
For this task we will be using the RAID dataset.
**Download RAID** by consulting the [RAID Github Repository](https://github.com/liamdugan/raid?tab=readme-ov-file#download-raid).

## <a name="important_dates"></a>Important Dates

- 18th September, 2024: Training & test set release
- **2nd November, 2024** ~~25th October, 2024~~: Submission phase closes
- **5th November, 2024** ~~28th October, 2024~~: Leaderboard to be public
- 15th November, 2024: System description paper submission

## <a name="data_format"></a>Prediction File Format and Format Checkers

In order to run our automatic evaluation, your submission must include a file named `predictions.json`.
This file should be valid JSON and should be of the following format:
```
[
  {"id": "64005577-3d63-4583-8945-7541d3e53e7d", "score": 0.0021110873541056},
  {"id": "c2b9df67-4e29-45ca-bdcc-7065fb907b77", "score": 0.9116235922302712},
  ...
]
```
The provided `run_detection` function from the [RAID Pypi package](https://pypi.org/project/raid-bench/) will output predictions in this format. 
If you would like to use your own code, you can run something like the below snippet to output in the correct format.
```py
with open(output_path, "w") as f:
  json.dump(df[["id", "score"]].to_dict(orient="records"), f)
```

To check your submission's correctness please run our provided format checker as follows:
```
$ python format_check.py --results_path <your_file>.json
```

## <a name="scorer_and_official_evaluation_metrics"></a>Scorer and Official Evaluation Metrics

The **official evaluation metric** is **TPR @ FPR=5%**. That is, the accuracy of the model on detecting machine-generated text at a fixed false positive rate of 5%.
To calculate this, our scorer uses the model predictions on human data to search a classification threshold that results in 5% FPR for each domain.

To run the scorer, first run `pip install raid-bench` then use the RAID pip package as follows:

```py
from raid import run_detection, run_evaluation
from raid.utils import load_data

# Define your detector function
def my_detector(texts: list[str]) -> list[float]:
    pass

# Download & Load the RAID dataset
train_df = load_data(split="train")

# Run your detector on the dataset
predictions = run_detection(my_detector, train_df)

# Evaluate your detector predictions
evaluation_result = run_evaluation(predictions, train_df)
```

## <a name="baselines"></a>Baselines

We have run a number of publicly available open-source detectors on RAID. [Binoculars](https://arxiv.org/abs/2401.12070) gets **79.0%**, [RADAR](https://huggingface.co/TrustSafeAI/RADAR-Vicuna-7B) gets **65.6%**, and [roberta-base-openai-detector](https://huggingface.co/openai-community/roberta-base-openai-detector) gets **59.2%** on the non-adversarial RAID test set.

We will also be releasing some simple baseline trained models on the RAID dataset. These will be released shortly.

## <a name="contact"></a>Contact

Website: [https://genai-content-detection.gitlab.io](https://genai-content-detection.gitlab.io)  
Email: genai-content-detection@googlegroups.com or directly to ldugan@seas.upenn.edu

## <a name="citation"></a>Citation

If you use our code or findings in your research, please cite us as:

```
@misc{dugan2025genaicontentdetectiontask,
      title={GenAI Content Detection Task 3: Cross-Domain Machine-Generated Text Detection Challenge}, 
      author={Liam Dugan and Andrew Zhu and Firoj Alam and Preslav Nakov and Marianna Apidianaki and Chris Callison-Burch},
      year={2025},
      eprint={2501.08913},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.08913}, 
}
```

COLING-2025-Workshop-on-MGT-Detection-Task-3

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README