slop-detector

KO | EN

gitlite — search

#hacktoberfest #react #typescript #golang #machine-learning #rust #deep-learning #macos #awesome #awesome-list #vue #security #pytorch #openai

(★ 60)

End-to-end implementation of an AI writing detector

slop-detector 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# AI Slop Detector
An end-to-end implementation of an AI writing detector

## Data
To train our classifier, we needed many examples of human-written text and AI-generated text.

### Human-written text
We gathered examples of human-written text by crawling and scraping Wikipedia.
- For each page seen, two random paragraphs were stored
- 10,001 paragraphs were stored in total
- The code for crawling and scraping is contained in `/scraping`
- The result is in `/scrapes`.

### AI-generated text
AI text was then generated by prompting a model to rewrite each paragraph from Wikipedia.
- This was done in two passes (1. summarize, 2. rewrite) with separate contexts, to accurately model real-world generation
- GPT 5 Nano was used because it is cheap, sufficient, and a member of the most popular model family
- The Batch API was used because it suited our use case
- The code for data generation is contained in `/data-generation`

### Full dataset
The final dataset with pairs of human-written and AI-generated text is available on [HuggingFace](https://huggingface.co/datasets/gouwsxander/wikipedia-human-ai).

It can also be found at `/data/wikipedia.jsonl` in this repository.

## Model
Our model is a LoRA adapter of `bert-base-cased`, with a classification head.
- Validation accuracy: 96%
- Training was done with HuggingFace
- The code for training is contained in `/training`

### Usage
The final model weights are available on [HuggingFace](https://huggingface.co/gouwsxander/slop-detector-bert).

It can be used as so:
```python
from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="gouwsxander/slop-detector-bert",
)

inputs = ...
results = classifier(inputs)
```

Texts with predicted label `LABEL_1` are estimated to be produced by an AI.

You may get a warning about some weights being initialized, but these can be ignored.

## Benchmark Performance and Limitations
We test our model on the [RAID benchmark](https://github.com/liamdugan/raid) training set to get a sense of the model's limitations.

### RAID Benchmark Performance
Our model performed better than random guessing across all model families and most domains, indicating that general, transferable properties of AI-generated writing were learned.

The best TPR@FPR=1% measures were seen for IMDB movie reviews, Wikipedia, and arXiv abstracts.
- The TPR@FPR=1% for reviews was about 90%
- The TPR@FPR=1% for Wikipedia, arXiv abstracts, and news articles was between 30% and 40%
- The TPR@FPR=1% for book excerpts, recipes, poetry, and Reddit posts was between 1% and 20%

Text generated by LLaMA was the most likely to be detected.

The code and results for this analysis is contained in `/raid`.

### Limitations
This model has several important limitations:
- Our dataset contains only Wikipedia-style text
- Our dataset contains only generations from GPT 5 Nano
- We assume Wikipedia is currently entirely human-generated
- The model has a context window of 512 tokens
- It was trained on one paragraph of text at a time

slop-detector

slop-detector 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

slop-detector

slop-detector 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README