LitMT_eval

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 12 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# [How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs](https://arxiv.org/abs/2410.18697)

## Quick Links
- [LitEval-Corpus](#)
    - paragraph-level parallel corpus with verified high-quality human translations of both classic and contemporary published works; 9 MT systems including the GPT-4o, commercial models (DeepL and Google Translate), popular LLMs of various sizes (7-8b Llama 3, TowerInstruct, Qwen, Gemma), and previous SOTA MT systems (M2M, NLLB). This dataset is intended for academic purposes only. The use of this dataset requires agreement to the conditions listed in the [form](https://forms.gle/tGi64MBt59HL4QBQ7). The dataset will be available for downloading after filling out the form.

- [Code for MT generation and evaluation](codes)

- [Meta information for source and translation](meta): the link indicates version information. The actual content may be acquired elsewhere, e.g., from PAR3 or via purchase. 
  
 #### News:
 - 29/Oct/2024: We release student annotation and metric evaluation datasets.
 - 25/02/2025: Codes to reproduce main findings and meta information

## LitEval-Corpus:

### Schemes Overview
![Scheme summary](https://raw.githubusercontent.com/zhangr2021/LitMT_eval/main/image/scheme_comparison.png)

### Example
![<img align="right" width="400">](https://raw.githubusercontent.com/zhangr2021/LitMT_eval/main/image/exmaple_deen.png)

## BibTex Citation

```
@inproceedings{zhang-etal-2025-good,
    title = "How Good Are {LLM}s for Literary Translation, Really? Literary Translation Evaluation with Humans and {LLM}s",
    author = "Zhang, Ran  and
      Zhao, Wei  and
      Eger, Steffen",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.naacl-long.548/",
    doi = "10.18653/v1/2025.naacl-long.548",
    pages = "10961--10988",
    ISBN = "979-8-89176-189-6"
}
```

LitMT_eval

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README