KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agent #developer-tools #codex #typescript #cli #deepseek-harness #claude-code #ai-agents #mcp #rust

tokenize-uk

★ 70 Open GitHub ↗

Simple python lib to tokenize texts into sentences and sentences to words. Small, fast and robust. Comes with ukrainian flavour

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# tokenize-uk

Ukrainian tokenization: paragraphs → sentences → words. Small, fast and
robust, now with LanguageTool-grade engines.

Since 2.0 the default engines are ports of what
[LanguageTool](https://languagetool.org) uses for Ukrainian, both
verified **byte-identical** to their Java originals on multi-million-token
real-world corpora:

- **words**: a port of LanguageTool's `UkrainianWordTokenizer` (Andriy
  Rysin's abbreviation-aware tokenizer) — verified on 3.87M tokens across
  four corpora, zero differences, faster than the Java original;
- **sentences**: [choppa-srx](https://pypi.org/project/choppa-srx/), the
  Python port of the Java `segment` SRX library with LanguageTool's rules.

The original 2016 regex engines are preserved and one argument away.

## Quick Start

```bash
pip install tokenize_uk
```

```python
import tokenize_uk

tokenize_uk.tokenize_words("Це проф. Артюхов.")
# ['Це', 'проф.', 'Артюхов', '.']

tokenize_uk.tokenize_sents("Це проф. Артюхов. Він приїхав у м. Київ.")
# ['Це проф. Артюхов.', 'Він приїхав у м. Київ.']

tokenize_uk.tokenize_text("Перший абзац. Ще речення.\nДругий абзац.")
# [[['Перший', 'абзац', '.'], ['Ще', 'речення', '.']], [['Другий', 'абзац', '.']]]
```

Command line:

```bash
echo "Це проф. Артюхов. Він приїхав у м. Київ." | tokenize-uk -l sents
```

## The two engines

Every function keeps the pre-2.0 signature and return shape. If the new
behavior breaks your pipeline, fall back per call:

```python
tokenize_uk.tokenize_words(text, legacy=True)   # 2016 regex engine
```

or import the old engines directly from `tokenize_uk.legacy`. The legacy
code is preserved byte-for-byte (regexes included) and covered by the
original test fixtures.

Differences you will notice with the default engine:

- abbreviations keep their dots and don't split (`проф.`, `т.`, `зв.`,
  `чл.-кор.`), and don't end sentences where they shouldn't;
- dates (`12.03.2022`), times (`15:30`), decimals (`10,5`) and web
  entities (`Цензор.НЕТ`) stay single tokens;
- sentence splitting follows LanguageTool's `segment.srx` rules instead
  of a punctuation-plus-uppercase heuristic.

For raw LanguageTool-compatible word tokens (including whitespace
tokens), use the class directly:

```python
from tokenize_uk import UkrainianWordTokenizer
UkrainianWordTokenizer().tokenize("а б")
# ['а', ' ', 'б']
```

## Using with spaCy

```bash
pip install tokenize_uk[spacy]
```

```python
from tokenize_uk.spacy import blank_pipeline

nlp = blank_pipeline()
doc = nlp("Це проф. Артюхов. Він приїхав у м. Київ.")
[t.text for t in doc]              # LT-grade word tokens
[sent.text for sent in doc.sents]  # LT-grade sentences
```

`doc.text` round-trips the input exactly, token offsets are exact, and
token text is taken verbatim from the input (unlike `tokenize_words`,
which canonicalizes a few apostrophe/quote characters).
`nlp.to_disk()`/`spacy.load()` work (the tokenizer is a registered
spaCy factory). You can also drop `UkrainianTokenizer` or the
`tokenize_uk_sentencizer` component into an existing pipeline — with
the caveat that pretrained statistical components were trained on
spaCy's own tokenization.

## Verification and performance

The word tokenizer is compared byte-for-byte against LanguageTool
master's `UkrainianWordTokenizer` (compiled from source; harness in
`scripts/java-harness/`):

| corpus | lines | tokens | Java | tokenize-uk 2.0 | output |
|---|---|---|---|---|---|
| Militarny news | 100,004 | 1,736,362 | 7.4 s | 5.4 s | identical |
| uanews.dp.ua | 133,120 | 1,973,203 | 8.9 s | 7.8 s | identical |
| Liga.net | 5,982 | 69,695 | 0.5 s | 0.2 s | identical |

The sentence layer's own verification (byte-identity with the Java
segment library on ~136k segments) is documented in
[choppa's README](https://github.com/lang-uk/choppa#performance-and-verification).

Reproduce with `scripts/benchmark.py` (see `docs/design.md`).

## Documentation

- [API](docs/api.md) — functions, shapes, engines, CLI
- [Design](docs/design.md) — how the LT port works and how it's verified
- [History](docs/history.md) — 2016 origins, the LanguageTool ports

## Copyrights and kudos

- Vsevolod Dyomkin, [Dmytro Chaplynskyi](https://github.com/dchaplinsky) —
  original library, [lang-uk](https://lang.org.ua) project
- Andriy Rysin and the LanguageTool team — the original
  `UkrainianWordTokenizer` and the Ukrainian SRX rules
- [Jarek Lipski](https://github.com/loomchild) — the segment library
  behind the sentence layer

MIT licensed.

## Як цитувати

Якщо ви використовуєте **tokenize-uk: Ukrainian tokenizer**, будь ласка, процитуйте статтю, а не посилання на репозиторій — це єдиний спосіб, у який внесок стає видимим у наукометрії.

> Chaplynskyi, D. (2023). Introducing UberText 2.0: A Corpus of Modern Ukrainian at Scale. In Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP), pages 1–10. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.unlp-1.1

```bibtex
@inproceedings{chaplynskyi-2023-introducing,
    title     = "Introducing {U}ber{T}ext 2.0: A Corpus of {M}odern {U}krainian at Scale",
    author    = "Chaplynskyi, Dmytro",
    editor    = "Romanyshyn, Mariana",
    booktitle = "Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP)",
    month     = may,
    year      = "2023",
    address   = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url       = "https://aclanthology.org/2023.unlp-1.1/",
    doi       = "10.18653/v1/2023.unlp-1.1",
    pages     = "1--10"
}
```

tokenize-uk

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README