WeLT

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #golang #android #machine-learning #deep-learning #linux #docker #macos #nodejs #java #vue

(★ 14)

Language modeling that treats text as images, leveraging visual structure for enhanced understanding.

WeLT 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🌎 WeLT: Word Embedding Latent Transformer

![Python](https://img.shields.io/badge/python-3.12-blue)
[![License](https://img.shields.io/badge/license-MIT-green)](https://raw.githubusercontent.com/sign/WeLT/main/LICENSE)

[See Motivational Examples](./MOTIVATION.md)

We present a language model that replaces conventional subword tokenization with a dual representation of text as
both bytes and rendered images of "words", allowing the model to directly process the visual and symbolic
structure of language. It segments text into whitespace-delimited units, encodes each as images and byte sequences,
and passes them through an encoder–decoder pipeline:
a bytes encoder, an image encoder, a large latent transformer, and a bytes decoder.
At inference, predicted bytes are rendered back into images, closing the loop for the next prediction.
This design could make learning and inference cheaper and faster for non-English languages,
since the heavy latent transformer only predicts high-level token representations while the actual byte
sequences are generated by a much smaller decoder.

![Model Architecture](https://raw.githubusercontent.com/sign/WeLT/main/assets/architecture.png)

## Quick Start

Clone and setup:

```shell
git clone https://github.com/sign/WeLT.git
cd WeLT
```

Build the environment using conda:

```shell
conda env create -f environment.yml -y
conda activate welt
pip install ".[dev,train]"
```

Or using docker:

```shell
docker build -t welt .

# Run an interactive shell inside the container
docker run -it --rm --gpus all \
  --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$(pwd)/welt:/app/welt" \
  -v "$(pwd)/welt_training:/app/welt_training" \
  welt /bin/bash

# Run a training job
docker run -it --rm --gpus all \
  --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$(pwd)/welt:/app/welt" \
  -v "$(pwd)/welt_training:/app/welt_training" \
  -v /shared/.cache:/root/.cache \
  -v ~/.netrc:/root/.netrc:ro \
  -e WANDB_PROJECT="ocr" \
  -e CONFIG="welt_training/experiments/easy-tasks/ocr.yaml" \
  welt

# Or without WANDB logging
docker run -it --rm --gpus all \
  --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$(pwd)/welt:/app/welt" \
  -v "$(pwd)/welt_training:/app/welt_training" \
  -v /shared/.cache/huggingface:/root/.cache/huggingface \
  -e WANDB_MODE="offline" \
  -e CONFIG="welt_training/experiments/easy-tasks/string-repetition.yaml" \
  welt
```

> [!TIP]
> Run tests using `pytest` to ensure everything is working correctly.

## Model Setup

- **Bytes Encoder** - You can use any language model as the bytes encoder (causal or masked).
- **Image Encoder** - You can use any image encoder.
- **Latent Transformer** - You can use any causal LM (recommended: large).
- **Bytes Decoder** - You can use any causal LM (recommended: small).

For language models, the parameter count is lower than reported, due to removing the embedding layers.

Our implementation allows for any mix-and-match. Some example setups are:

| Name   | Bytes Encoder                                                                | Image Encoder                                                                                             | Latent Transformer                                                    | Bytes Decoder                                                             | Total Parameters |
|--------|------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|---------------------------------------------------------------------------|------------------|
| tiny   | [bert-tiny](https://huggingface.co/prajjwal1/bert-tiny) (0.5m)               | [vit-tiny-patch16-224](https://huggingface.co/WinKawaks/vit-tiny-patch16-224) (5m)                        | [pythia-70m](https://huggingface.co/EleutherAI/pythia-70m) (19m)      | [tiny-lm](sbintuitions/tiny-lm) (3m)                                      | 28m              |
| small  | [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) (111m) | [swinv2-tiny-patch4-window16-256](https://huggingface.co/microsoft/swinv2-tiny-patch4-window16-256) (27m) | [gemma-3-270m](https://huggingface.co/google/gemma-3-270m) (100m)     | [SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) (106m)  | 346m             |
| medium | [deberta-v3-large](https://huggingface.co/microsoft/deberta-v3-large) (303m) | [clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32) (87m)                        | [Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B) (973m) | [gpt2-medium](https://huggingface.co/openai-community/gpt2-medium) (304m) | 1,674m           |

To turn off bytes encoding, set `bytes_encoder=False`, and similarly for images, set `image_encoder=False`.
You can also turn off a specific encoder after training has completed, for testing purposes.

> [!WARNING]
> In implementation of the bytes decoder, we concatenate the embeddings of the bytes of the current token with
> all the embeddings of the previous tokens (on the word level). This is done since not all causal LMs support
> cross-attention, and so we want to avoid using it, and rely on the self-attention mechanism instead.

## Data Preparation

You can prepare datasets offline using the `welt-prepare-data` CLI.
It streams a HuggingFace dataset, samples raw text with unit-based limits, and writes sharded `.jsonl.gz` files:

```shell
welt-prepare-data \
    --dataset_name HuggingFaceFW/fineweb \
    --dataset_config sample-10BT \
    --train_split_units 3200000000 \
    --validation_split_units 100000000 \
    --num_units_per_file 100000000 \
    --max_seq_length 512 \
    --seed 42 \
    --output_path /scratch/data/pretrain
```

Multiple datasets can be prepared into the same output directory:

```shell
welt-prepare-data \
    --dataset_name monology/pile-uncopyrighted \
    --train_split_units 1000000000 \
    --validation_split_units 50000000 \
    --num_units_per_file 100000000 \
    --max_seq_length 512 \
    --output_path /scratch/data/pretrain
```

The output directory contains sharded `.jsonl.gz` files and a `{prefix}-metadata.json` per dataset:

```
/scratch/data/pretrain/
├── fineweb-sample-10BT-00000000.jsonl.gz
├── fineweb-sample-10BT-00000001.jsonl.gz
├── fineweb-sample-10BT-metadata.json
├── pile-uncopyrighted-00000000.jsonl.gz
└── pile-uncopyrighted-metadata.json
```

Then train using the prepared data:

```shell
welt-train config.yaml --prepared_data_path /scratch/data/pretrain
```

| Argument | Description |
|----------|-------------|
| `--dataset_name` | HuggingFace dataset identifier (required) |
| `--dataset_config` | Dataset config name (optional) |
| `--dataset_split` | Split to use (default: "train") |
| `--text_column` | Column containing text (default: "text") |
| `--text_template` | Python format string template (optional) |
| `--language` | Language tag to store with each example (e.g., "eng_Latn") |
| `--unit_type` | Unit type for counting: "words" or "chars" (default: "words") |
| `--train_split_units` | Number of units for the train split (default: 0, no train shards) |
| `--validation_split_units` | Number of units for the validation split (default: 0, no validation shards) |
| `--num_units_per_file` | Max units per shard file (optional) |
| `--max_seq_length` | Max words per example; splits long documents using word segmentation |
| `--max_bytes_per_word` | Max UTF-8 bytes per word; should match training config `max_word_length - 2` (default: 126) |
| `--seed` | Random seed for shuffling |
| `--drop_remainder` | Drop partial chunks at document boundaries |
| `--output_path` | Output directory path (required) |

### Verifying Prepared Data

After preparing data, verify integrity with `welt-verify-data`:

```shell
welt-verify-data --data_path /scratch/data/pretrain
```

This checks that shard counts and example counts match the metadata, and warns if train/validation splits from the same source were created separately (risking data contamination).

## Training

Training instructions are available in the [welt_training/README.md](./welt_training/README.md).
There, you can select the model architectures you want to use for each component, and the dataset you want to train on.

## Inference

Since we have two decoders, the autoregressive prediction logic is a bit more complex than the usual,
and supporting decoding algorithms like beam-search is not trivial.

Thus, on the latent-transformer level,
[we only support greedy decoding](https://github.com/sign/WeLT/issues/5) for now.
On the bytes decoder level, we support all classical decoding algorithms supported by HuggingFace Transformers.

## Contributing

See [open issues](https://github.com/search?q=repo%3Asign%2FWeLT+%22%2Fissues%2F%22&type=code)
and [TODOs](https://github.com/search?q=repo%3Asign%2FWeLT%20TODO&type=code) in the codebase.

During the creation of this repository, we created several others to support it:
- [`sign/words-segmentation`](https://github.com/sign/words-segmentation) as a universal word level pretokenizer.
- [`sign/utf8-tokenizer`](https://github.com/sign/utf8-tokenizer) as a robust byte-level tokenizer.
- [`sign/pixel-renderer`](https://github.com/sign/pixel-renderer) as a reproducible text-to-image renderer.

> [!WARNING]
> Training runs are experimental until core issues are resolved.

## Cite

If you use this code in your research, please consider citing the work:

```bibtex
@misc{moryossef2025welt,
  title={{WeLT}: Word Embedding Latent Transformer for Equitable Modeling of the Languages of the World},
  author={Moryossef, Amit},
  howpublished={\url{https://github.com/sign/WeLT}},
  year={2025}
}
```

WeLT

WeLT 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

WeLT

WeLT 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README