ai21-tokenizer

KO | EN

gitlite — search

#javascript #llm #go #docker #java #windows #awesome #cli #kubernetes #awesome-list #pytorch #openai #mcp #open-source

(★ 34)

AI21's Jamba models tokenizers

ai21-tokenizer Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">
    <a href="https://github.com/AI21Labs/ai21-tokenizer">AI21 Labs Tokenizer</a>
</h1>

<p align="center">
    <em>A SentencePiece based tokenizer for production uses with AI21's models</em>
</p>

---

## Prerequisites

- If you wish to use the tokenizers for `Jamba Mini` or `Jamba Large`, you will need to request access to the relevant model's HuggingFace repo:
  - [Jamba Mini](https://huggingface.co/ai21labs/AI21-Jamba-Mini-1.6)
  - [Jamba Large](https://huggingface.co/ai21labs/AI21-Jamba-Large-1.6)

## Installation

### pip

```bash
pip install ai21-tokenizer
```

### poetry

```bash
poetry add ai21-tokenizer
```

## Usage

### Basic Usage

```python
from ai21_tokenizer import Tokenizer

# Create tokenizer (defaults to Jamba Mini)
tokenizer = Tokenizer.get_tokenizer()

# Encode text to token IDs
text = "Hello, world!"
encoded = tokenizer.encode(text)
print(f"Encoded: {encoded}")

# Decode token IDs back to text
decoded = tokenizer.decode(encoded)
print(f"Decoded: {decoded}")
```

### Specific Tokenizer Selection

```python
from ai21_tokenizer import Tokenizer, PreTrainedTokenizers

# Jamba Mini tokenizer
tokenizer = Tokenizer.get_tokenizer(PreTrainedTokenizers.JAMBA_MINI_TOKENIZER)

# Jamba Large tokenizer
tokenizer = Tokenizer.get_tokenizer(PreTrainedTokenizers.JAMBA_LARGE_TOKENIZER)
```

### Async Usage

```python
import asyncio
from ai21_tokenizer import Tokenizer

async def main():
    tokenizer = await Tokenizer.get_async_tokenizer()

text = "Hello, world!"
    encoded = await tokenizer.encode(text)
    decoded = await tokenizer.decode(encoded)

print(f"Original: {text}")
    print(f"Encoded: {encoded}")
    print(f"Decoded: {decoded}")

asyncio.run(main())
```

### Advanced Token Operations

```python
# Convert between tokens and IDs
tokens = tokenizer.convert_ids_to_tokens(encoded)
print(f"Tokens: {tokens}")

ids = tokenizer.convert_tokens_to_ids(tokens)
print(f"IDs: {ids}")
```

### Direct Class Usage

```python
from ai21_tokenizer import SyncJambaTokenizer

# Using local model file
model_path = "/path/to/your/tokenizer.model"
tokenizer = SyncJambaTokenizer(model_path=model_path)

text = "Hello, world!"
encoded = tokenizer.encode(text)
decoded = tokenizer.decode(encoded)
```

### Async Direct Class Usage

```python
from ai21_tokenizer import AsyncJambaTokenizer

async def main():
    model_path = "/path/to/your/tokenizer.model"
    tokenizer = await AsyncJambaTokenizer.create(model_path=model_path)

text = "Hello, world!"
    encoded = await tokenizer.encode(text)
    decoded = await tokenizer.decode(encoded)

asyncio.run(main())
```

**For more examples, please see our [examples](examples) folder.**

ai21-tokenizer

ai21-tokenizer Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

ai21-tokenizer

ai21-tokenizer Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README