KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #codex #typescript #developer-tools #ai-agents #deepseek-harness #claude-code #ai-agent #cli #react #mcp

sentencepiece-jni

★ 38 Open GitHub ↗

Java JNI wrapper for SentencePiece: unsupervised text tokenizer for Neural Network-based text generation.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# SentencePiece Java Wrapper

![Build](https://github.com/levyfan/sentencepiece-jni/workflows/Build/badge.svg)

Java wrapper for [SentencePiece](https://github.com/google/sentencepiece) with JNI. This module wraps 
`sentencepiece::SentencePieceProcessor` class with the following modifications:

* Encode and Decode methods are re-defined as EncodeAsIds, EncodeAsPieces, DecodeIds and DecodePieces respectively.
* SentencePieceText proto is not supported.

## SentencePiece Version

[v0.1.96](https://github.com/google/sentencepiece/releases/tag/v0.1.96)

## Build and Install SentencePiece

To build and install the Java wrapper from source, please try the following commands:

```bash
% mvn clean install
```

## Using sentencepiece-jni as a dependency

Because the resulting JAR is platform-dependent, resolving this dependency is managed by the [os-maven-plugin](https://github.com/trustin/os-maven-plugin). Follow the instructions there to use this platform-dependent JAR.

Please note you need to have a C++ compiler and cmake installed.

## Usage

See [SentencePieceProcessorTest](src/test/java/com/github/google/sentencepiece/SentencePieceProcessorTest.java) for more.

sentencepiece-jni

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README