gemma4.java

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 65)

Fast Gemma 4 inference in pure Java

gemma4.java Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Gemma4.java

![Java 21+](https://img.shields.io/badge/Java-21%2B-007396?logo=java&logoColor=white)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-green.svg?logo=apache)](https://raw.githubusercontent.com/mukel/gemma4.java/main/LICENSE)
[![GraalVM](https://img.shields.io/badge/GraalVM-Native_Image-F29111?labelColor=00758F)](https://www.graalvm.org/latest/reference-manual/native-image/)
![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows-lightgrey)

Fast, zero-dependency, inference engine for [Gemma 4](https://ai.google.dev/gemma) in pure Java.

</div>

----

## Features

- Single file, **no dependencies**, based on [llama3.java](https://github.com/mukel/llama3.java)
- Supports **all** Gemma 4 model families: `E2B`, `E4B`, `31B`, and `26B-A4B` (MoE)
- Fast [GGUF format](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) parser
- Supported dtypes/quantizations: `F16`, `BF16`, `F32`, `Q4_0`, `Q4_1`, `Q4_K`, `Q5_K`, `Q6_K`, `Q8_0`
- Matrix-vector kernels using Java's [Vector API](https://openjdk.org/jeps/469)
- CLI with `--chat` and `--prompt` modes
- Thinking mode control with `--think off|on|inline`
- GraalVM Native Image support
- AOT model preloading for **instant time-to-first-token**

## Setup

Download GGUF models from Hugging Face:

| Model | Architecture | GGUF Repository |
|-------|-------------|-----------------|
| E2B | Dense, ~5B total params | [unsloth/gemma-4-E2B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF) |
| E4B | Dense, ~8B total params | [unsloth/gemma-4-E4B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) |
| 31B | Dense | [unsloth/gemma-4-31B-it-GGUF](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF) |
| 26B-A4B | Mixture of Experts (MoE) | [unsloth/gemma-4-26B-A4B-it-GGUF](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF) |

#### Optional: pure quantizations

`Q4_0` files are often mixed-quant in practice (for example, `token_embd.weight` and `output.weight` may use `Q6_K`).
A pure quantization is not required, but can be generated from an F32/F16/BF16 GGUF source with `llama-quantize` from [llama.cpp](https://github.com/ggml-org/llama.cpp):

```bash
./llama-quantize --pure ./gemma-4-E2B-it-BF16.gguf ./gemma-4-E2B-it-Q4_0.gguf Q4_0
```

Pick any supported target quantization, for example `Q4_0`, `Q4_1`, `Q4_K`, `Q5_K`, `Q6_K`, or `Q8_0`.

## Build and run

Java 21+ is required, in particular for the [`MemorySegment` mmap-ing feature](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/channels/FileChannel.html#map(java.nio.channels.FileChannel.MapMode,long,long,java.lang.foreign.Arena)).

[`jbang`](https://www.jbang.dev/) is a good fit for this use case.

No-setup one-liner, no git clone, no manual model download required ... ~5GB download once, then cached by `jbang`:
```bash
jbang gemma4@mukel \
    --model %https://hf.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/gemma-4-E2B-it-Q8_0.gguf \
    --system-prompt "like Master Yoda, reply you must" \
    --chat
```
Alternatively:
```
jbang Gemma4.java --help
jbang Gemma4.java --model ./gemma-4-E2B-it-Q4_0.gguf --chat
jbang Gemma4.java --model ./gemma-4-E2B-it-Q4_0.gguf --prompt "Explain quantum computing like I'm five"
```

Or run it directly (still via [`jbang`](https://www.jbang.dev/)):
```bash 
chmod +x Gemma4.java
./Gemma4.java --help
```

#### Optional: Makefile

A simple [Makefile](./Makefile) is provided. Run `make jar` to produce `gemma4.jar`.

Run the resulting `gemma4.jar` as follows: 
```bash
java --enable-preview --add-modules jdk.incubator.vector -jar gemma4.jar --help
```

### GraalVM Native Image

Compile with `make native` to produce a `gemma4` executable, then:

```bash
./gemma4 --model ./gemma-4-E2B-it-Q4_0.gguf --chat
```

### AOT model preloading

`Gemma4.java` supports AOT model preloading to reduce parse overhead and time-to-first-token (TTFT).

To AOT pre-load a GGUF model:
```bash
PRELOAD_GGUF=/path/to/model.gguf make native
```

A larger specialized binary is generated with parse overhead removed for that specific model.
It can still run other models with the usual parsing overhead.

## Benchmarks

\*\**Hardware specs: AMD Ryzen 9950X 16C/32T 64GB (6400) Linux 6.18.12.*

[GraalVM 25+](https://www.graalvm.org/downloads) is recommended for the absolute best performance (JIT mode), it provides partial, but good support for the [Vector API](https://openjdk.org/jeps/469), also in Native Image.

By default, the "preferred" vector size is used, it can be force-set with `-Dllama.VectorBitSize=0|128|256|512`, `0` means disabled.

## Related Repositories

- [llama3.java](https://github.com/mukel/llama3.java)
- [gemma4.java](https://github.com/mukel/gemma4.java)
- [gptoss.java](https://github.com/mukel/gptoss.java)
- [qwen35.java](https://github.com/mukel/qwen35.java)
- [nemotron3.java](https://github.com/mukel/nemotron3.java)

## License

Apache 2.0

gemma4.java

gemma4.java Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

gemma4.java

gemma4.java Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README