KO
|
EN
gitlite — search
Search
#java
#javascript
#android
#api
#php
#kotlin
#python
#spring-boot
#cli
#gmail
#git
#csharp
gemma4.java
★ 65
Open GitHub ↗
Fast Gemma 4 inference in pure Java
Download README (.md)
Explore Similar Repositories
Gemma-4-31B-MTP-vLLM-Server
:
A production-minded FastAPI sidecar for serving Gemma 4 31B on vLLM with Gemma 4 Multi-Token Prediction (MTP) speculative decoding.
gemma-skills
:
Gemma 4 Agent Skills
Relay
:
Mobile PWA that turns hard-to-understand speech into clear phrases using local Gemma 4 via Ollama, no cloud LLM on the interpret path.
gemma4-mac
:
Run Gemma 4 e4b locally on Apple Silicon — text, vision, and Apple Photos auto-captioning
Swift-gemma4-core
:
The first native Swift inference engine for Gemma 4 on Apple Silicon / iOS
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
gemma4.java
?
Download (.md)
# Gemma4.java <p align="center"> <img src="https://ai.google.dev/static/gemma/images/gemma4_banner.png"> </p> <div align="center">  [](https://raw.githubusercontent.com/mukel/gemma4.java/main/LICENSE) [](https://www.graalvm.org/latest/reference-manual/native-image/)  Fast, zero-dependency, inference engine for [Gemma 4](https://ai.google.dev/gemma) in pure Java. </div> ---- ## Features - Single file, **no dependencies**, based on [llama3.java](https://github.com/mukel/llama3.java) - Supports **all** Gemma 4 model families: `E2B`, `E4B`, `31B`, and `26B-A4B` (MoE) - Fast [GGUF format](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) parser - Supported dtypes/quantizations: `F16`, `BF16`, `F32`, `Q4_0`, `Q4_1`, `Q4_K`, `Q5_K`, `Q6_K`, `Q8_0` - Matrix-vector kernels using Java's [Vector API](https://openjdk.org/jeps/469) - CLI with `--chat` and `--prompt` modes - Thinking mode control with `--think off|on|inline` - GraalVM Native Image support - AOT model preloading for **instant time-to-first-token** ## Setup Download GGUF models from Hugging Face: | Model | Architecture | GGUF Repository | |-------|-------------|-----------------| | E2B | Dense, ~5B total params | [unsloth/gemma-4-E2B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF) | | E4B | Dense, ~8B total params | [unsloth/gemma-4-E4B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) | | 31B | Dense | [unsloth/gemma-4-31B-it-GGUF](https://huggingface.co/unsloth/gemma-4-31B-it-GGUF) | | 26B-A4B | Mixture of Experts (MoE) | [unsloth/gemma-4-26B-A4B-it-GGUF](https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF) | #### Optional: pure quantizations `Q4_0` files are often mixed-quant in practice (for example, `token_embd.weight` and `output.weight` may use `Q6_K`). A pure quantization is not required, but can be generated from an F32/F16/BF16 GGUF source with `llama-quantize` from [llama.cpp](https://github.com/ggml-org/llama.cpp): ```bash ./llama-quantize --pure ./gemma-4-E2B-it-BF16.gguf ./gemma-4-E2B-it-Q4_0.gguf Q4_0 ``` Pick any supported target quantization, for example `Q4_0`, `Q4_1`, `Q4_K`, `Q5_K`, `Q6_K`, or `Q8_0`. ## Build and run Java 21+ is required, in particular for the [`MemorySegment` mmap-ing feature](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/channels/FileChannel.html#map(java.nio.channels.FileChannel.MapMode,long,long,java.lang.foreign.Arena)). [`jbang`](https://www.jbang.dev/) is a good fit for this use case. No-setup one-liner, no git clone, no manual model download required ... ~5GB download once, then cached by `jbang`: ```bash jbang gemma4@mukel \ --model %https://hf.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/gemma-4-E2B-it-Q8_0.gguf \ --system-prompt "like Master Yoda, reply you must" \ --chat ``` Alternatively: ``` jbang Gemma4.java --help jbang Gemma4.java --model ./gemma-4-E2B-it-Q4_0.gguf --chat jbang Gemma4.java --model ./gemma-4-E2B-it-Q4_0.gguf --prompt "Explain quantum computing like I'm five" ``` Or run it directly (still via [`jbang`](https://www.jbang.dev/)): ```bash chmod +x Gemma4.java ./Gemma4.java --help ``` #### Optional: Makefile A simple [Makefile](./Makefile) is provided. Run `make jar` to produce `gemma4.jar`. Run the resulting `gemma4.jar` as follows: ```bash java --enable-preview --add-modules jdk.incubator.vector -jar gemma4.jar --help ``` ### GraalVM Native Image Compile with `make native` to produce a `gemma4` executable, then: ```bash ./gemma4 --model ./gemma-4-E2B-it-Q4_0.gguf --chat ``` ### AOT model preloading `Gemma4.java` supports AOT model preloading to reduce parse overhead and time-to-first-token (TTFT). To AOT pre-load a GGUF model: ```bash PRELOAD_GGUF=/path/to/model.gguf make native ``` A larger specialized binary is generated with parse overhead removed for that specific model. It can still run other models with the usual parsing overhead. ## Benchmarks <p align="center"> <img src="https://github.com/user-attachments/assets/8554de11-c028-4ff4-88b7-b5c0665423da"> </p> \*\**Hardware specs: AMD Ryzen 9950X 16C/32T 64GB (6400) Linux 6.18.12.* [GraalVM 25+](https://www.graalvm.org/downloads) is recommended for the absolute best performance (JIT mode), it provides partial, but good support for the [Vector API](https://openjdk.org/jeps/469), also in Native Image. By default, the "preferred" vector size is used, it can be force-set with `-Dllama.VectorBitSize=0|128|256|512`, `0` means disabled. ## Related Repositories - [llama3.java](https://github.com/mukel/llama3.java) - [gemma4.java](https://github.com/mukel/gemma4.java) - [gptoss.java](https://github.com/mukel/gptoss.java) - [qwen35.java](https://github.com/mukel/qwen35.java) - [nemotron3.java](https://github.com/mukel/nemotron3.java) ## License Apache 2.0
