rllama

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

★ 92 Open GitHub ↗

Ruby FFI bindings for llama.cpp to run open-source LLMs such as GPT-OSS, Qwen 3.5, Gemma 4, and Llama 3 locally with Ruby.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Rllama

Ruby bindings for [llama.cpp](https://github.com/ggml-org/llama.cpp) to run open-source language models locally. Run models like Gemma 4, Qwen 3.5, GLM 4.7, Nemotron, LFM2, Llama 3, and many others directly in your Ruby application code.

## Installation

Add this line to your application's Gemfile:

```ruby
gem 'rllama'
```

And then execute:

```bash
bundle install
```

Or install it yourself as:

```bash
gem install rllama
```

## CLI Chat

The `rllama` command-line utility provides an interactive chat interface for conversing with language models. After installing the gem, you can start chatting immediately:

```bash
rllama
```

When you run `rllama` without arguments, it will display:

- **Downloaded models**: Any models you've already downloaded to `~/.rllama/models/`
- **Popular models**: A curated list of popular models available for download, including:
  - Gemma 4 E4B / Gemma 4 26B-A4B
  - Nemotron 3 Nano 4B
  - Qwen 3.5 35B-A3B
  - LFM2 24B-A2B
  - GLM 4.7 Flash
  - GPT-OSS 20B
  - Llama 3.2 3B
  - Phi-4

Simply enter the number of the model you want to use. If you select a model that hasn't been downloaded yet, it will be automatically downloaded from Hugging Face.

You can also specify a model path or URL directly:

```bash
rllama path/to/your/model.gguf
```

```bash
rllama https://huggingface.co/microsoft/phi-4-gguf/resolve/main/phi-4-Q3_K_S.gguf
```

Once the model has loaded, you can start chatting.

## Usage

### Text Generation

Generate text completions using local language models:

```ruby
require 'rllama'

# Load a model
model = Rllama.load_model('lmstudio-community/gemma-3-1B-it-QAT-GGUF/gemma-3-1B-it-QAT-Q4_0.gguf')

# Generate text
result = model.generate('What is the capital of France?')
puts result.text
# => "The capital of France is Paris."

# Access generation statistics
puts "Tokens generated: #{result.stats[:tokens_generated]}"
puts "Tokens per second: #{result.stats[:tps]}"
puts "Duration: #{result.stats[:duration]} seconds"

# Don't forget to close the model when done
model.close
```

#### Generation parameters

By default the sampling parameters are taken from the model's own
`general.sampling.*` GGUF metadata when present (falling back to llama.cpp's
standard defaults: `temperature: 0.8`, `top_k: 40`, `top_p: 0.95`,
`min_p: 0.05`). You can inspect the resolved values with
`model.sampling_defaults`, or read any metadata key directly with
`model.meta('general.sampling.temp')`.

Pass any parameter explicitly to override the model default:

```ruby
result = model.generate(
  'Write a short poem about Ruby programming',
  max_tokens: 2024,
  temperature: 0.8,
  top_k: 40,
  top_p: 0.95,
  min_p: 0.05
)
```

The parameters actually used are reported back in `result.stats`
(`temperature`, `top_k`, `top_p`, `min_p`, `seed`).

#### Streaming generation

Stream generated text token-by-token:

```ruby
model.generate('Explain quantum computing') do |token|
  print token
end
```

#### System prompt

Include system promt to guide model behavior:

```ruby
result = model.generate(
  'What are best practices for Ruby development?',
  system: 'You are an expert Ruby developer with 10 years of experience.'
)
```

#### Messages list

Pass multiple messages with roles for more complex interactions:

```ruby
result = model.generate([
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'What is the capital of France?' },
  { role: 'assistant', content: 'The capital of France is Paris.' },
  { role: 'user', content: 'What is its population?' }
])
puts result.text
```

#### Tool definitions

Pass tool definitions to include them in the model's prompt for function calling:

```ruby
tools = [
  {
    type: 'function',
    function: {
      name: 'get_weather',
      description: 'Get the current weather for a location',
      parameters: {
        type: 'object',
        properties: {
          location: { type: 'string', description: 'City name' }
        },
        required: ['location']
      }
    }
  }
]

result = model.generate('What is the weather in Paris?', tools:)
puts result.text
```

#### Tool call results

Messages are passed to the model's chat template in the OpenAI format, so tool
calls and their results are wired through the standard message schema: the
assistant message carries a structured `tool_calls:` array, and the tool output
is sent back as a `role: 'tool'` message.

Pass `tools:` (and optionally `system:`) once when creating the context. They
are remembered for all subsequent turns. When the model emits a tool call, it
is parsed into `result.tool_calls` and stored in the conversation history in
structured form automatically. Execute the calls and send each result back as
a `role: 'tool'` message:

```ruby
context = model.init_context(tools:)

result = context.generate('What is the weather in Paris?')
result.tool_calls
# => [{ name: 'get_weather', arguments: { 'location' => 'Paris' }, id: 'call_1' }]

result.tool_calls.each do |call|
  weather = get_weather(call[:arguments]['location'])

result = context.generate(
    { role: 'tool', tool_call_id: call[:id], name: call[:name], content: weather.to_json }
  )
end

puts result.text
# => "The weather in Paris right now is 21°C."
```

The tool call markup is parsed by llama.cpp's chat parser, which derives the
format automatically from the model's own chat template. Any model with a
tool-capable template is supported. If no tool call is detected,
`result.tool_calls` is empty and `result.text` contains the raw output.

The context detects the edited history and automatically rolls the KV cache
back to the common prefix, so only the changed suffix is re-processed.

### Reasoning

For models with reasoning support, pass `reasoning: true` to
enable the model's reasoning phase. The reasoning block is parsed out of the
output: `result.text` contains only the final answer and the reasoning is
available separately:

```ruby
result = model.generate('Why is the sky blue?', reasoning: true)

result.reasoning
# => "Thinking Process:\n\n1. Analyze the request..."

result.text
# => "The sky appears blue because..."
```

Like `tools:`, the flag can also be set once per context via
`model.init_context(reasoning: true)`. It defaults to off; `result.reasoning`
is `nil` when reasoning is disabled or the model produced no reasoning block.
Note that when streaming with a block, the raw reasoning tokens are yielded
as they are generated.

### Chat

For ongoing conversations, use a context object that maintains the conversation history:

```ruby
# Initialize a chat context
context = model.init_context

# Send messages and maintain conversation history
response1 = context.message('What is the capital of France?')
puts response1.text
# => "The capital of France is Paris."

response2 = context.message('What is the population of that city?')
puts response2.text
# => "Paris has a population of approximately 2.1 million people..."

response3 = context.message('What was my first message?')
puts response3.text
# => "Your first message was asking about the capital of France."

# The context remembers all previous messages in the conversation

# Close context when done
context.close
```

### Embeddings

Generate vector embeddings for text using embedding models:

```ruby
require 'rllama'

# Load an embedding model
model = Rllama.load_model('lmstudio-community/embeddinggemma-300m-qat-GGUF/embeddinggemma-300m-qat-Q4_0.gguf')

# Generate embedding for a single text
embedding = model.embed('Hello, world!')
puts embedding.length
# => 724 (depending on your model)

# Generate embeddings for multiple sentences
embeddings = model.embed([
  'roses are red',
  'violets are blue',
  'sugar is sweet'
])

puts embeddings.length
# => 3
puts embeddings[0].length
# => 768

model.close
```

#### Vector parameters

By default, embedding vectors are normalized. You can disable normalization with `normalize: false`:

```ruby
# Generate unnormalized embeddings
embedding = model.embed('Sample text', normalize: false)
```

## Finding Models

You can download GGUF format models from various sources:

- [Hugging Face](https://huggingface.co/models?library=gguf) - Search for models with "GGUF" format

## License

MIT

## Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/docusealco/rllama.

rllama

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README