jalki

KO | EN

gitlite — search

#javascript #python #rust #docker #macos #java #nodejs #windows #awesome #cli #pytorch #security #mcp #ios

(★ 28)

Programmable eBPF fentry/fexit tracing framework for Linux. Hook any kernel function with one Rust trait — structured JSON events out. TCP connects, retransmits, closes, and any function you define.

jalki Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# jälki

**The kernel knows what's wrong. jälki lets you ask it.**

---

Today, asking the Linux kernel "why is this connection slow?" requires eBPF expertise that maybe a few hundred people in the world have. You need to know BTF, aya, ring buffers, CO-RE, the BPF verifier, kernel struct offsets, and how to interpret raw tracing data. It's a week of work before you see a single structured event.

jälki removes that barrier. You ask a question. jälki hooks the right kernel function, collects the events, and interprets them:

```
❯ jalki ask "why is postgres slow"

Probes selected:
  tcp_connect (fexit/kernel.tcp.connect)
  tcp_retransmit_skb (fentry/kernel.tcp.retransmit)
  attached tcp_connect → probe_001
  attached tcp_retransmit_skb → probe_002
Collecting events for 5s...
Collected 47 events. Interpreting...

# Question: why is postgres slow

## Events observed (47 total in 5s)
  jalki/tcp_connect: 12 events
  jalki/tcp_retransmit: 35 events

## Interpretation

**tcp_retransmit_skb** (warning)

packets are being lost on an active connection — network congestion,
  switch issue, or physical layer problem between nodes

Action: check network path between 10.42.1.15 and 10.42.2.8.
  this is a network problem, not application.
```

The kernel knew the answer all along. 35 retransmits in ESTABLISHED state on the path to Postgres. Network problem, not application. jälki just made that knowledge accessible.

---

## Why this matters

**For humans**: Network debugging is dark magic. When connections are slow, you guess. You restart things. You blame the application. The kernel has the answer — retransmit counts, TCP states, connection errnos — but that data is locked behind eBPF expertise. jälki unlocks it with a single command.

**For AI agents**: An agent debugging a production issue can now ask the kernel directly. No human eBPF expertise in the loop. The agent identifies the right kernel function, deploys a probe, reads structured events, and reasons about root cause. This is the foundation for autonomous infrastructure debugging.

**For the eBPF ecosystem**: Writing a new fentry/fexit probe is one Rust trait. jälki handles BTF loading, program attachment, ring buffer management, self-filtering, sampling, serialization, and emission. The framework does the hard parts so you can focus on what to observe and how to interpret it.

---

## How it works

```
                    kernel space
   ┌────────────────────────────────────────────────┐
   │  tcp_connect()      → fexit  → eBPF program ──┐│
   │  tcp_close()        → fexit  → eBPF program   ││
   │  tcp_retransmit_skb → fentry → eBPF program   ││
   │                                                ││
   │  PID_FILTER: skip jälki's own syscalls         ││
   │  per-probe ring buffers (4MB each) ◄───────────┘│
   └────────────────────┬───────────────────────────┘
                        │
                    userspace
   ┌────────────────────▼───────────────────────────┐
   │  jälki daemon                                   │
   │                                                 │
   │  loader     → attach probes via BTF metadata    │
   │  reader     → drain ring buffers → EventStore   │
   │  probes     → raw bytes → FALSE Protocol JSON   │
   │  sinks      → vartio (gRPC) / stdout / file     │
   │  IPC server → /run/jalki/jalki.sock             │
   │  metrics    → :9090 /metrics /healthz /readyz    │
   └────────────────────┬───────────────────────────┘
                        │
   ┌────────────────────▼───────────────────────────┐
   │  CLI / MCP / agents                             │
   │                                                 │
   │  jalki ask   → question → probes → interpret    │
   │  jalki watch → collect events from one probe    │
   │  jalki-mcp   → AI agent tool interface          │
   └────────────────────────────────────────────────┘
```

**fentry/fexit** — BPF trampolines, not kprobes. Near-zero overhead. Safe for production 24/7.

**CO-RE** — Compile Once, Run Everywhere. One binary, any kernel 5.5+ with BTF.

**Self-filter** — jälki's own PID is excluded in kernel space. No feedback loops.

---

## Quick start

```bash
# Build
cargo run -p xtask -- build-ebpf --release
cargo build --release -p jalki

# Terminal 1: start the daemon (needs root for eBPF)
sudo ./target/release/jalki --sink stdout --cluster dev

# Terminal 2: ask a question
./target/release/jalki ask "why are connections failing"

# Or explore
./target/release/jalki list --layer tcp
./target/release/jalki status
./target/release/jalki watch tcp_connect --seconds 10
./target/release/jalki stream tcp_retransmit_skb
```

`jalki ask` works without a daemon too — it falls back to a knowledge base analysis showing which probes to deploy and what to look for.

---

## CLI

| Command | What it does |
|---------|-------------|
| `jalki` (no subcommand) | Daemon mode — load eBPF, attach probes, emit events |
| `jalki ask "question"` | KB search → auto-deploy → collect → interpret → answer |
| `jalki watch <function>` | Deploy probe, collect for N seconds, print events |
| `jalki stream [function]` | Live ndjson event stream |
| `jalki list [--layer tcp]` | Browse the knowledge base |
| `jalki status` | Show attached probes, event counts, drops |

---

## What you get

Every kernel function you care about becomes a structured event:

```json
{
  "source": "jalki/tcp_retransmit",
  "type": "kernel.tcp.retransmit",
  "severity": "warning",
  "correlation_keys": ["10.42.1.15:48210->10.42.2.8:5432"],
  "network_data": {
    "src_ip": "10.42.1.15",
    "dst_ip": "10.42.2.8",
    "dst_port": 5432,
    "protocol": "tcp"
  },
  "process_data": {
    "pid": 1847,
    "command": "api-server"
  }
}
```

Your API server is retransmitting to Postgres. The kernel knows this. Now you know it too.

---

## The knowledge base

jälki ships a built-in knowledge base of kernel functions — which function to hook for a given question, what fields matter, and how to interpret the events.

The TCP state field on `tcp_retransmit_skb` is the most important signal:

| State | Value | What it means |
|-------|-------|---------------|
| SYN_SENT | 2 | Handshake failing — remote unreachable, firewall, host down |
| ESTABLISHED | 1 | Active connection losing packets — network congestion |
| CLOSE_WAIT | 7 | Application hung, not reading from socket |

**SYN_SENT retransmit = not an application problem.** The connection never established.

**ESTABLISHED retransmit = network problem, not application.** The packets are being lost in transit.

Different problems, different fixes. The kernel knows which one it is.

---

## The framework

jälki is a framework, not just a tool. The three TCP probes are batteries-included. Adding your own probe is one trait:

```rust
impl Probe for MyProbe {
    fn name(&self) -> &str { "my_probe" }
    fn program_name(&self) -> &str { "jalki_my_probe" }
    fn attachments(&self) -> &[Attachment] {
        &[Attachment::Fentry { function: "some_kernel_function" }]
    }
    fn ring_buffer_map(&self) -> &str { "MY_EVENTS" }
    fn decode_event(&self, raw: &[u8]) -> Result<KernelEvent, ProbeError> {
        // decode raw ring buffer bytes into a typed KernelEvent
        // (to_evidence / normalization are provided defaults)
    }
}
```

jälki handles eBPF loading, BTF attachment, ring buffer management, self-filtering, sampling, batching, and emission. You describe what to observe and how to interpret it. The framework does the rest.

---

## MCP server

`jalki-mcp` exposes kernel observability to AI agents via the Model Context Protocol:

```
jalki_find_probe("why are connections slow")  → tcp_retransmit_skb, tcp_connect
jalki_deploy_probe("tcp_retransmit_skb")      → probe_001
jalki_get_events("probe_001", filter={...})   → [Occurrence, ...]
jalki_explain_event(function, tcp_state=1)    → "network problem, not application"
jalki_probe_status()                          → attached probes + counts
```

An agent asks the knowledge base before guessing. Deploys probes. Reads events. Gets interpretations. No eBPF expertise required.

---

## Python SDK

For agents and rapid iteration. `pip install jalki` and ask the kernel from Python:

```python
import jalki

# one call: find → deploy → collect → interpret
result = await jalki.ask("why are connections failing")
print(result.interpretation, result.action)

# or control each step
matches = jalki.find("packet loss")                         # local KB, no daemon
handle  = await jalki.deploy("tcp_retransmit_skb")          # attach probe
async for event in jalki.stream(handle, interpreted=True):  # live events
    print(event.net.dst, event.severity, event.interp)
```

`find()` works offline — the knowledge base ships in the wheel. `ask()` falls back to KB-only analysis when no daemon is running, so it never raises.

---

## Built-in probes

| Probe | Hook | What it gives you |
|-------|------|-------------------|
| `TcpConnect` | `fexit/tcp_connect` | Connection attempts — 4-tuple, success/failure, errno |
| `TcpClose` | `fexit/tcp_close` | Connection teardown — 4-tuple, process info |
| `TcpRetransmit` | `fentry/tcp_retransmit_skb` | Retransmissions — 4-tuple, TCP state |

These three, joined on the 4-tuple, answer: which backends are being connected to, which connections are failing, which are retransmitting, and what the TCP state was when it happened. Alongside them ship `ProcessExec` (tracepoint `sched_process_exec` — what binaries run), and the `FileOpen`/`FileOpenAttempt` pair (`fexit/security_file_open` + the openat tracepoints — who touched, or tried to touch, watched files).

---

## Evidence delivery

Beyond the local `stdout` / `file` / `composite` sinks, the native Vartio sink ships evidence off the node (ADR-0003/0004):

```bash
jalki --sink vartio --vartio-endpoint http://vartio.vartio.svc:50061
```

Captured events are normalized, bound to their pod (`cgroup → container → pod`), and delivered to Vartio's source ingress over gRPC. Vartio interprets the evidence and writes product records to Ahti; jälki itself never writes to Ahti.

---

## Kubernetes

Deploys as a DaemonSet with `hostPID`, `hostNetwork`, and privileged access for eBPF.

**Start at [`deploy/kubernetes/`](deploy/kubernetes/)** — a reference manifest
that is a copy of what we run in production, mechanically checked against our
live spec in CI so it cannot drift into fiction, with a README covering the
values you must change, the parts you must not remove (each fails looking like
something else), how to verify evidence is actually flowing, and which metrics
to alert on — thresholds that have each fired for a real incident. (The
retired in-repo Helm chart's story is in `helm/README.md`: it agreed with
reality by hand and by luck, and nothing compared them.)

---

## Requirements

- Linux kernel 5.5+ x86, 6.0+ ARM64
- `CONFIG_DEBUG_INFO_BTF=y`, `CONFIG_BPF_JIT=y`
- BTF at `/sys/kernel/btf/vmlinux`
- Root or `CAP_BPF` + `CAP_PERFMON`

---

## Testing

jälki uses requirement-based testing. Specs define what must be true. The oracle validates it.

```
specs/                          ← requirements (natural language markdown)
  protocol/find.md                "find must return tcp_connect for connection questions"
  protocol/ask.md                 "ESTABLISHED retransmit must say network problem"
  knowledge/knowledge-base.md     "at least 20 probes across 5 layers"
       │
       │  each requirement maps to an oracle test case
       ▼
eval/oracle/                    ← standalone Rust binary, reads JSON from disk
  case_014_retransmit_established_says_network_problem
  case_080_econnrefused_says_not_listening
  case_060_at_least_20_probes
```

The oracle never imports jälki code. It reads knowledge base JSON and generated SDK files, then asserts they match the spec. When a case fails, fix the system — not the test.

```bash
# Run all 50 oracle cases
cargo test --manifest-path eval/oracle/Cargo.toml

# Run workspace tests (probes, codegen, store, SDK meta)
cargo test --workspace

# Python SDK conformance (no daemon needed)
cd jalki-sdk-python && .venv/bin/pytest tests/ -m "not daemon"
```

---

## Known limitations

- **dst_ip 0.0.0.0 on Cilium-managed connections** — `skc_daddr` reads 0 when Cilium drops the packet before destination resolution (policy denial), when the conntrack table has no entry for the connection, or during loopback SNAT where the address is temporarily 0.0.0.0. Not fixable from jälki — requires Cilium debug monitor logs (`cilium monitor --type drop`) to diagnose the specific cause.
- **src_port 0 on tcp_close events** — the kernel clears `skc_num` before `tcp_close` returns, so fexit sees 0. This is correct kernel behavior. Use the `tcp_connect` event's `src_port` and correlate by 4-tuple to get the full picture.
- **IPv4 only** — IPv6 in v0.2.
- **Privileged required** — `CAP_BPF` + `CAP_PERFMON` at minimum.

---

## Part of False Systems

```
jälki     kernel observation (this)
TAPIO     k8s observation
RAUTA     L7 gateway
VARTIO    actor attribution — receives jälki's evidence, writes to AHTI
AHTI      append-first structured datastore
syva      enforcement
rauha     container runtime
```

jälki is the deepest layer. It sees what the kernel sees.

---

> *jälki* (Finnish) — footprint, trace, track.

*false systems · berlin · 2026 · apache 2.0*

jalki

jalki Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

jalki

jalki Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README