VulnFanatic-NG

KO | EN

gitlite — search

#javascript #hacktoberfest #llm #golang #machine-learning #rust #deep-learning #docker #nodejs #awesome #kubernetes #vue #awesome-list #ios

(★ 15)

BianryNinja plugin for identifying vulnerabilities in decompiled binaries with both programmatic scans and LLM support.

VulnFanatic-NG Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# VulnFanatic-NG

LLM-assisted vulnerability research for [Binary Ninja](https://binary.ninja/).

VulnFanatic-NG adds a sidebar panel that scans the current binary and asks an
LLM — a **locally hosted** OpenAI-compatible model by default, or Anthropic
Claude, Google Gemini, or Azure OpenAI (see **LLM backends**) — to judge whether
suspicious code is actually vulnerable. It works primarily from Binary Ninja's
**decompiler output (HLIL)**, falling back to assembly when needed, and reports
only confirmed issues with clickable references back to the code.

---

## How it works

A scan runs in **up to three phases** (Phase 3 is opt-in and online-only):

### Phase 1 — dangerous function calls
Finds call sites of dangerous functions defined in
[`rules/phase1_rules.json`](rules/phase1_rules.json) — `strcpy`,
`memcpy`, `sprintf`/format strings, `system`, `alloca`, `scanf`, command/exec
APIs, weak RNG, the `free`/`delete` family (**use-after-free / double-free**),
**reads of untrusted input into fixed buffers** (`recv`/`read`/`fread`/`ReadFile`),
**SQL injection** (`sqlite3_exec`/`mysql_query`/`PQexec`), **disabled TLS
certificate verification** (`SSL_CTX_set_verify`/curl), **SSRF**, and **improper
privilege management** (`setuid`/`setresgid`), the **`memset`/`bzero` family**, and
**comparisons with an attacker-controlled length** (`memcmp`/`strncmp` →
authentication bypass), across C/C++, Win32, and (best-effort) Rust FFI. Coverage
includes fortified `_chk` (FORTIFY) and Annex-K `_s` variants. Bounded
formatted-output functions (`snprintf` and variants) have their own default-safe
rule so a correct size argument is not reported as an overflow. Call sites are
found three ways: direct calls to the named symbols; calls routed through
**forwarding thunks / PLT stubs** (the real callers are recovered, so an import
reached only through a stub is not missed); and — unless
`vulnfanatic.scanIndirectCalls` is off — **indirect calls dispatched through a
function pointer or vtable** that Binary Ninja resolved to a dangerous function.
For each call site it builds an **interprocedural**, decompiler-centric context,
budgeted to a token limit (default 100k):

- the **call expression and its arguments**,
- the **declared prototype of the called function** (from Binary Ninja's type info,
  else a built-in table) so the model maps arguments to parameters correctly —
  fortified `__*_chk` and bounds-checked `*_s` variants take extra leading
  arguments, shifting the format/size/destination position,
- the **type and byte size of each call argument** (buffer capacities), derived
  from the argument's HLIL expression type so that a struct field like `s->buf`
  resolves to the field's real array size rather than the pointer size of `s`;
  struct definitions in the types section also carry per-field byte sizes,
- the **concrete value / range of each argument** resolved by Binary Ninja's
  constant-propagation and value-set analysis (e.g. a length proven constant `0x40`
  or bounded to `[0, 0xff]`), which the model uses as ground truth when comparing a
  size against a buffer capacity instead of guessing,
- the **stack frame layout** of the calling function (variable offsets and byte
  sizes) when it contains a fixed-size buffer, so a stack overflow can be judged
  against the adjacent variables and the saved return address
  (`vulnfanatic.includeStackLayout`),
- the **path constraints** (the `if`/loop/`switch` conditions guarding the call),
- an **argument data-flow summary** — where each call argument is defined and
  used within the function,
- **parameter resolution across callers** — when a dangerous argument is a
  *parameter* of the calling function, the context reports what every caller
  actually passes for it (e.g. "all callers pass a string literal"), so a format/
  size parameter that is always constant isn't mistaken for attacker-controlled,
- the **full decompiled body** of the calling function,
- **data-type definitions** (struct/union/enum) for the types referenced across
  the call chain and the argument variables, so the model knows real buffer/field
  sizes and integer widths,
- the **decompiled bodies of functions that produce or consume the call's
  argument variables** (traced via HLIL def/use), which is what makes
  use-after-free / double-free and tainted-size reasoning possible,
- **call paths** from entry points / exported functions down to the call,
- the **decompiled body of every function along those call paths** (nearest the
  dangerous call first), each annotated with the call site and the conditions
  guarding the next hop,
- the **bodies of other functions those path functions call** (e.g. for
  `MAIN→ABCD→strcpy`, also the functions `MAIN` and `ABCD` call elsewhere), since
  they may hold the bounds/validation checks that gate the dangerous value
  (`vulnfanatic.includeCallPathSiblings`, filled while budget allows), and
- **tainted-source hints** (input functions like `recv`/`read`/`getenv` called in
  the same function).

That context plus a rule-specific prompt is sent to the model, which returns a
structured verdict. Non-issues are dropped. The prompts are tuned for a strong
local code model (e.g. **Qwen2.5-Coder**) and instruct it to analyze the entire
flow and emit JSON only.

### Reasoning, scratchpad, and confidence

The model is told to favor **recall** — report plausible, security-relevant issues
and express uncertainty through a **Confidence** rather than dropping anything it
can't fully prove. It shows its work in a **scratchpad** that quotes the verbatim
code snippets it relied on (the input source, each guard, the size/length, the
relevant type, and the sink), which is stored on the finding so you can audit the
reasoning.

Each finding carries a **Confidence** (high/medium/low): high = the whole chain is
shown in the context; medium = likely, with a link or two inferred; low = a lead
worth manual review. This is the headline metric (the model's severity estimate is
a secondary field). Set `vulnfanatic.minConfidence` to drop anything below a threshold.

### Tuning precision vs. recall

By default VulnFanatic-NG favors recall (catching real issues). If you get too many false
positives, tighten with any of:

- **`vulnfanatic.validationPass`** (default **off**) — runs a second LLM pass that
  double-checks each flagged issue against the same context (verifying the
  scratchpad snippets and re-tracing the flow) and can correct the verdict or
  confidence. Doubles LLM calls for flagged candidates.
  - **Separate validator model (recommended when you use the validation pass).** Set
    **`vulnfanatic.validatorModel`** (plus `validatorProvider` / `validatorBaseUrl` /
    `validatorApiKey`) to run the second pass on a **different** model. A second
    opinion is far more useful from an *independent* model — it shares fewer blind
    spots and is much less likely to rubber-stamp the first verdict (models tend to
    prefer their own answers). A good pattern is a **cascade**: a fast model as the
    analyst (broad recall) and your strongest model as the validator, which only runs
    on flagged candidates. Leave the validator model blank to validate with the
    analyst model. The validator should be at least as capable as the analyst — a
    weaker one mostly adds false rejections. Everything except provider/base URL/key/
    model is inherited from the analyst connection settings; a blank validator key
    reuses the analyst key; and if the validator endpoint is unreachable the first
    verdict is kept (the finding is never lost to a validator outage).
- **`vulnfanatic.minConfidence`** (default `low`) — raise to `medium`/`high` to report
  only stronger findings.
- **`vulnfanatic.skipConstantArgCalls`** (default **off**) — skip overflow-class call
  sites whose arguments are all compile-time constants.

**Speed.** Most per-call latency is the written reasoning, so
**`vulnfanatic.verdictReasoning`** controls how much the model writes:
- **`concise`** (default) — a brief 1–3 sentence rationale, no verbatim code. Much
  faster than `full` with little accuracy loss; you can also lower
  `vulnfanatic.maxResponseTokens`.
- **`full`** — the detailed scratchpad with quoted snippets (most auditable, slowest).
- **`none`** — verdict only. Fastest; pair it with a reasoning-capable backend
  (`vulnfanatic.reasoningEffort`) so the model's *internal* thinking does the work.
  On a plain local model `none` loses accuracy (no chain-of-thought at all).

Supporting precision features that are always on (they inform the model without
suppressing findings):

- **Argument provenance** — the context tells the model, per argument, whether it
  is a *compile-time constant*, a *function parameter*, or *derived from a tainted
  source*, which the model uses to set confidence.
- **Bounds-checked-variant awareness** — the prompts treat `_s` (Annex K) and
  `_chk` (FORTIFY) variants and length-bounded APIs as safe unless the size
  argument itself is wrong.
- **Explicit buffer sizes** — argument and struct-field byte capacities are
  provided so the model compares capacity against bytes written instead of
  guessing.

### Scan Offline (no LLM)

The **Scan Offline** button runs Phase 1 with **no model** — purely programmatic
heuristics declared in the `offline` block of each rule in `phase1_rules.json`. It
flags dangerous call sites and *eliminates the obviously-safe ones*, assigning a
heuristic **Confidence**:

- **Eliminated** (not reported): a `memcpy`/`memmove` with a constant length, a
  `strcpy` from a constant string, a `printf` with a constant format, a `system`
  with a constant command, etc. — calls whose governing argument is a compile-time
  constant and so can't be attacker-controlled. "Constant" includes values that
  Binary Ninja's value-set analysis pinned to a fixed number upstream, not just
  literal arguments.
- **High** confidence: flagged with no mitigating check found anywhere on the
  execution flow.
- **Downgraded** (→ medium): the size/length argument was either resolved to a
  constant/bounded range by Binary Ninja's value-set analysis, or a real
  bounds/validation check *comparing* the relevant argument (e.g. a `strlen`/size
  comparison, `if (len < …)`) was found somewhere on the flow — including in the
  functions called along the path — so it may already be handled. (A branch that
  merely mentions the variable without comparing it no longer counts, removing a
  source of spurious downgrades.)

The heuristics use a small **declarative vocabulary** in the rules
(`constant_safe_args`, `eliminate_if_all_args_constant`, `format_arg_lookup`,
`length_guard_vars`, `base_confidence`, `skip`) evaluated by Python predicates —
no embedded code to `exec`. Most rules have an offline definition (overflow,
format-string, command-exec, scanf, path handling, weak-RNG, weak numeric parsing,
privilege changes, allocation size, …). Only the two categories that genuinely need
semantic analysis are skipped offline and left for the LLM: the `free`/`delete` family
(**use-after-free / double-free**, which needs pointer-lifetime tracking) and **TLS
verification** (the bug is a specific constant value like `SSL_VERIFY_NONE`). The
offline summary reports how many sites were flagged / eliminated / skipped (need the
LLM) / failed, so the counts add up. This is a fast triage; for real judgment — and
for the skipped categories — run the full LLM scan.

Offline findings still build the **same full interprocedural context** an online
scan would send (only for the flagged sites) and store it, so once you triage them
they can be **exported as fine-tuning data** just like online findings. Disable with
`vulnfanatic.offlineBuildContext` if you want maximum offline speed.

### Phase 2 — security-sensitive code (symbol-gated)
Only runs when the binary appears to have real symbols / variable names. Locates
security-sensitive functions defined in
[`rules/phase2_rules.json`](rules/phase2_rules.json) —
authentication, cryptography (incl. weak algorithms), signature/certificate
verification, session/token handling, access control, secret/key handling,
input validation, non-constant-time secret comparison, and insecure
deserialization — matched by function name and referenced strings, then audited
by the model.

### Phase 3 — hardware-attack hardening audit (online, opt-in)

A **firmware hardening** audit against fault-injection (voltage/clock/EM glitching)
and side-channel (timing/power) attacks, based on hardware-attack mitigation
guidance. Unlike Phases 1–2 (which find *bugs*), Phase 3 reports a **missing or
violated hardening control** on a security-critical function — for example:
default-fail branches, double-checked security decisions, post-loop counter
validation, high-Hamming-distance state constants (vs plain 0/1), constant-time
full-length secret comparison, randomized-offset secret access/clearing,
encrypt-then-verify (anti-DFA), control-flow integrity counters, avoiding user-land
crypto, and not handling raw key material directly
([`rules/phase3_rules.json`](rules/phase3_rules.json)).

Because compiler optimizations can strip source-level protections, these controls
are best verified on the **compiled binary** — exactly what this checks. Phase 3 is
**LLM-only (online)**, symbol-gated, and **disabled by default**; enable it per scan
with the **Phase 3** checkbox on the New Scan tab (it never runs in offline mode).

Findings are listed in a table (status, confidence, phase, CWE, function, address,
title) with a detail pane that shows the explanation, the analysis scratchpad, and
the validation notes. **Double-click a row to navigate** the binary view to the code.

### Triage workflow

Every finding starts **Untriaged**. **Right-click** a row to set its status —
**Mark as Real Issue**, **Mark as False Positive**, or **Mark as Untriaged**. Each
status change pops up a **"Provide reason:"** text box (the reason is stored with
the finding). The table makes status obvious: Real Issues are **green/bold and sort
to the top**, False Positives are **grey/struck-through and sort to the bottom**,
Untriaged sit in between with their confidence colour. A summary line shows the
counts.

Each result tab has an **Export triaged (fine-tuning)…** button that exports *only*
the triaged findings (Real Issue + False Positive) as **OpenAI chat-format JSONL**
for fine-tuning: each example pairs the original system+user prompt with the
**human-corrected verdict** as the assistant target (a False Positive teaches
`is_vulnerable=false` with your reason; a Real Issue reinforces `is_vulnerable=true`),
so you can iteratively improve the model's accuracy on your binaries.

The per-finding context shown in the detail pane (and used to reconstruct the
fine-tuning prompts) is, by default, kept **in full** — controlled by
`vulnfanatic.storedContextChars` (`0` = unlimited; set a positive cap, e.g. `4000`,
to limit BNDB growth at the cost of context fidelity).

### Multiple scans (tabs)

The panel is tabbed. The first tab is always **New Scan**, where you set:

- an optional **Scan name** (blank → `<timestamp> <mode>`, e.g.
  `2026-06-15 14:03:50 offline`),
- an optional **Phase 1 / Phase 2 / Phase 3 custom rules path** (blank → the bundled
  defaults), so you can run an alternative rule set,
- a **Phase 3** checkbox (off by default) to additionally run the online-only
  hardware-attack hardening audit,

then press **Start Scan** or **Scan Offline**. Each run opens its own result tab
and findings stream into it live. **All scans are stored in the BNDB**, so you can
e.g. keep an offline scan and later add an online scan, or compare runs with
different rule sets, side by side — they reappear as tabs when you reopen the
database. **Closing a tab permanently deletes that scan from the BNDB** — to
prevent accidents it pops up a confirmation that requires ticking *"I confirm that
I will lose the results from <name> forever."* before the **Delete results forever**
button activates. **Export current scan…** writes the selected tab to Markdown/JSON.

Each **open binary has its own independent panel state** — its own scan tabs and
running scan. Starting a scan in one binary and switching to another shows the
second binary's results (and lets you scan it separately); the first binary's scan
keeps running in the background and is intact when you switch back.

---

## Installation

This plugin's package folder is named `vulnfanatic_ng` (a valid Python identifier —
Binary Ninja imports the plugin folder name as a module, so a hyphenated name
like `VulnFanatic-NG` would not load).

1. (Optional) Install accurate token counting into Binary Ninja's Python:
   ```
   pip install tiktoken
   ```
2. Symlink or copy the **`vulnfanatic_ng`** folder into your Binary Ninja user plugins
   directory:
   - macOS: `~/Library/Application Support/Binary Ninja/plugins/`
   - Linux: `~/.binaryninja/plugins/`
   - Windows: `%APPDATA%\Binary Ninja\plugins\`

For example, on macOS:
   ```
   ln -s "$(pwd)/vulnfanatic_ng" "$HOME/Library/Application Support/Binary Ninja/plugins/vulnfanatic_ng"
   ```
3. Restart Binary Ninja (or run *Reload Plugins*). A **VF** icon appears in the
   right sidebar.

---

## Configuration

Open **Settings** (the gear / `Edit ▸ Preferences ▸ Settings`) and search for
**`vulnfanatic`**. Set at minimum:

| Setting | Meaning |
| --- | --- |
| `vulnfanatic.apiProvider` | Which LLM backend to call: `openai` (default), `anthropic`, `google`, or `azure`. See **LLM backends** below. All providers are reached over the Python standard library — nothing to `pip install`. |
| `vulnfanatic.apiBaseUrl` | Endpoint base for the selected provider (see the table below). Default `http://localhost:8080/v1`. Set to the literal `TEST` to enable **test mode** (see below). |
| `vulnfanatic.apiKey` | API key / bearer token. May be blank for local servers. Overridden by the `VULNFANATIC_API_KEY` or `OPENAI_API_KEY` environment variables. |
| `vulnfanatic.model` | **Required** (except in test mode). The model identifier (for `azure`, the **deployment name**). |
| `vulnfanatic.apiMode` | `openai` only: `chat` (default, `/chat/completions`) vs `completions` (single flattened prompt — for base/instruct models served **without a chat template**). |
| `vulnfanatic.azureApiVersion` | `azure` only: the `api-version` query parameter (default `2024-10-21`). |

### LLM backends

`vulnfanatic.apiProvider` selects how requests are formed and authenticated. The
verdict contract (and all rule prompts) are identical across providers.

| Provider | `apiBaseUrl` | Auth | Notes |
| --- | --- | --- | --- |
| `openai` | your server, e.g. `http://localhost:8080/v1` | `Authorization: Bearer` | OpenAI-compatible Chat/Completions: local llama.cpp / ollama / vLLM, OpenAI, and **AWS Bedrock's OpenAI-compatible endpoint**. |
| `anthropic` | blank → `https://api.anthropic.com` | `x-api-key` + `anthropic-version` | Claude **Messages API** (`POST <base>/v1/messages`). `temperature` is not sent (current Claude models reject it). |
| `google` | blank → `https://generativelanguage.googleapis.com` | API key in the URL | Gemini `generateContent` (`<base>/v1beta/models/<model>:generateContent`). |
| `azure` | `https://<resource>.openai.azure.com` | `api-key` header | Azure OpenAI; set `model` to the **deployment name** and `azureApiVersion` to your API version. |

> AWS Bedrock can be used through the `openai` provider via its OpenAI-compatible
> endpoint, so it does not need a dedicated backend.

Other useful settings: `vulnfanatic.maxContextTokens` (default 100000),
`vulnfanatic.maxResponseTokens`, `vulnfanatic.temperature`,
`vulnfanatic.reasoningEffort` (`off`/`low`/`medium`/`high`; default **high** — asks
the model to think before answering where supported, mapped per provider:
openai/azure `reasoning_effort`, anthropic adaptive thinking + `output_config.effort`,
google dynamic `thinkingConfig`; auto-stripped and retried if a model rejects it),
`vulnfanatic.requestTimeoutSec`,
`vulnfanatic.callPathMaxDepth` / `vulnfanatic.callPathMaxPaths`,
`vulnfanatic.callPathIncludeBodies` (include the decompiled bodies of functions along
the call path; default on) / `vulnfanatic.callPathMaxBodies` (cap, default 12),
`vulnfanatic.includeCallPathSiblings` (also include other functions called along the
path, which may hold the bounds/validation checks; default on) /
`vulnfanatic.callPathSiblingMaxBodies` (cap, default 12),
`vulnfanatic.includeDataTypes` (include struct/union/enum definitions; default on) /
`vulnfanatic.maxTypeDefs` (cap, default 24),
`vulnfanatic.includeVariableDataflow` (trace call arguments back through their producers
/consumers and include those bodies; default on) / `vulnfanatic.dataflowMaxFunctions`
(cap, default 8),
`vulnfanatic.includeStackLayout` (include the calling function's stack-variable layout
when it has a fixed-size buffer; default on),
`vulnfanatic.scanIndirectCalls` (also match dangerous calls dispatched through a resolved
function pointer/vtable; default on — turn off for a faster scan on very large binaries),
`vulnfanatic.validationPass` (run the second double-check pass; default off) /
`vulnfanatic.validatorModel` / `vulnfanatic.validatorProvider` /
`vulnfanatic.validatorBaseUrl` / `vulnfanatic.validatorApiKey` (run the validation pass on
a separate, independent model — blank = same model as the analyst) /
`vulnfanatic.minConfidence` (`low`/`medium`/`high`; drop findings below this; default
`low`), `vulnfanatic.flagUnparseableResponses` (report sites the model couldn't score as
`UNKNOWN`-confidence "Unscored" leads instead of dropping them; default on),
`vulnfanatic.skipConstantArgCalls` (skip all-constant overflow call sites;
default off), `vulnfanatic.verdictReasoning` (`concise`/`full`/`none`; how much
reasoning the model writes per verdict — the main speed lever; default `concise`),
`vulnfanatic.runPhase1` /
`vulnfanatic.runPhase2` / `vulnfanatic.runPhase3` (enable each phase; Phase 3 is
online-only and usually toggled per-scan via the New Scan checkbox rather than here),
`vulnfanatic.phase2RequireSymbols` / `vulnfanatic.phase2ForceEnable`,
`vulnfanatic.tokenizerEncoding` (tiktoken encoding for token estimates; falls back to
a character heuristic if tiktoken is not installed),
`vulnfanatic.offlineBuildContext` (build full context for offline findings so they can
be exported for fine-tuning; default on),
`vulnfanatic.debugLogging` (verbose pipeline trace to the console; default off) /
`vulnfanatic.debugAnonymous` (redact all binary-identifying details so the log can be
shared — see **Debug logging** below),
`vulnfanatic.sendJsonResponseFormat`, `vulnfanatic.tlsVerify` (verify HTTPS
certificates; default on) / `vulnfanatic.caBundlePath` (CA bundle for HTTPS — see
Troubleshooting if you hit `CERTIFICATE_VERIFY_FAILED`), and
`vulnfanatic.rulesPhase1Path` / `vulnfanatic.rulesPhase2Path` /
`vulnfanatic.rulesPhase3Path` (point these at your own rule files to customize
detections and prompts).

> **Security note:** the API key is stored in Binary Ninja's settings in
> plaintext. Prefer the environment-variable override for sensitive keys.

### Test mode (dry run, no LLM)

Set `vulnfanatic.apiBaseUrl` to the literal value **`TEST`** to run without any LLM:

- The model is **never called** (no network, no API key/model needed).
- **Every** candidate (each dangerous call site in Phase 1, each
  security-sensitive function in Phase 2) is flagged.
- The full prompt — system prompt **and** the complete generated context — for
  each candidate is written to its own file under
  `/tmp/vulnfanatic_ng/<binary>-<timestamp>/`.
- The findings table shows a **Prompt File** column (hover for the full path),
  and the detail pane and exports include the path.

Use this to inspect and validate exactly what VulnFanatic-NG would send to the model,
and to iterate on the rule prompts/context without spending model time.

### Debug logging

Turn on **`vulnfanatic.debugLogging`** to print a verbose, step-by-step trace of the
scan pipeline (both online and offline) to the Binary Ninja log/console: each call
site, every skip/elimination decision, context build (size only), each LLM request
(provider/model/endpoint, retries, fallbacks), every verdict, and each reported
finding. **API keys are never logged.**

While debug logging is on, an **online** scan keeps **every candidate in the results
table** instead of dropping the ones that don't become confirmed issues, each tagged
with a debug-only status (dimmed, sorted to the bottom):

- **REJECTED** — the LLM returned a verdict of *not an issue*.
- **SKIPPED** — eliminated *before* the LLM by a provably-safe rule (a constant format
  string, or all-constant arguments); the row explains which.
- **ERROR** — the candidate could not be analyzed (context build failed, or the LLM
  response was unparseable / the connection failed); the row carries the error.

So a debug scan shows one row per candidate in the `/N` total, and the summary reports
issues vs. rejected/skipped/error counts separately. You can right-click any of these
rows to re-triage it as a Real Issue or False Positive (which makes it eligible for
fine-tuning export). (Offline scans are unaffected — they never call the LLM.)

Independently of debug mode, when a model returns an unparseable response — a stray
token like Gemma's `<unused…>`, prose instead of JSON, or an **empty message** (only a
`role`, no `content`) — the client makes **one corrective retry**, re-asking for JSON
only with the structured-output format disabled; if that succeeds it keeps the format off
for the rest of the scan. The client also reads the **reasoning channel**
(`reasoning_content` / `reasoning`) when `content` is empty, so reasoning models that put
their answer there still work.

The empty-message case is common with **reasoning models such as GPT-OSS / o1** served
over an OpenAI-compatible API (e.g. `mlx-community/gpt-oss-20b`): with
`response_format=json_object` set, the harmony "final" answer channel is often suppressed
and the server returns `{"role": "assistant"}` with no content. These models can also burn
their whole output budget on the reasoning channel and get **truncated mid-thought**,
returning prose with no JSON at all. The auto-retry recovers the format-related cases; if
it persists, **turn off `vulnfanatic.sendJsonResponseFormat`**, **lower
`vulnfanatic.reasoningEffort`** (so less budget goes to thinking), and/or **raise
`vulnfanatic.maxResponseTokens`**. A persistent `<unused…>`/garbage reply instead usually
means the prompt exceeds the model's context window (set `vulnfanatic.modelContextWindow`
and/or raise the server's context length), or that the model is a poor fit for strict JSON
output (a code model such as Qwen2.5-Coder behaves far better than Gemma here).

**Recall-preserving fallback.** When a candidate still can't be scored after the retry,
`vulnfanatic.flagUnparseableResponses` (default **on**) reports it anyway as an
**"Unscored"** finding with **`UNKNOWN` confidence** — a value distinct from `low` (the
model never produced a verdict, so it isn't a low-confidence *judgement*) that sorts to
the bottom — keeping the model's partial output as the explanation, so you don't lose the
site, you just review it manually. Turn it off to drop such sites instead (they then
surface only as analysis errors, or debug ERROR rows).

Also enable **`vulnfanatic.debugAnonymous`** to make the log safe to share: it
redacts everything that could identify the analyzed file — symbol/variable names and
addresses become per-run salted hashes (still consistent within a run so the flow is
followable), the file name is hidden, finding text is replaced with `<redacted>`, the
LLM endpoint host is hashed, and decompiled code / prompts / context are logged as
sizes only (never the content). So you can send a debug log to report an issue without
disclosing anything about your binary.

---

## Usage

1. Open a binary and let analysis finish.
2. Click the **VF** sidebar icon to open VulnFanatic-NG.
3. Press **Start Scan** (full LLM scan) or **Scan Offline** (fast, no-LLM
   programmatic Phase 1 — see above). Progress shows in the panel and the Binary
   Ninja status bar; findings appear live and can be cancelled.
4. Click a finding to read the explanation; double-click to jump to the code.
5. **Right-click** a finding to *Mark as false positive* — it moves to the bottom
   of the table, greyed out and struck through, and the menu entry flips to *Mark
   as real issue* to undo it. (The right-click menu also has *Go to code*.)
6. **Export** to Markdown or JSON, or **Clear** to discard saved findings.

Findings — including their false-positive status — are stored in the Binary Ninja
database. They are written into the `.bndb` when you save the database (and
flushed immediately if a `.bndb` already exists), so they survive reopening.

The scan analyzes **every** matched call site (no cap), which is appropriate for
local models. For a hosted/paid endpoint, be mindful of volume on large
binaries.

---

## Customizing rules

Both rule files share an envelope with a shared `system_prompt` and
`output_schema`, plus a list of `rules`. Copy a bundled file, edit the
functions/keywords/prompts, and point `vulnfanatic.rulesPhase1Path` /
`vulnfanatic.rulesPhase2Path` at your copy. Phase 1 rules match by `functions` (exact)
and `name_regex`; Phase 2 rules match by `name_keywords`, `name_regex`, and
`string_keywords`. Each rule's `prompt` may use the `{function}` placeholder.

---

## Fine-tuning the model (MLX, Apple Silicon)

The triaged exports are designed to be fed straight back into the model. After
triaging findings across **several binaries** and clicking **Export triaged
(fine-tuning)…** on each (collecting the `.jsonl` files into one folder),
`scripts/finetune_mlx.py` runs an [MLX](https://github.com/ml-explore/mlx) LoRA
fine-tune on them.

```bash
pip install mlx-lm     # Apple Silicon / macOS

# Fine-tune a local 4-bit model on every *.jsonl under ./exports
python scripts/finetune_mlx.py ./exports \
    --model mlx-community/Qwen2.5-Coder-7B-Instruct-4bit \
    --adapter-path ./vf-adapters --iters 800

# ...then fuse the adapters into a standalone model
python scripts/finetune_mlx.py ./exports --model <base> \
    --fuse --fused-path ./vf-qwen-coder-vuln
```

The script takes the **training-data folder** as its positional argument and the
base **`--model`** (local path or MLX/HF repo id); other parameters are optional:
`--adapter-path`, `--valid-split` (0.1), `--iters`, `--batch-size` (auto-clamped to
fit a tiny split), `--num-layers`, `--learning-rate`, `--max-seq-length`
(`0` = **auto-fit** to the longest example, capped at 16384; set a positive value to
force it), `--fine-tune-type` (`lora`/`dora`/`full`), `--seed`, `--fuse`/`--fused-path`,
and `--dry-run` (prepare data + print the command without training). Anything after a
literal `--` is forwarded verbatim to `mlx_lm lora`. It recursively merges every
`*.jsonl` in the folder, validates and **de-duplicates** the chat examples, makes the
`train.jsonl`/`valid.jsonl` split MLX expects, then launches `python -m mlx_lm lora`
(and `mlx_lm fuse` with `--fuse`).

Serve the result with an OpenAI-compatible server (`mlx_lm.server --model <path>`)
and point `vulnfanatic.apiBaseUrl` back at it to scan with your tuned model.

> VulnFanatic-NG contexts are large, so by default the script **auto-fits**
> `--max-seq-length` to your longest example (rounded up, capped at **16384 tokens**).
> Long sequences dominate training memory, so a large model near this cap can run a
> smaller Mac out of memory. If your examples exceed the cap they're truncated — pass a
> higher `--max-seq-length` (more memory) or lower `vulnfanatic.storedContextChars`
> before exporting. If training is killed by a signal (e.g. `exit -10` / SIGBUS), that's
> an out-of-memory crash: lower `--max-seq-length`, add `-- --grad-checkpoint`, or use a
> smaller model.

---

## Development & testing

The plugin has zero required third-party dependencies. Pure modules
(`rules`, `tokens`, `llm`, `findings`, `settings`, `prototypes`) are covered by an
offline test suite that needs neither Binary Ninja nor a network. The `tests/`
suite lives in the project's source repository (it is not shipped inside the
published plugin); run it from there. From the package directory you can still
syntax-check every module:

```
python3 -m py_compile *.py ui/*.py
python3 -m unittest discover -s tests   # from the source repository
```

The Binary Ninja-facing modules (`context_builder`, `phase1`, `phase2`) import
cleanly without Binary Ninja (their API access is guarded) but require a running
Binary Ninja to exercise.

### Manual in-Binary-Ninja test
1. Build a small vulnerable C program (e.g. one that `strcpy`s `argv[1]` into a
   fixed stack buffer and calls `system()` on input). Compile **with symbols**
   to also exercise Phase 2.
2. Start your local OpenAI-compatible server and set `vulnfanatic.apiBaseUrl`,
   `vulnfanatic.apiKey`, and `vulnfanatic.model`.
3. Open the binary, run **Start Scan**, and confirm the dangerous calls are
   reported and double-clicking navigates to the call sites.

---

## Troubleshooting

**`SSL: CERTIFICATE_VERIFY_FAILED ... unable to get local issuer certificate`** —
the HTTPS endpoint's certificate is fine, but Binary Ninja's bundled Python has no
CA bundle to verify it against (common on macOS and in embedded Pythons; you'll see
this with hosted endpoints like AWS Bedrock, Anthropic, Google, Azure). Fix with one
of, in order of preference:
- **Install certifi** into the Python Binary Ninja uses: `pip install certifi`.
  VulnFanatic-NG picks it up automatically.
- **Point at a CA bundle:** set `vulnfanatic.caBundlePath` to a bundle file (or
  directory) — e.g. the path printed by `python3 -m certifi`, or `/etc/ssl/cert.pem`.
- **Last resort:** turn off `vulnfanatic.tlsVerify` (only for a trusted/internal
  endpoint or a self-signed local server — this disables certificate checking).

**`HTTP 400 ... tokenizer.chat_template is not set`** — the model you are serving
has no chat template, so the `/chat/completions` endpoint cannot format the
messages. VulnFanatic-NG **automatically falls back** to the `/completions` endpoint for
the rest of the scan when it sees this error, so scanning continues. To avoid the
first failed request entirely, set `vulnfanatic.apiMode` to `completions`. Alternatively,
fix it server-side by serving a model that ships a chat template, or pass one to
your server — e.g. for vLLM: `--chat-template <template.jinja>` (or use an
`-Instruct`/`-Chat` model variant). The dedicated chat template usually gives
better results than the flattened completions prompt.

**`No JSON object found ... response looks truncated`** — the model's reply was cut
off before the JSON finished. Two causes:
- The scratchpad exceeded the response budget → raise `vulnfanatic.maxResponseTokens`.
- **More commonly:** the prompt fills the model's **context window**, leaving no room
  to generate, so the reply stops after a few tokens *no matter how high
  `maxResponseTokens` is*. Local servers often have a small window (ollama defaults
  to `num_ctx=2048`!). Fix it by setting **`vulnfanatic.modelContextWindow`** to your
  server's window (e.g. ollama `num_ctx`, llama.cpp `-c`, vLLM `--max-model-len`) —
  VulnFanatic-NG then automatically caps the context it sends so prompt + response fit.
  Also keep `vulnfanatic.maxResponseTokens` reasonable (≈8192, not 65535) and/or raise the
  server's window. Tiny windows (≤8k) cannot hold the full interprocedural context;
  use a model/server configured for 32k+.

**`IncompleteRead` / `Could not complete request ... after N attempt(s)`** — the
server accepted the request but closed the connection before sending the full
response. This almost always means the model server died or stalled mid-generation:
out-of-memory (large context + long output), an internal/worker timeout, or a proxy
resetting the connection. VulnFanatic-NG retries once automatically and then skips that
site. **Check the model server's own logs** for the real cause; reducing
`vulnfanatic.maxContextTokens` and/or `vulnfanatic.maxResponseTokens`, or giving the server more
memory / a larger context window, usually resolves it.

## Limitations

- **Phase 2 is symbol-aware.** On a stripped binary, name-based matches are
  unreliable, so by default only the **string-evidence** rules run (functions that
  reference tell-tale string constants are still auditable). Set
  `vulnfanatic.phase2ForceEnable` to also audit name-based matches, or
  `vulnfanatic.phase2RequireSymbols`=off. The symbol gate is a heuristic.
- HLIL ↔ call-site mapping can fail; VulnFanatic-NG falls back to MLIL/assembly and
  notes the representation used per finding.
- Verdicts are only as good as the model. Treat findings as leads for manual
  review, not ground truth.
- Some local servers ignore or reject `response_format=json_object`; the client
  tolerates that and still extracts JSON. Disable `vulnfanatic.sendJsonResponseFormat`
  if your server rejects the parameter outright.

## License

VulnFanatic-NG

VulnFanatic-NG Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

VulnFanatic-NG

VulnFanatic-NG Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README