sparse

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 9 Open GitHub ↗

Parse Microsoft' Windows SDK/Driver API documentation (MSDN) fast, and locally! Export to stable JSON format.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# sparse

> **s**dk-api + **parse**

Turn MSDN's Windows SDK API **and Windows Driver DDI** docs into a `.json` file, fast.

Now with `--mode driver` for [windows-driver-docs-ddi](https://github.com/MicrosoftDocs/windows-driver-docs-ddi): IRQL constraints, KMDF/UMDF version tags, `target-type`, `tech.root`, etc.

---

## Quick start

```bash
# install uv once (Windows)
winget install astral-sh.uv

# clone with submodules
git clone --recurse-submodules https://github.com/cristeigabriela/sparse.git
cd sparse

# parse sdk-api (the submodule path is the default for --mode sdk)
uv run python sparse.py -o sdk-api.json

# parse windows-driver-docs-ddi
uv run python sparse.py --mode driver -o driver-docs.json
```

`uv` handles the venv and dependencies automatically (see `pyproject.toml`). No `pip install` needed.

## Pre-built JSONs

The latest builds of both files are attached to the rolling [`latest`](https://github.com/cristeigabriela/sparse/releases/tag/latest) release. They get rebuilt on every push to `main` (skip the release by including `norelease` in the commit message, e.g. `docs(norelease): tweak README`).

## CLI

```text
usage: sparse.py [-h] [--mode {sdk,driver}] [-o OUTPUT] [--chunk-size CHUNK_SIZE] [--workers WORKERS] [--silent]
                 input_dir

Parse MSDN Native Function .md files to JSON (sdk-api or windows-driver-docs-ddi)

positional arguments:
  input_dir             Root content directory to search for nf-*.md files

options:
  -h, --help            show this help message and exit
  --mode {sdk,driver}   Frontmatter dialect: 'sdk' for sdk-api (default), 'driver' for windows-driver-docs-ddi
  -o, --output OUTPUT   Output JSON file path
  --chunk-size CHUNK_SIZE
                        Files per worker chunk
  --workers WORKERS     Number of parallel workers, or "max" to use _MAX_WINDOWS_WORKERS
  --silent              Suppress all output
```

### Example invocations

Each `--mode` defaults `input_dir` to the matching submodule, so the usual invocations are short:

```bash
# sdk-api (default mode)
uv run python sparse.py -o sdk-api.json

# windows-driver-docs-ddi
uv run python sparse.py --mode driver -o driver-docs.json
```

Pass an explicit path to override the default — useful for pointing at a checkout outside the repo:

```bash
uv run python sparse.py --mode driver -o driver-docs.json /path/to/windows-driver-docs-ddi/wdk-ddi-src/content/
```

![Expected output.](https://raw.githubusercontent.com/cristeigabriela/sparse/main/media/output.png)
> ~8 seconds to finish **43,337 entries** on `Intel(R) Core(TM) i9-14900HX`, 16c/32t, default settings.

## Example output

Real output from the parser, side-by-side. `--mode driver` adds the extra fields shown on the right (`include_header`, `target_type`, `kmdf_ver`, `umdf_ver`, `tech_root`, `irql`, `irql_raw`); the base shape is shared.

> Long string values are trimmed inline (`… ` marker) so the table fits without a horizontal scrollbar. See [`example/output.json`](./example/output.json) and [`example/output-driver.json`](./example/output-driver.json) for the full verbatim output.

<table>
<tr>
<th><code>--mode sdk</code> + <code>CreateFileW</code></th>
<th><code>--mode driver</code> + <code>WdfCollectionAdd</code></th>
</tr>
<tr>
<td valign="top">

```json
{
  "CreateFileW": {
    "header": "fileapi.h",
    "lib": "Kernel32.lib",
    "dll": "Kernel32.dll",
    "min_client_version": "Windows XP [desktop apps only]",
    "min_server_version": "Windows Server 2003 [desktop apps only]",
    "metadata": {
      "UID": "NF:fileapi.CreateFileW",
      "title": "CreateFileW function (fileapi.h)",
      "description": "Creates or opens a file or I/O device. … (Unicode)",
      "api_type": [
        "DllExport"
      ],
      "api_location": [
        "api-ms-win-core-file-l1-2-5.dll",
        "api-ms-win-core-file-l1-2-4.dll",
        "api-ms-win-core-file-l1-2-3.dll",
        "Kernel32.dll",
        "API-MS-Win-Core-File-l1-1-0.dll",
        "KernelBase.dll",
        "API-MS-Win-Core-File-l1-2-0.dll",
        "API-MS-Win-Core-File-l1-2-1.dll",
        "API-MS-Win-Core-File-l1-2-2.dll",
        "API-MS-Win-DownLevel-Kernel32-l1-1-0.dll",
        "MinKernelBase.dll"
      ],
      "api_name": [
        "CreateFile",
        "CreateFileA",
        "CreateFileW"
      ]
    },
    "params": {
      "lpFileName": {
        "directions": [
          "in"
        ],
        "values": {}
      },
      "dwDesiredAccess": {
        "directions": [
          "in"
        ],
        "values": {}
      },
      "dwShareMode": {
        "directions": [
          "in"
        ],
        "values": {
          "FILE_SHARE_DELETE": 4,
          "FILE_SHARE_READ": 1,
          "FILE_SHARE_WRITE": 2
        }
      },
      "lpSecurityAttributes": {
        "directions": [
          "in",
          "optional"
        ],
        "values": {}
      },
      "dwCreationDisposition": {
        "directions": [
          "in"
        ],
        "values": {
          "CREATE_ALWAYS": 2,
          "CREATE_NEW": 1,
          "OPEN_ALWAYS": 4,
          "OPEN_EXISTING": 3,
          "TRUNCATE_EXISTING": 5
        }
      },
      "dwFlagsAndAttributes": {
        "directions": [
          "in"
        ],
        "values": {
          "FILE_ATTRIBUTE_ARCHIVE": 32,
          "FILE_ATTRIBUTE_ENCRYPTED": 16384,
          "FILE_ATTRIBUTE_HIDDEN": 2,
          "FILE_ATTRIBUTE_NORMAL": 128,
          "FILE_ATTRIBUTE_OFFLINE": 4096,
          "FILE_ATTRIBUTE_READONLY": 1,
          "FILE_ATTRIBUTE_SYSTEM": 4,
          "FILE_ATTRIBUTE_TEMPORARY": 256,
          "FILE_FLAG_BACKUP_SEMANTICS": 33554432,
          "FILE_FLAG_DELETE_ON_CLOSE": 67108864,
          "FILE_FLAG_NO_BUFFERING": 536870912,
          "FILE_FLAG_OPEN_NO_RECALL": 1048576,
          "FILE_FLAG_OPEN_REPARSE_POINT": 2097152,
          "FILE_FLAG_OVERLAPPED": 1073741824,
          "FILE_FLAG_POSIX_SEMANTICS": 16777216,
          "FILE_FLAG_RANDOM_ACCESS": 268435456,
          "FILE_FLAG_SESSION_AWARE": 8388608,
          "FILE_FLAG_SEQUENTIAL_SCAN": 134217728,
          "FILE_FLAG_WRITE_THROUGH": 2147483648,
          "SECURITY_ANONYMOUS": null,
          "SECURITY_CONTEXT_TRACKING": null,
          "SECURITY_DELEGATION": null,
          "SECURITY_EFFECTIVE_ONLY": null,
          "SECURITY_IDENTIFICATION": null,
          "SECURITY_IMPERSONATION": null
        }
      },
      "hTemplateFile": {
        "directions": [
          "in",
          "optional"
        ],
        "values": {}
      }
    }
  }
}
```

</td>
<td valign="top">

```json
{
  "WdfCollectionAdd": {
    "header": "wdfcollection.h",
    "lib": "Wdf01000.sys (KMDF); WUDFx02000.dll (UMDF)",
    "dll": null,
    "min_client_version": null,
    "min_server_version": null,
    "metadata": {
      "UID": "NF:wdfcollection.WdfCollectionAdd",
      "title": "WdfCollectionAdd function (wdfcollection.h)",
      "description": "The WdfCollectionAdd method adds a specified framework object to an object collection.",
      "api_type": [
        "LibDef"
      ],
      "api_location": [
        "Wdf01000.sys",
        "Wdf01000.sys.dll",
        "WUDFx02000.dll",
        "WUDFx02000.dll.dll"
      ],
      "api_name": [
        "WdfCollectionAdd"
      ]
    },
    "params": {
      "Collection": {
        "directions": [
          "in"
        ],
        "values": {}
      },
      "Object": {
        "directions": [
          "in"
        ],
        "values": {}
      }
    },
    "include_header": "Wdf.h",
    "target_type": "Universal",
    "construct_type": null,
    "kmdf_ver": "1.0",
    "umdf_ver": "2.0",
    "tech_root": "wdf",
    "irql": {
      "level": "DISPATCH_LEVEL",
      "op": "<="
    },
    "irql_raw": "<= DISPATCH_LEVEL"
  }
}
```

</td>
</tr>
</table>

The `irql` object normalizes ~95% of `req.irql` strings into `{level, op}` where `level` is one of `PASSIVE_LEVEL`, `APC_LEVEL`, `DISPATCH_LEVEL`, `DPC_LEVEL`, `DEVICE_LEVEL`, `DIRQL`, `HIGH_LEVEL`, `IPI_LEVEL`, or `ANY`, and `op` is one of `<=`, `<`, `=`, `==`, `>=`, `>`, or `null`. The original string is always preserved in `irql_raw`.

## Submodules

Two pinned upstreams, both managed as git submodules:

| path | upstream | branch |
|---|---|---|
| `sdk-api/` | [MicrosoftDocs/sdk-api](https://github.com/MicrosoftDocs/sdk-api) | default (`docs`) |
| `windows-driver-docs-ddi/` | [cristeigabriela/windows-driver-docs-ddi](https://github.com/cristeigabriela/windows-driver-docs-ddi) (fork) | `integration/all-open-prs` |

> The driver-docs submodule points at a **fork**, not Microsoft's repo directly. It tracks my integration branch that bundles all my open patch PRs against [MicrosoftDocs/windows-driver-docs-ddi](https://github.com/MicrosoftDocs/windows-driver-docs-ddi) so we parse the cleanest available data. We'll switch back to upstream if/once the PRs land.

To bump both submodules to the tip of their tracked branches:

```pwsh
./scripts/update-submodules.ps1
git diff --submodule=log    # review what moved
git commit -am "chore: bump submodules"
```

## How it works

Both [sdk-api](https://github.com/MicrosoftDocs/sdk-api) and [windows-driver-docs-ddi](https://github.com/MicrosoftDocs/windows-driver-docs-ddi) keep their markdown under `*-src/content/`, split into per-header (or per-domain) folders.

We glob every file that starts with `nf-` **(Native Function)** and ends in `.md`, then run them through the [parser](./parser/) Python module.

Each file has the same general shape:
- a YAML frontmatter block (where most of the metadata lives — `req.header`, `req.lib`, `req.irql`, …)
- a function/method/macro header (`# Foo function`, `# Foo macro`, `# IBar::Baz`) — used to derive the top-level JSON key
- zero or more `## -param Name [annotations]` subheaders, sometimes containing an HTML table of constants

Driver mode is purely additive: the same parser, plus a handful of extra frontmatter keys and an IRQL normalizer (see `parser/irql.py`).

## Precedent

A long, long time ago, the person behind Hexacorn published ["Hunting for Windows API prototypes and descriptions…"](https://www.hexacorn.com/blog/2023/10/25/hunting-for-windows-api-prototypes-and-descriptions/), discussing existing projects that parse WinAPI metadata.

The following projects discussed are:
- [msdocsviewer](https://github.com/alexander-hanel/msdocsviewer) by Alexander Hanel
- [Mandiant's IDA MSDN annotations](https://cloud.google.com/blog/topics/threat-intelligence/msdn-annotations-plugin/)
- etc.

They also discuss the history of how Microsoft distributed the documentation that is today available on MSDN in the past (HLP files, CHM files most recently, etc.)

Those are particularly helpful to avoid web requests to Microsoft's MSDN, and the (even more) unreliable process of parsing the HTML generated from the Markdown they are based on.

Today the golden standard is `sdk-api` (and `windows-driver-docs-ddi` / kernel / virt / etc. friends), as was first most aptly demonstrated by msdocsviewer in our community — and by Microsoft themselves in [win32metadata](https://github.com/microsoft/win32metadata/blob/223f4b9723d8fb7c83c286b6b4ad75dff18985c4/scripts/CreateApiInfoCsv.ps1#L3).

Unfortunately, none of those generate lengthy, parsable metadata for every method.

This is why, I believe, the person behind Hexacorn went on to write their own parser (which, in its structure, is very similar to sparse).

Matter of fact, I'm actually fairly sure that they use the exact same expectations to extract data that we do, as we seem to have the same parsing limitations for common constant values.

So, in some ways, this is just an open-source way to reproduce the same outputs without needing to write your own parser for their data (like I have in the past!)

## License

[UNLICENSE](./LICENSE)

sparse

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README