pdf2md

KO | EN

gitlite — search

#javascript #hacktoberfest #llm #golang #android #deep-learning #linux #macos #nodejs #awesome #vue #pytorch #mcp #open-source

(★ 57)

将 PDF 技术文档（规格书、数据手册、参考手册）批量转换为高质量 Markdown，保留表格结构、提取嵌入图片、自动清除水印。Batch convert PDF technical documents (datasheets, reference manuals, specs) to clean Markdown with table preservation, image extraction, and watermark removal.

pdf2md Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# PDF2MD — PDF to Markdown Converter

> 将 PDF 技术文档（规格书、数据手册、参考手册）批量转换为高质量 Markdown，保留表格结构、提取嵌入图片、自动清除水印。

> Batch convert PDF technical documents (datasheets, reference manuals, specs) to clean Markdown with table preservation, image extraction, and watermark removal.

[中文](#中文) | [English](#english)

---

## 中文

### 项目动机

对于需要将芯片规格书、数据手册、参考手册等 PDF 技术文档转为 Markdown 的工程师而言，Microsoft 的 [MarkItDown](https://github.com/microsoft/markitdown) 在基础文本转换方面表现不错，但在三个关键场景下存在明显短板：

1. **图片提取**——大多数 PDF 中嵌入的图片无法被 MarkItDown 提取，输出中完全没有图片引用
2. **水印处理**——保密文档中的水印文本会与正文交叠，产生大量乱码
3. **表格检测**——ARM TRM 和 Intel EDS 等文档中的无边框寄存器表格经常漏检

PDF2MD 通过三步流水线系统性解决上述问题：先在 PDF 源层面去除水印，再通过频率分析过滤页眉页脚，最后用 pdfplumber + PyMuPDF + 启发式规则三者协作完成转换。

### 核心特性

- **水印去除**：使用 `pikepdf` 在 PDF 内部结构中扫描并移除使用 `AAAAAB+Helvetica-Bold` 字体的水印 Form XObject。水印字体名称可配置。
- **页眉页脚过滤**：基于频率分析——跨页重复出现在页面顶部/底部的文本块被自动识别并过滤。
- **表格保留**：三路表格检测策略：
    - pdfplumber（主力）处理无边框表格（ARM 指令编码表、Intel 寄存器表）
    - PyMuPDF 处理有边框表格
    - 启发式后备规则处理 pdfplumber 漏检的表格（ARM 风格 `\n` 分列、Intel 风格 `\s{2,}` 分列）
- **图片提取**：提取嵌入图片到 `<文件名>_images/` 子目录。大于 3KB 的图片在 Markdown 中生成引用。
- **目录镜像**：输出目录结构与输入完全一致。
- **文字顺序**：带有浮点容差的栏感知排序——同行块 y 坐标四舍五入到 0.1pt 后按 x 从左到右排列。多栏页面通过 ≥100pt 的水平间距自动识别。
- **水印文本清理**：从混合文本块中移除已知水印词片段。

### 快速开始

```bash
# 安装依赖
pip install pikepdf PyMuPDF pdfplumber

# 将 PDF 放入 input/ 目录
mkdir -p input
cp /path/to/your/pdfs/*.pdf input/

# 运行完整流水线
python remove_watermark.py
python convert_hybrid.py

# 输出在 output_hybrid/
```

### 架构

```
input/                      # PDF 源文件（支持任意嵌套深度）
    └── Spec/
        └── Vendor/Chip/
            └── document.pdf

python remove_watermark.py  # 第一步：水印去除
    input/ → clean_pdfs/

python convert_hybrid.py    # 第二步+第三步：过滤 + 转换
    clean_pdfs/ → output_hybrid/
        └── Spec/Vendor/Chip/
            ├── document.md
            └── document_images/
                ├── page0001_img000.png
                └── page0011_img001.jpeg
```

### 处理流水线

| 步骤 | 脚本 | 引擎 | 作用 |
|------|------|------|------|
| 1 | `remove_watermark.py` | pikepdf | 从 PDF 源码层面移除水印 XObject |
| 2 | 内置 | 频率分析 | 检测并过滤跨页重复的页眉/页脚/水印文本 |
| 3 | `convert_hybrid.py` | pdfplumber + PyMuPDF + 启发式 | 提取文字、表格、图片，组装 Markdown |

### 表格检测详解

**ARM 风格（单块多行）**

表格的每一行是一个 PyMuPDF 文本块，列间以 `\n` 分隔。取连续块的列数众数确定表格列数，超出列以 `<br>` 合并入末列。

**Intel 风格（多块空格分列）**

列间以 `\s{2,}` 分隔的多个相邻文本块。要求连续 3 行以上且列数一致才判定为表格，以避免正文中多栏文本被误判。

### 文字排序

PyMuPDF 以浮点数报告块坐标，同行块可能相差不足 0.001pt。PDF2MD 将 y 坐标四舍五入到 0.1pt 后再排序，确保同行块按 x 坐标从左到右正确排列。

多栏页面通过检测正文块之间 ≥100pt 的水平间距自动识别，按「先栏后行」排序。

### 依赖

- Python 3.10+
- pikepdf ≥ 10.0
- PyMuPDF ≥ 1.24
- pdfplumber ≥ 0.11

### 局限性

- **加密 PDF**：无法处理水印去除（原样复制），后续转换不受影响
- **字符级水印**：水印与正文在 PDF 渲染层面字符级交织的场景（如贯穿全文的斜向层叠文本），可能有残留乱码
- **跨页表格**：跨越页面边界的寄存器表，续页上会重复表头行
- **矢量图**：方框图、流水线图等矢量图形不会被提取为图片
- **扫描件 PDF**：不支持（需要 OCR）

### 许可

MIT

---

## English

### Motivation

Converting PDF technical documents to Markdown is a common pain point for engineers who need to search, diff, or feed chip specifications into LLMs. Existing tools like Microsoft's [MarkItDown](https://github.com/microsoft/markitdown) handle basic text well but fall short on three critical fronts:

1. **Image extraction** — MarkItDown produces zero images from most PDFs
2. **Watermark handling** — Confidential watermarks interleave with content, producing garbled output
3. **Table detection** — Borderless register tables (common in ARM TRMs and Intel EDS documents) are missed

PDF2MD addresses all three through a three-step pipeline: watermark removal → header/footer filtering → hybrid extraction (pdfplumber + PyMuPDF + heuristics).

### Features

- **Watermark removal**: Strips `AAAAAB+Helvetica-Bold` font watermarks at the PDF source level using `pikepdf`. Customizable font name.
- **Header/footer filtering**: Frequency-based analysis detects repeated page headers/footers and filters them before text extraction.
- **Table preservation**: Dual-strategy table detection:
    - *pdfplumber* for borderless tables (ARM TRM instruction encodings, Intel register tables)
    - *PyMuPDF* for bordered tables
    - *Heuristic fallback* for ARM-style (newline-separated) and Intel-style (whitespace-separated) patterns
- **Image extraction**: Extracts embedded images to a `*_images/` subdirectory. Images larger than 3KB are referenced in the Markdown output.
- **Directory mirroring**: Output directory structure exactly mirrors the input structure.
- **Text ordering**: Column-aware sorting with floating-point-tolerant y-coordinate alignment ensures correct reading order.
- **Watermark text cleaning**: Removes known watermark substrings from mixed content blocks.

### Quick Start

```bash
# Install dependencies
pip install pikepdf PyMuPDF pdfplumber

# Prepare PDFs in input/
mkdir -p input
cp /path/to/your/pdfs/*.pdf input/

# Run full pipeline
python remove_watermark.py
python convert_hybrid.py

# Output in output_hybrid/
```

### Architecture

```
input/                      # Place PDF files here (any nesting depth)
    └── Spec/
        └── Vendor/Chip/
            └── document.pdf

python remove_watermark.py  # Step 1: Watermark removal
    input/ → clean_pdfs/

python convert_hybrid.py    # Steps 2-3: Filtering + conversion
    clean_pdfs/ → output_hybrid/
        └── Spec/Vendor/Chip/
            ├── document.md
            └── document_images/
                ├── page0001_img000.png
                └── page0011_img001.jpeg
```

### Processing Pipeline

| Step | Script | Engine | Purpose |
|------|--------|--------|---------|
| 1 | `remove_watermark.py` | pikepdf | Remove Form XObject watermarks from PDF source |
| 2 | built-in | frequency analysis | Detect and filter repeated headers/footers/watermarks |
| 3 | `convert_hybrid.py` | pdfplumber + PyMuPDF + heuristics | Extract text, tables, and images; assemble Markdown |

### Table Detection in Detail

**ARM-style (single-block, multi-line rows)**

Each table row is a PyMuPDF text block with `\n`-separated column cells. The modal column count across consecutive blocks determines the table structure. Overflow cells (multi-line detail text) are merged into the last column with `<br>` separators.

**Intel-style (multi-block, spaced-field rows)**

Columns separated by `\s{2,}` across multiple adjacent text blocks. Requires ≥3 consecutive rows with consistent column counts to avoid false positives.

### Text Ordering

PyMuPDF reports block coordinates with floating-point precision. Same-line blocks may differ by ≤0.001pt in y-coordinate. PDF2MD rounds y-coordinates to 0.1pt before sorting, ensuring visually same-line blocks sort left-to-right by x-coordinate.

Multi-column layouts are detected by finding ≥100pt x-gaps between body text blocks, then sorting column-first, top-to-bottom within each column.

### Requirements

- Python 3.10+
- pikepdf ≥ 10.0
- PyMuPDF ≥ 1.24
- pdfplumber ≥ 0.11

### Limitations

- **Encrypted PDFs**: Cannot be processed for watermark removal (copied as-is)
- **Character-level watermark**: PDFs with watermarks interleaved at the character rendering level (e.g., diagonal overlay text) may have residual garbled output
- **Page-spanning tables**: Register tables that cross page boundaries may have header rows repeated on continuation pages
- **Vector graphics**: Block diagrams, pipeline diagrams, and other vector art are not extracted as images
- **Scanned PDFs**: Not supported (requires OCR)

### License

MIT

---

*Generated with [Claude Code](https://claude.ai/code)*

pdf2md

pdf2md Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

pdf2md

pdf2md Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README