parse-gateway

KO | EN

gitlite — search

#javascript #python #ai #go #machine-learning #linux #nodejs #java #windows #awesome #cli #kubernetes #vue #open-source

(★ 26)

Route every page to the right parser

parse-gateway 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Parse Gateway

Parse Gateway is a small web app that shows off a cost-aware document
parsing pipeline. Upload a PDF, and it will:

1. Score every page for complexity (text density, OCR needs, layout noise,
   embedded images, etc.) using [`@llamaindex/liteparse`](https://www.npmjs.com/package/@llamaindex/liteparse).
2. Optionally factor in **layout complexity** (multi-column text, ruled
   tables, dense figures) via the "Include Layout Complexity" switch in
   the UI, which can escalate a page's tier even if it needs no OCR at all.
3. Route each page to the cheapest parser tier that can still handle it:
   - **LiteParse (local)** — free, in-process parsing for simple, text-heavy pages.
   - **LlamaParse — Cost Effective** — for pages that need OCR but are otherwise simple.
   - **LlamaParse — Agentic** — for scanned pages, sparse text, embedded images, or moderate layout complexity (e.g. two columns).
   - **LlamaParse — Agentic Plus** — for garbled text, heavy vector-text, or pages with dense tables/figures/columns that need the most capable tier.
4. Stream progress back to the browser over Server-Sent Events (SSE) as
   complexity estimation and parsing happen.
5. Show the resulting markdown per page, with a breakdown of which tier
   handled which pages, and let you copy or download the combined result.

Under the hood it's a [TanStack Start](https://tanstack.com/start) app
(React + TanStack Router) with a single server API route
(`src/routes/api/parse.ts`) that does the complexity estimation, fans work
out across LiteParse and the [LlamaCloud](https://www.npmjs.com/package/@llamaindex/llama-cloud)
parsing API, and streams results back as SSE.

## How it works

- **Complexity estimation** — `LiteParse.isComplex()` inspects the PDF and
  returns per-page stats: OCR-related signals (`needsOcr`, `reasons`, text
  coverage, image coverage, garbled text, etc.) and, separately, layout
  signals (`layout.isComplex`, column count, ruled-table coverage, figure
  coverage). Pages are bucketed into one of four tiers by
  `classifyPageTier()` (`src/routes/api/parse.ts`):
  - Pages with no OCR reasons and no complex layout go straight to the
    cheapest tier (or local LiteParse, see below) without further scoring.
  - Otherwise, each OCR `reason` (`no-text`, `scanned`, `sparse-text`,
    `embedded-images`, `garbled`, `vector-text`) maps to a baseline tier,
    with a few reasons escalated further based on magnitude (e.g. very
    sparse text with no full-page image, or many small embedded images).
  - **Compounding escalation** bumps `agentic` up to `agentic_plus` once 3
    or more OCR reasons fire on the same page, since multiple independent
    problems compound.
  - **Layout escalation** (only applied when "Include Layout Complexity"
    is enabled) independently pushes the tier up based on layout signals
    (heavy ruled-table coverage, 2+ column layouts, dense figure coverage,
    or multiple layout issues firing together) even on pages that don't
    need OCR at all.
  - The final tier is the max across all of these signals; layout
    escalation can only raise the tier, never lower it.
- **Tiered parsing** — Each bucket of pages is parsed in parallel:
  - Pages that don't need OCR are parsed locally with LiteParse.
  - Everything else is uploaded to LlamaCloud and parsed with the
    appropriate LlamaParse tier (`cost_effective`, `agentic`, or
    `agentic_plus`), scoped to just those page numbers.
- **Streaming** — The `/api/parse` route responds with a `text/event-stream`
  body, emitting `complexity_estimation` and `parsing` events as each stage
  starts, progresses, fails, or completes. The client
  (`src/hooks/use-parse-gateway.ts`) parses these events and drives the UI.
- **Results** — Parsed pages from every tier are merged, sorted by page
  number, and rendered as markdown in the results viewer, alongside a
  breakdown of which tier handled which pages.

The "Include Layout Complexity" switch in the UI is sent to `/api/parse`
as an `include_layout` form field (`"true"`/`"false"`) and controls whether
layout escalation runs at all — leave it off to score pages purely on OCR
signals.

Your LlamaParse Platform API key is only ever stored in your browser's
local storage and sent as a bearer token on the `/api/parse` request — it
is never persisted server-side.

## Getting Started

Install dependencies and run the dev server:

```bash
pnpm install
pnpm dev
```

The app will be available at `http://localhost:3000`. You'll need a
[LlamaParse Platform API key](https://developers.llamaindex.ai/llamaparse/general/api_key/)
to parse pages that escalate beyond local LiteParse — paste it into the
API key field in the UI.

## Building For Production

```bash
pnpm build
pnpm start
```

`pnpm start` serves the production build with `vite preview` on
`$PORT` (defaults to `4173`).

## Project Structure

```
src/
  routes/
    index.tsx          # Main UI: upload, progress, results
    api/parse.ts        # SSE endpoint: complexity estimation + tiered parsing
  hooks/
    use-parse-gateway.ts # Client-side SSE consumer / state machine
    use-local-storage.ts
  components/parse-gateway/
    ApiKeyInput.tsx      # LlamaParse API key input (stored in localStorage)
    FileDropzone.tsx     # PDF upload
    PipelineStatus.tsx   # Live status of estimation/parsing stages
    ComplexityBreakdown.tsx # Per-tier page counts
    ResultsViewer.tsx    # Combined markdown viewer, copy/export
  lib/parse-gateway-types.ts # Shared types, tier labels/colors
```

## Linting & Formatting

This project uses [eslint](https://eslint.org/) and [prettier](https://prettier.io/).
Eslint is configured using [tanstack/eslint-config](https://tanstack.com/config/latest/docs/eslint).

```bash
pnpm lint
pnpm format
pnpm check
```

## Styling

Styled with [Tailwind CSS](https://tailwindcss.com/) and
[shadcn/ui](https://ui.shadcn.com/) components (`src/components/ui`). Add
new shadcn components with:

```bash
pnpm dlx shadcn@latest add <component>
```

parse-gateway

parse-gateway 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

parse-gateway

parse-gateway 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README