NightOwlScraper

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

NightOwlScraper

★ 22 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# NightOwl Scraper

A headless web scraper built with **Playwright**, **FastAPI**, and **Next.js**.  
It crawls any URL with a real Chromium browser and exports the page as a screenshot, PDF, Markdown, or MHTML archive — all stored locally.

---
![NightOwl Scraper Screenshot](https://raw.githubusercontent.com/NightOwlDev19/NightOwlScraper/master/Screenshot.png)

## Features

- **Full-page screenshot** (PNG or JPEG)
- **PDF export** (A4, Letter, A3 with background printing)
- **Markdown extraction** — clean, readable text
- **MHTML archive** — self-contained offline web page
- **Live browser view** — watch the browser while it scrapes (via WebSocket screencast)
- **Interactive mode** — take control of the browser and trigger capture manually
- **History panel** — all past runs persisted locally, browse and download anytime
- **Stop scraping** — cancel any in-progress job immediately

---

## Project Structure

```
NightOwlScraper/
├── backend/
│   ├── main.py          # FastAPI server (REST API + WebSocket)
│   ├── .env             # Configuration (host, port, defaults)
│   └── example.env      # Template config
├── scraper/
│   ├── engine.py        # Playwright browser lifecycle & scrape logic
│   ├── models.py        # Pydantic data models
│   ├── utils.py         # URL utilities, file helpers
│   └── exporters/       # Format-specific export modules
│       ├── image.py
│       ├── pdf.py
│       ├── markdown.py
│       └── mhtml.py
├── frontend/            # Next.js UI
├── output/              # Scrape results saved here (auto-created)
├── test_scraper_engine.py   # Quick scraper demo / smoke test
├── run_tests.bat            # Launch the demo test
├── run_backend.ps1          # Start the FastAPI backend
└── run_frontend.ps1         # Start the Next.js frontend
```

---

## Requirements

- **Python 3.10+** with a virtual environment (`.venv`)
- **Node.js 18+** and npm
- **Playwright Chromium** browser

---

## Installation

### 1. Clone and set up Python environment

```powershell
cd NightOwlScraper
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
playwright install chromium
```

### 2. Install frontend dependencies

```powershell
cd frontend
npm install
```

---

## Running

Open two terminals from the project root:

**Terminal 1 — Backend (FastAPI)**
```powershell
.\run_backend.ps1
```
Starts the API server at `http://localhost:8000`.

**Terminal 2 — Frontend (Next.js)**
```powershell
.\run_frontend.ps1
```
Opens the UI at `http://localhost:3000`.

---

## Configuration

Edit `backend/.env` to change defaults:

| Variable | Default | Description |
|---|---|---|
| `HOST` | `127.0.0.1` | Server bind address |
| `PORT` | `8000` | Server port |
| `OUTPUT_DIR` | `output` | Directory where scrape results are saved |
| `DEFAULT_FORMATS` | `image,md` | Comma-separated export formats |
| `DEFAULT_VIEWPORT_WIDTH` | `1280` | Browser viewport width (px) |
| `DEFAULT_VIEWPORT_HEIGHT` | `800` | Browser viewport height (px) |
| `DEFAULT_TIMEOUT_MS` | `60000` | Navigation timeout (ms) |
| `DEFAULT_WAIT_UNTIL` | `networkidle` | Page load event: `load`, `domcontentloaded`, `networkidle` |
| `DEFAULT_FULL_PAGE` | `true` | Capture the full scrollable page |
| `DEFAULT_IMAGE_FORMAT` | `png` | `png` or `jpeg` |
| `DEFAULT_PAGE_SIZE` | `A4` | PDF page size: `A4`, `Letter`, `A3` |
| `DEFAULT_PDF_PRINT_BACKGROUND` | `true` | Include background colors/images in PDF |
| `DEFAULT_HEADLESS` | `true` | Run browser headlessly |

---

## REST API

| Method | Endpoint | Description |
|---|---|---|
| `GET` | `/api/config/defaults` | Returns current default scrape options from `.env` |
| `POST` | `/api/scrape` | Run a synchronous scrape (blocks until done) |
| `POST` | `/api/scrape/start` | Start a scrape job in the background, returns `job_id` |
| `GET` | `/api/jobs` | List all historical scrape results |
| `GET` | `/api/jobs/{job_id}` | Get a single job result |
| `GET` | `/api/jobs/{job_id}/status` | Poll status of a background job (`running` or `done`) |
| `DELETE` | `/api/jobs/{job_id}` | Delete a job and all its exported files |
| `POST` | `/api/jobs/{job_id}/trigger` | Trigger manual capture for an interactive job |
| `POST` | `/api/jobs/{job_id}/cancel` | Stop an in-progress job immediately |
| `GET` | `/api/jobs/{job_id}/mhtml-preview` | Serve the MHTML archive as inline HTML |
| `WS` | `/ws/browser/{job_id}` | WebSocket live browser view (screencast + input) |
| `GET` | `/static/{job_id}/{filename}` | Download any exported file |

### Example: POST /api/scrape

```json
{
  "url": "https://example.com",
  "options": {
    "formats": ["image", "pdf", "md"],
    "full_page": true,
    "viewport_width": 1280,
    "viewport_height": 800,
    "wait_until": "networkidle",
    "timeout_ms": 30000
  }
}
```

---

## Scraper Engine (Python API)

Use the engine directly without the web server:

```python
import asyncio
from pathlib import Path
from scraper.models import ExportFormat, ScrapeOptions, ScrapeRequest
from scraper.engine import ScraperEngine

async def main():
    req = ScrapeRequest(
        url="https://example.com",
        options=ScrapeOptions(
            formats=[ExportFormat.IMAGE, ExportFormat.MD, ExportFormat.PDF],
            full_page=True,
        ),
    )
    async with ScraperEngine() as engine:
        result = await engine.scrape(req, job_id="my_job", output_base=Path("./output"))

print(result.meta.title)
    for exp in result.exports:
        print(exp.path, exp.size_bytes)

asyncio.run(main())
```

---

## Quick Test

Run the demo script to verify the scraper is working:

```powershell
.\run_tests.bat
```

Or with a custom URL:

```powershell
.venv\Scripts\python test_scraper_engine.py https://news.ycombinator.com
```

Expected output:
```
  Scraping: https://example.com

Title   : Example Domain
  URL     : https://example.com/
  Links   : 1  |  Images: 0

[IMAGE]  example.com.png  (10.6 KB)
  [MD   ]  example.com.md   (0.3 KB)
  [PDF  ]  example.com.pdf  (20.6 KB)
```

---

## Data Models

### ScrapeRequest
| Field | Type | Description |
|---|---|---|
| `url` | `str` | Target URL to scrape |
| `options` | `ScrapeOptions` | All scraping and export configuration |

### ScrapeResult
| Field | Type | Description |
|---|---|---|
| `job_id` | `str` | Unique identifier for this scrape run |
| `url` | `str` | Final URL of the scraped page |
| `success` | `bool` | Whether the scrape completed successfully |
| `error` | `str?` | Error message if `success` is false |
| `meta` | `PageMeta?` | Page title, description, link count, etc. |
| `exports` | `ExportedFile[]` | List of all produced output files |
| `created_at` | `float?` | Unix timestamp of job creation |

### Export Formats
| Value | Description |
|---|---|
| `image` | Full-page PNG or JPEG screenshot |
| `pdf` | PDF document |
| `mhtml` | Self-contained MHTML web archive |
| `md` | Clean Markdown extraction of page content |

---

## Output Files

Each scrape job saves files to `output/<job_id>/`:

```
output/
└── example.com-a1b2c3/
    ├── result.json          # Full job metadata
    ├── example.com.png      # Screenshot
    ├── example.com.pdf      # PDF
    ├── example.com.md       # Markdown
    └── example.com.mhtml    # MHTML archive
```

NightOwlScraper

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README