focal-harvest

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 20 Open GitHub ↗

An interactive CLI research assistant that scrapes, sanitizes, and synthesizes web content locally or via Gemini/OpenAI/Claude, with Discord/Telegram alerts & built-in automation.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# <img src="docs/assets/focal-harvest.png" width="38" valign="middle"> Focal Harvest

Focal Harvest is a lightweight Python CLI tool with two primary use-cases, featuring a zero-config local statistical fallback for offline operation:
* **Automated Research Pipeline**: Query search engines, scrape and clean target pages, and synthesize findings into structured reports using Gemini, Claude, or GPT-4o-mini.
* **OSINT Collection Framework**: Monitor topics across Reddit, Hacker News, Stack Overflow, arXiv, and 17+ platforms with daemon scheduling, Wayback Machine fallbacks, and Discord/Telegram alert dispatch.

[![GitHub Release](https://img.shields.io/github/v/release/techno-neighbour/focal-harvest?color=blue)](https://github.com/techno-neighbour/focal-harvest)
[![Documentation](https://img.shields.io/badge/docs-Docusaurus-blueviolet)](https://techno-neighbour.github.io/focal-harvest/)
[![License](https://img.shields.io/github/license/techno-neighbour/focal-harvest?color=green)](https://github.com/techno-neighbour/focal-harvest/blob/main/LICENSE)
[![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/)

![Focal Harvest Demo](https://raw.githubusercontent.com/techno-neighbour/focal-harvest/main/docs/assets/demo.gif)

```
Python • AI Agent • Web Research • OSINT • Multi-Engine • 3-Tier Archive Cascade • Zero-Config Offline Fallback
```

---

## 🚀 What's New in v1.6.0

* **📱 Bidirectional Remote Mobile Bot Listener (Telegram & Discord)**: Message your Focal Harvest instance directly from your mobile device via Telegram or Discord slash commands (`/research`, `/deep`, `/url`, `/status`, `/help`). Supports single-word and quoted multi-word topics (e.g. `/research "robotics engineering"`), multi-URL deep dives (e.g. `/url 'https://site1.com', 'https://site2.com'`), custom focus areas, User ID authorization whitelisting, compact mobile summaries, and native `.pdf` and `.md` report uploads. See [BOT_LISTENER_GUIDE.md](file:///D:/focal-harvest/BOT_LISTENER_GUIDE.md) for full setup instructions.
---

## ⚡ Quick Start (Get Started in 30 Seconds)

### Installation
Ensure you have Python 3.12+ installed. Run the platform-independent installer:
```bash
python install.py
```
*(Or run `pip install -r requirements.txt` for standard manual installation).*

### Minimal Working Example (CLI Launch)
Launch the interactive Terminal User Interface (TUI):
```bash
python main.py
```

### Programmatic Python API Usage
For custom automation scripts, you can run the scraping and synthesis pipeline programmatically in Python:
```python
from scraper import search_duckduckgo, scrape_urls_concurrently
from analyzer import synthesize_topics

# 1. Search for a query
results = search_duckduckgo("Gemini 1.5 Flash vs Pro", max_results=3)
urls = [r["url"] for r in results]

# 2. Scrape target pages concurrently in parallel threads
scraped_data = scrape_urls_concurrently(urls)

# 3. Synthesize the findings into a structured Markdown report
report = synthesize_topics(
    scraped_data, 
    query="Gemini 1.5 Flash vs Pro", 
    spec_topic="pricing comparisons"
)
print(report)
```

---

## 🎯 The Problem & Our Solution

Every developer, researcher, and tech blogger repeats the same tedious manual workflow:
`Search Query ➔ Open 20 Browser Tabs ➔ Ignore Cookie Walls & Ads ➔ Copy-Paste Snippets ➔ Synthesize with LLM ➔ Save Report`

**Focal Harvest** automates this entire process inside a lightweight, terminal-based pipeline.

### 📊 Performance Benchmarks (Typical Runs)

---

## ⚙️ How It Works

```mermaid
graph LR
    A["Focus Query"] --> B["Concurrent Scraper (utils.py)"]
    B --> C["Readability Engine (scraper.py)"]
    C --> D["Synthesis Engine (analyzer.py)"]
    D --> E["TUI Reports / Webhooks (notifier.py)"]
    
    style A fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px,color:#000000
    style E fill:#e8f5e9,stroke:#4caf50,stroke-width:2px,color:#000000
```

1. **Orchestrated Search**: Resolves search parameters via Tavily or DuckDuckGo.
2. **Concurrent Fetching**: Scrapes pages in parallel threads with Cloudflare bypasses.
3. **HTML Sanitization**: Stubs out ads, widgets, and cookie overlays, keeping only clean prose.
4. **Adaptive Replenishment**: Drops blocked/empty pages and automatically fetches replacements to ensure high-density inputs.
5. **AI/Local Synthesis**: Summarizes content using Gemini, Claude, or OpenAI with automatic key failover. Automatically falls back to an offline PositionRank statistical ranker if no API keys are provided.
6. **Instant Dispatch**: Saves structured Markdown/JSON reports locally and pushes alerts to Discord/Telegram.

![CLI in Action](https://raw.githubusercontent.com/techno-neighbour/focal-harvest/main/docs/assets/notification.jpeg)

---

## 📋 Example Output

Here is a real example of the structured Markdown report generated when researching **"Gemini 1.5 Flash vs Gemini 1.5 Pro"**:

![Report Preview](https://raw.githubusercontent.com/techno-neighbour/focal-harvest/main/docs/assets/report.png)

---

## ✨ Features

### 📱 Bidirectional Remote Mobile Bot Listener
* **Remote Mobile Slash Commands**: Control your Focal Harvest instance from your mobile device via Telegram or Discord slash commands (`/research`, `/deep`, `/url`, `/status`, `/help`). See [BOT_LISTENER_GUIDE.md](BOT_LISTENER_GUIDE.md) for full setup.
* **Strict Whitelist Protection**: Enforces user ID authorization whitelisting (`discord_allowed_user_ids` & `telegram_allowed_user_ids`) to block unauthorized access.
* **Dual Document Delivery**: Receives a compact mobile summary preview along with both **`.pdf`** and **`.md`** report document file uploads.

### 🎨 Dynamic Adaptive Domain Categories
* **Intent-Aware Categorization**: AI synthesis automatically classifies queries into specialized 3-section frameworks tailored to the domain:
  * **Software & Tech**: Architecture & System Design ➔ Performance & Benchmarks ➔ Tradeoffs & Best Practices
  * **Entertainment & Media**: Overview & Plot Synopsis ➔ Production & Creative Execution ➔ Critical Reception & Impact
  * **Business & Finance**: Industry Overview & Market ➔ Business Model & Financial Metrics ➔ Risk Factors & Strategic Outlook
  * **Science & Healthcare**: Background & Hypotheses ➔ Methodology & Data Analysis ➔ Practical Applications & Future Scope
  * **News & History**: Context & Timeline ➔ Key Developments & Main Findings ➔ Broader Implications & Impact
  * **General / Informational**: Overview & Core Concepts ➔ Detailed Breakdown ➔ Key Takeaways

### 🔍 Multi-Engine Search Replenishment & 3-Tier Web Archive Cascade
* **Multi-Engine Cascading**: Automatically detects search result shortfalls and cascades across **Tavily → DuckDuckGo → Bing HTML → Brave Search → Google Aggregated** with URL deduplication.
* **3-Tier Web Archive Cascade**: Recovers blocked or deleted pages by cascading through **Internet Archive Wayback Machine → Memento Web Federated API (Library of Congress & global archives) → Archive.today**.

### 🛡️ Resilient Web Scraping & Proxy Integration
* **HTTP/SOCKS5 Proxy Support**: Full proxy configuration with password masking (`***`) and Option 16 live diagnostic tester querying `api.ipify.org`.
* **Optional 403 Bypass (`curl_cffi`)**: Impersonates standard Chrome TLS/JA3 signatures and client hints to bypass Akamai and Cloudflare anti-bot blocks.
* **Wayback Machine Fallback**: Automatically redirects protected SPAs (Reddit, Stack Overflow, Quora) to the Internet Archive to bypass login gates.
* **Conditional User-Agent Routing**: Automatically swaps to `Discordbot/2.0` when scraping guest pages, and shifts to a desktop Chrome UA when user cookies are configured to avoid bot-signature blocks.
* **Unified Cookie Ingestion**: Supports Netscape `config/cookies.txt` files and universal domain-to-cookie maps to scrape pages behind login walls.

### 🧹 17+ Built-In Custom Plugins
Pre-configured parser modules designed to clean and extract structured layouts:
* **HN & Reddit**: Captures nested comments threads and recursive tree hierarchies.
* **Stack Overflow**: Cleans code blocks and identifies accepted answers.
* **arXiv & Google Scholar**: Extracts academic metadata, abstracts, and authors metrics.
* **Finance (Yahoo/SEC EDGAR)**: Scrapes financial streams and converts data tables to Markdown tables.
* **ReadTheDocs & Dev.to**: Strips navigation sidebars and headers.

### 🧠 Smart Synthesis & AI Routing
* **Multi-Provider Failover**: Cascades automatically across API keys (Gemini ➡️ OpenAI ➡️ Claude) before dropping to the offline keyword PositionRanker.
* **Live Search Grounding**: Integrates Gemini's live Google search grounding to retrieve cited real-time facts directly in the model.
* **Local-First Cache**: Saves MD5-hashed results in `reports/cache/` to skip network requests on recurring topics.

### 📄 Multi-Format Exporters & Document Delivery
* **Styled ReportLab PDF**: Exports beautiful PDF documents with custom headers, callout boxes, and cell-normalized tables.
* **Microsoft Word (.docx)**: Generates structured DOCX reports formatted with headers and source tables.
* **Markdown & JSON**: Generates clean raw Markdown reports and JSON data payloads for programmatic pipelines.

---

## 🛠️ Codebase Architecture & File Structure

### System Execution Workflow

```mermaid
sequenceDiagram
    participant User
    participant CLI as main.py
    participant Scraper as scraper.py
    participant AI as analyzer.py
    participant Notifier as notifier.py
    
    User->>CLI: Selects Option 1 (Single Scrape)
    CLI->>Scraper: Search Query & Crawl Targets
    Scraper->>Scraper: Run concurrent thread scrapes
    Scraper->>CLI: Return clean list of page contents
    CLI->>AI: Synthesize contents (AI or Local)
    AI->>CLI: Return structured report
    CLI->>Notifier: Export Markdown/JSON & Send Webhooks
    Notifier->>User: Renders Report in Terminal + sends Alerts
```

### Folder Layout

```text
├── install.py            # Platform-independent setup script
├── main.py               # Interactive TUI Controller (Version v1.6.0)
├── config_manager.py     # Reads and writes config.json
├── scraper.py            # Multi-engine search replenishment, 3-tier archive cascade, crawler
├── analyzer.py           # Multi-LLM provider wrappers, dynamic category synthesis, and PositionRank
├── notifier.py           # Markdown/JSON/PDF/DOCX exporter and Discord/Telegram webhook dispatch
├── utils.py              # Proxy manager, URL slug topic extractor, safe_request retries
├── std_plugins/          # Built-in plugins (Hacker News, Reddit, Stack Overflow, YouTube, etc.)
└── tests/                # Isolated unit tests suite (123 unit tests, 100% green)
```

---

## ⚖️ The 5 Strict Design Boundaries (Do NOT Violate)
Focal Harvest adheres to the following core maintainer guardrails:
1. **Lightweight Footprint**: Application remains under 5MB and installs in seconds.
2. **Strict TUI Only**: Runs entirely in the console—no local React dashboards or local REST servers.
3. **Low Hardware Requirements**: Runs smoothly on low-spec student laptops.
4. **No Local AI Downloads**: Uses remote APIs (Gemini/OpenAI/Claude) to bypass multi-gigabyte Ollama/Llama downloads.
5. **No SQL Databases**: Stores reports in plain JSON and Markdown files for absolute filesystem transparency.

---

## ⚖️ Legal Disclaimer & Responsible Use
Focal Harvest is a command-line utility for personal and research purposes.
* **Public Scraping**: Programmatic scraping of public data is legally protected under established U.S. case law. Respect target servers by running crawls with polite rates.
* **Authenticated Scraping**: Do not use active personal accounts with session cookies. Use burner/dedicated accounts exclusively to prevent account suspensions.

---

## 📜 License
Distributed under the MIT License. See `LICENSE` for details.

focal-harvest

Today's picks

Was this content helpful?

Recent Feedback

focal-harvest

Today's picks

Was this content helpful?

Recent Feedback

Download README