DiscovAI-crawl

KO | EN

gitlite — search

#javascript #hacktoberfest #typescript #llm #machine-learning #rust #deep-learning #windows #awesome #kubernetes #pytorch #security #openai #open-source

(★ 19)

🕷️ DiscovAI Crawl API(🚧 Work in Progress 🚧): A powerful web scraping solution for AI tools and vector databases. Extract clean HTML, generate LLM-friendly content, and create embeddings from any URL.

DiscovAI-crawl 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DiscovAI Crawl API 🕷️🔍

> One API to scrape everything you need from URLs for your AI tool and vector database.

🚧 **Work in Progress** 🚧

## 🌟 Features

Our API provides a comprehensive suite of data extraction and processing capabilities:

- 🧼 Clean HTML (JavaScript and CSS removed)
- 📝 LLM-friendly Markdown conversion
- 🚫 Ad-free, cookie banner-free, and dialog-free content
- 📸 Website screenshots (auto-saved to AWS S3 or Cloudflare R2)
- 🤖 LLM-generated SEO-friendly content
- 🔑 LLM-extracted key information (summary, features, FAQs, etc.)
- 🧠 Ready-to-use embeddings for vector database integration (auto-saved to db)

## 🔧 Installation

```bash
pnpm i
cd apps/api && pnpm exec playwright install
```

## 🚀 Usage

```bash
pnpm dev
open http://localhost:3000
```

## 📦 API Response Structure

```json
{
  "clean_html": "...",
  "LLM_friendly_markdown": "...",
  "clean_text": "...",
  "screenshot_url": "...",
  "llm_extracts_key_info": {
    "what": "...",
    "summary": "...",
    "features": ["...", "..."],
    "faqs": [{"q": "...", "a": "..."}]
  },
  "llm_summarized_detail": "...",
  "embeddings": [...]
}
```

## 📚 Documentation

TODO

## 🤝 Contributing

TODO

DiscovAI-crawl

DiscovAI-crawl 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

DiscovAI-crawl

DiscovAI-crawl 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README