cloudflare-scraper-openai

KO | EN

gitlite — search

#python #hacktoberfest #ai #typescript #llm #go #machine-learning #macos #nodejs #awesome #cli #mcp #ios #openai

cloudflare-scraper-openai

(★ 10)

Combine a scraper with a Cloudflare Worker and OpenAI's API and you get a decently powerful scraper for cheap.

cloudflare-scraper-openai 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# AI Scraping Hub Project

## Overview

This project includes a Node.js Scheduler Service, two Cloudflare Workers (a scraper and an Apollo GraphQL server), and uses Neon PostgreSQL, Redis via Upstash, and Render.com for deployment. The application is designed to create and schedule web scraping jobs, execute them, and store and serve the results.

## Tech Stack

- Node.js Scheduler Service: A Node.js application responsible for polling the Neon PostgreSQL database every 15 minutes to retrieve the latest cron schedule based on the scraping_job table.
- Cloudflare Worker 1 - Scraper: This Cloudflare worker is responsible for executing the web scraping tasks.
- Cloudflare Worker 2 - Apollo GraphQL Server: This Cloudflare worker provides a GraphQL API for creating, reading, and updating scraping cron jobs.
- Neon PostgreSQL: The serverless PostgreSQL database used for storing the scraping job schedules and historical job results.
- Redis with Upstash.com: Used for caching.
- Render.com: Used for deploying the Node.js Scheduler Service.

## Using Docker Compose

In the root directory of the project, you will find a `docker-compose.yml` file. This file is used to define and run the Scheduler Service in a Docker container.

To start the service, run the following command:

```
docker-compose up
```

Make sure Docker is installed on your machine before running the command.

## Cloudflare Workers

Refer to the individual README.md files in the Cloudflare worker directories for specific instructions on their usage.

## Billing

Here's a quick summary of the monthly costs associated with the services used in this project:

| Service             | Cost    |
|---------------------|---------|
| Render.com          | $7/mo   |
| Cloudflare Workers  | Free tier    |
| Redis (Upstash.com) | Free tier    |
| Neon Serverless PostgreSQL | Free tier    |
| **Total**           | **$7/mo** |

## System Architecture

The following diagram describes how the services interact:

```
Cloudflare Worker 1 (Scraper) <-------- Node.js Scheduler Service -------> Neon PostgreSQL
   ^                                       ^                                      |
   |                                       |                                      v
   |                                       |                                 Redis (Cache)
   |                                       |
Cloudflare Worker 2 (GraphQL Server) <----|
```

In this architecture:

- The Node.js Scheduler Service regularly checks the Neon PostgreSQL database for new scraping jobs.
- When a new job is found, it sends a request to the Scraper Cloudflare worker to execute the job.
- The Scraper worker retrieves the web data and sends it back to the Scheduler Service.
- The Scheduler Service then updates the job status and result in the Neon PostgreSQL database, and stores the result in Redis for caching.
- The GraphQL Server Cloudflare worker provides an API for managing the scraping jobs and accessing the latest job results.

cloudflare-scraper-openai

cloudflare-scraper-openai 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

cloudflare-scraper-openai

cloudflare-scraper-openai 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README