moviemod-scraper

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

moviemod-scraper

★ 16 Open GitHub ↗

A Python-based web scraper for extracting movie/series details, download links, and images from MoviesMod. Uses Selenium, multiprocessing, MySQL, and Telegram for notifications. Stores data in a database and sends real-time updates. Perfect for automating media scraping tasks.

다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# MovieMod Scraper

A Python-based web scraper for extracting movie and series details, including download links and images, from the [MoviesMod](https://moviesmod.red/) website. The project uses **Selenium** for web scraping, **multiprocessing** for parallel execution, **MySQL** for data storage, and **Telegram** for real-time notifications.

---

![vlcsnap-2025-01-16-12h42m26s612](https://github.com/user-attachments/assets/c6a9e319-b6dd-44b7-a3cd-859c59888cab)

**Demo video sped up 4x.**
- note: code is run on headfull mode for demostration. code in repo is in headless mode

https://github.com/user-attachments/assets/8ab58885-8724-4059-9793-96fea4f3d916

## Features

- **Web Scraping**: Extracts movie/series details (titles, descriptions, image URLs, and download links).
- **Multiprocessing**: Scrapes multiple pages simultaneously for faster execution.
- **Database Integration**: Stores scraped data in a MySQL database.
- **Telegram Notifications**: Sends real-time updates and error alerts to a Telegram group.
- **Retry Mechanism**: Handles connection errors and timeouts gracefully.
- **Headless Mode**: Runs Selenium in headless mode for efficient scraping.

---

## Prerequisites

- Python 3.x
- MySQL Server
- ChromeDriver (for Selenium)

---

## Setup

1. **Clone the Repository**:
   ```bash
   git clone https://github.com/barryallen16/moviemod-scraper.git
   cd moviemod-scraper
   ```

2. **Install Dependencies**:
   - Install the required Python libraries using `requirements.txt`:
     ```bash
     pip install -r requirements.txt
     ```

3. **Set Up Environment Variables**:
   - Create a `.env` file and add the following:
     ```plaintext
     TELEGRAM_BOT_TOKEN=your-telegram-bot-token
     TELEGRAM_GROUP_CHAT_ID=your-telegram-group-chat-id
     DB_HOST=your-database-host
     DB_USER=your-database-username
     DB_PASSWORD=your-database-password
     DB_NAME=your-database-name
     NUM_PROCESSES=2
     START_PAGE=1
     END_PAGE=5
     MOVIEMOD_BASE_URL=https://moviesmod.red/
     ```

4. **Set Up MySQL Database**:
   - Create the required tables using the provided SQL schema:
     ```sql
     CREATE DATABASE moviemod;

USE moviemod;

CREATE TABLE movies (
         id INT AUTO_INCREMENT PRIMARY KEY,
         image_url TEXT,
         caption TEXT,
         org_links TEXT
     );

CREATE TABLE series (
         id INT AUTO_INCREMENT PRIMARY KEY,
         image_url TEXT,
         movie_descrp TEXT,
         links TEXT,
         org_links TEXT
     );

CREATE TABLE ongoing (
         id INT AUTO_INCREMENT PRIMARY KEY,
         serieslink TEXT,
         episodelinks TEXT
     );

CREATE TABLE zip (
         id INT AUTO_INCREMENT PRIMARY KEY,
         image_url TEXT,
         links TEXT,
         org_links TEXT
     );
     ```

5. **Run the Script**:
   ```bash
   python moviescraper.py
   ```

---

## Configuration

- **Environment Variables**:
  - `TELEGRAM_BOT_TOKEN`: Your Telegram bot token.
  - `TELEGRAM_GROUP_CHAT_ID`: The chat ID of the Telegram group for notifications.
  - `DB_HOST`, `DB_USER`, `DB_PASSWORD`, `DB_NAME`: MySQL database credentials.
  - `NUM_PROCESSES`: Number of processes for multiprocessing.
  - `START_PAGE`, `END_PAGE`: Range of pages to scrape.
  - `MOVIEMOD_BASE_URL`: Base URL of the MoviesMod website.

- **ChromeDriver**:
  - Ensure ChromeDriver is installed and added to your system's PATH.
---
## Selenium setup in Cloud

```bash
apt update \
&& apt install -y wget unzip \
&& wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb \
&& apt-get install -y -f ./google-chrome-stable_current_amd64.deb \
&& wget https://storage.googleapis.com/chrome-for-testing-public/149.0.7827.155/linux64/chromedriver-linux64.zip \
&& unzip chromedriver-linux64.zip \
&&  mv  chromedriver-linux64/chromedriver /usr/local/bin/ \
&&  google-chrome --version \
&& chromedriver --version \ 
```

---
## License

This project is licensed under the **GNU General Public License v3.0**. See the [LICENSE](LICENSE) file for details.

moviemod-scraper

다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

moviemod-scraper

다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README