KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

GitLLMTrainer

★ 27 Open GitHub ↗

A tool to crawl GitHub repositories, extract their content, and generate structured datasets for training LLMs.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# GitHub Repository Crawler

![Project Banner](https://github.com/user-attachments/assets/d619ab42-6544-4252-8d65-a9547dde9e61)

## Overview
The **GitHub Repository Crawler** is a Python-based tool that enables users to scrape the contents of public GitHub repositories. It recursively fetches files, directories, and their raw contents, saving the data in a structured JSON format. The tool is especially useful for developers, researchers, and analysts seeking to analyze repository content programmatically.
The data gathered from this app can be used to train large language models (LLMs) to enhance their knowledge and capabilities.

## Demo
https://github.com/user-attachments/assets/b8a0a88d-f834-49bb-9d14-f0c57dd9f89a

![image](https://github.com/user-attachments/assets/52f882a5-e928-410a-a800-a008ee53f5c6)

## Features
- **Recursively Scrape Repositories**: Extract all files and folders from a given repository.
- **Fetch Raw File Content**: Retrieve and store the raw content of files.
- **Structured JSON Output**: Save the scraped data in an organized and human-readable JSON format.
- **Train your LLM Models**: Use the scraped data to train your models.

## Table of Contents
1. [Getting Started](#getting-started)
2. [Project Structure](#project-structure)
3. [Installation](#installation)
4. [Usage](#usage)
5. [Examples](#examples)
6. [Coming Soon](#coming-soon)
8. [Contributing](#contributing)
9. [License](#license)

## Getting Started
Follow the steps below to set up and use the GitHub Repository Crawler on your local machine.

### Prerequisites
- Python 3.10.6 or +
- GitHub API token (optional, for private repositories and to avoid rate limit(you have more requests))

### Project Structure
```
repo/
├── src/
│   ├── crawler/
│   │   ├── github_api.py
│   │   ├── json_formatter.py
│   │   ├── logger.py
│   │   ├── scraper.py
│   └── utils/
│       ├── helpers.py
├── tests/
│   ├── test_github_api.py
│   ├── test_scrapper.py
├── requirements.txt
├── setup.py
└── main.py
```

## Installation
### Step 1: Clone the Repository
```bash
git clone https://github.com/sgm1018/GitLLMTrainer.git
cd GitLLMTrainer
```

### Step 2: Set Up a Virtual Environment
```bash
python -m venv venv
source venv/bin/activate # On Windows use `venv\Scripts\activate`
```

### Step 3: Install Dependencies
```bash
pip install -r requirements.txt
```

## Usage
### Run the Tool
```bash
python src/main.py <GitHub repository URL>
```
### Parameters
- `<GitHub repository URL>`: URL of the GitHub repository to scrape (e.g., `https://github.com/sgm1018/GitLLMTrainer`).

### Example Output
- The data will be saved in `repo_data.json`.
```json
[
  {
    "path": "src/crawler/logger.py",
    "type": "file",
    "content": "import logging\nfrom colorlog ..."
  },
  ...
]
```

## Examples
To scrape the contents of the `sgm1018/GitLLMTrainer` repository:
```bash
python src/main.py https://github.com/sgm1018/GitLLMTrainer
```
Expected output:
```
INFO: Starting scraping for https://github.com/sgm1018/GitLLMTrainer
INFO: File scraped: /README.md
INFO: Entering directory: /src
...
INFO: Scraping completed for https://github.com/sgm1018/GitLLMTrainer
INFO: Data saved to repo_data.json
```

## Use Cases
The structured JSON output generated by this tool can be used in advanced scenarios such as:

- **Training Custom Models**: Use the JSON data to fine-tune custom OpenAI models by converting repository content into meaningful datasets.
- **Knowledge Integration for Private Models**: The extracted data can serve as a knowledge base for private AI models, enabling personalized applications for your projects.

## Best Practices
- Use a GitHub API token to avoid rate limits.
- Run the tool in a virtual environment.

## Coming Soon:
- Web App

## Contributing
Contributions are welcome!

## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.

## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=sgm1018/GitLLMTrainer&type=Date)](https://star-history.com/#sgm1018/GitLLMTrainer&Date)

---

GitLLMTrainer

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README