gpt4all-datalake - GitLite

KO | EN

gitlite — search

#react #golang #android #rust #deep-learning #linux #docker #macos #cli #kubernetes #security #openai #ios #open-source

gpt4all-datalake

(★ 398)

API to the GPT4All Datalake

File Explorer

Download Latest Version (.zip)

- - __init__.py
  - docs.py
  - main.py
- - build.sh
  - push_ecr.sh
- .gitignore
- Dockerfile.buildkit
- requirements.txt
- ingestor.py
- map_gpt4all_datalake.py
- README.md
.gitignore
docker-compose.yaml
LICENSE
makefile
README.md

# Installation Guide

1. Get the code

git clone https://github.com/nomic-ai/gpt4all-datalake

Downloads the entire project code from GitHub to your computer.

cd gpt4all-datalake

Moves into the project folder you just downloaded.

2. Docker

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Docker Desktop Needed to build and run containers. Install it and keep it running in the background.

docker compose up -d --build

Builds and starts all defined containers (server, database, etc.) at once, in the background.

✅ Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

3. Python

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r api/requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

4. Make

Medium

Prerequisites

Git Needed to download the project code from GitHub.
Make Usually pre-installed on Linux/macOS. On Windows, install separately (e.g. via MSYS2 or WSL).

make

Compiles the code based on the generated build configuration to produce an executable.

✅ If it finishes without errors, it worked. Try running the generated executable directly.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# gpt4all-datalake
An open-source datalake to ingest, organize and efficiently store all data contributions made to gpt4all.

Hosted version: https://api.gpt4all.io

### Architecture
The core datalake architecture is a simple HTTP API (written in FastAPI) that ingests JSON in a fixed schema, performs some integrity checking and stores it. This JSON is transformed into storage efficient Arrow/Parquet files and stored in a target filesystem.

#### Data formats
- Data is stored on disk / S3 in parquet files in subdirectories organized by day. These parquet files have a standardized schema allowing for easy manipulation in any programming language.
- The input data model can be found [here](api/app/api_v1/models/models.py).

### Open sourcing the data.
Nomic AI will provide automatic snapshots of this raw parquet data.
You will be able to interact with the snapshots:
- In their raw exported form.
- In automatic [Atlas](https://atlas.nomic.ai/) maps over its raw, cleaned and curated form.
- Through downloads where the data has been curated, de-duplicated and cleaned for LLM training/finetuning.

### Data Privacy
By sending data to the GPT4All-Datalake you agree to the following.

Data sent to this datalake will be used to train open-source large language models and released to the public.
There is no expectation of privacy to any data entering this datalake. You can, however, expect attribution. If you attach a unique identifier
that associates you as the data contributor, Nomic will retain that identifier in any LLM trains that it conducts.
You will receive credit and public attribution if Nomic releases any model trained on your submitted data.
You can also submit data anonymously.

### Where does the gpt4all-datalake run?
While open-sourced under an Apache-2 License, this datalake runs on infrastructure managed and paid for by Nomic AI.
You are welcome to run this datalake under your own infrastructure! We just ask you also release the underlying data
that gets sent into it under the same attribution terms.

### Development
1. Clone down the repository.
2. Run `make testenv` to build all docker images and launch the HTTP server.
3. Go to 'http://localhost/docs' to view the API documentation.
4. You can run the unit tests with `make test`. Any edits made to the FastAPI app will hot reload.