data-universe

KO | EN

gitlite — search

#javascript #python #react #ai #llm #go #deep-learning #linux #windows #vue #pytorch #openai #mcp #open-source

(★ 77)

A Bittensor subnet for collecting and storing valuable data for other subnets

data-universe Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# **Bittensor Subnet 13: Data Universe** 
[![Discord Chat](https://img.shields.io/discord/308323056592486420.svg)](https://discord.gg/bittensor)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

---
</div>

---

# MUST READ: Miner Data Compliance Policy

By participating as a miner on Subnet 13, you are agreeing to adhere to our [Miner Policy](docs/miner_policy.md) below. 
<details>
  <summary>
    Macrocosmos Miner Data Compliance Policy
  </summary>

---

*Version 1.0, March 2025*

---

## Introduction
As a miner on Macrocosmos Subnets, you play a crucial role in handling various types of data. This summary outlines your potential obligations under the UK General Data Protection Regulation (UK GDPR) should you inadvertently collect personal data, and the expectations regarding prohibited content and acceptable use of Macrocosmos subnet code.
We expect any participant (however they may define themselves or their involvement) in our subnet ecosystems to adhere to the guidelines set out in this policy. Deliberate and consistent violation of these guidelines may result in Macrocosmos seeking to limit your ability to participate, support for your participation and/or your incentive rewards.
Miners and all other participants are responsible for their own legal and regulatory compliance procedures and are encouraged to seek advice if in any doubt as to how to proceed. Macrocosmos is available to provide informal guidance if required (see contact information below).

## 1. Your Responsibilities Under UK GDPR
While Macrocosmos does not directly collect or process data, and seeks to avoid incentivising any collection or interaction with personal data, as a miner, you may be subject to GDPR obligations if your activities result in the inadvertent or accidental collection of personal data. We recommend that you put in place appropriate policies and procedures to accommodate this eventuality and set out below a summary of key responsibilities:
1. Lawful Basis for Processing
    - You must ensure there is a lawful basis for collecting and processing any personal data (e.g., consent, legitimate interests, or legal obligation).
2. Transparency
    - Inform individuals about how their data is being collected, processed, and stored.
    - Provide clear privacy information, including the purpose and lawful basis for processing.
3. Data Minimization
    - Only collect and process data that is strictly necessary for your stated purpose.
    - Avoid collecting sensitive personal data unless absolutely required and lawful.
4. Upholding Individuals' Rights
    - Be prepared to handle requests or objections from individuals regarding their right to access, rectify, limit processing of or delete their personal data in compliance with GDPR.
5. Data Security
    - Implement robust security measures to protect any data you collect from unauthorized access or breaches.
    - Ensure encryption and secure storage practices are in place.
6. Breach Reporting
    - Notify the appropriate data protection authority (e.g., ICO) within 72 hours of becoming aware of a personal data breach.

## 2. Prohibited Content
As a miner, you are strictly prohibited from collecting, processing, or transmitting the following types of content:
1. Illegal Content
    - Child abuse material or exploitation.
    - Hate speech, extremist propaganda, or content inciting violence.
    - Content related to human trafficking or modern slavery.
2. Copyrighted Material
    - Content protected by copyright unless you have explicit permission or a valid license to use it.
3. Explicit or Harmful Content
    - Pornographic or explicit imagery.
    - Content promoting self-harm, suicide, or drug abuse.

3. Acceptable Use Expectations
Macrocosmos expects all miners to:
Comply with platform-specific terms of service and relevant laws, including GDPR.
Use Macrocosmos subnets for ethical and lawful purposes only.
Regularly review and update your data collection and processing practices to ensure compliance with legal and ethical standards.
Immediately cease processing any flagged or prohibited material and report concerns to the appropriate authorities where required.

4. Commitment to Support
Macrocosmos is committed to supporting miners in understanding and meeting their GDPR obligations. To help you navigate these requirements and ensure compliance, we provide the following guidance and resources:

## **GDPR Guidance and Resources**
1. Overview of GDPR Requirements
    - The UK Information Commissioner's Office (ICO) provides a comprehensive guide to GDPR obligations, including lawful bases for processing, data minimization, and security requirements:
    - [ICO Guide to GDPR](https://ico.org.uk/media/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr-1-1.pdf)
2. Lawful Basis for Processing Data
    - Understand the six lawful bases for processing personal data as defined under GDPR:
    - [ICO Lawful Basis Guide](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/lawful-basis/a-guide-to-lawful-basis/)
3. Transparency and Privacy Notices
    - Guidance on providing clear and accessible privacy notices to individuals:
    - [ICO Privacy Notice Checklist](https://ico.org.uk/media/for-organisations/documents/1625126/privacy-notice-checklist.pdf)
4. Handling Data Subject Rights
    - Information on responding to requests from individuals to access, rectify, or delete their personal data:
    - [ICO Individual Rights Guide](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/individual-rights/individual-rights/)
5. Data Security and Minimization
    - Best practices for securing personal data and ensuring data minimization:
    - [ICO Security Guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/security/a-guide-to-data-security/)
6. Reporting Data Breaches
    - Guidance on recognizing and reporting data breaches to the ICO within the required 72-hour window:
    - [ICO Guide to Data Breach Reporting](https://ico.org.uk/for-organisations/report-a-breach/personal-data-breach/personal-data-breaches-a-guide/)

## **General Resources**
1. UK GDPR Key Definitions
    - A quick reference guide to key GDPR definitions and principles:
    - [ICO Key Definitions](https://ico.org.uk/for-organisations/guide-to-eidas/key-definitions/)
2. Data Protection Impact Assessments (DPIAs)
    - Information on when and how to conduct a DPIA for high-risk data processing activities:
    - [ICO DPIA Guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/accountability-and-governance/data-protection-impact-assessments-dpias/)
3. FAQs on GDPR Compliance
    - Practical answers to common GDPR compliance questions:
    - [GDPR FAQs from the European Data Protection Board](https://edpb.europa.eu/our-work-tools/our-documents/faqs_en)

## **Support from Macrocosmos**
1. Regular Updates and Communication
    - Macrocosmos will provide updates on GDPR-related requirements and best practices through periodic communications.
2. Consultation Support
    - If miners have specific questions or require clarification on GDPR obligations, Macrocosmos offers a support channel to address these concerns: compliance@macrocosmos.com
3. Training Materials
    - Macrocosmos may share training materials and resources from time to time to help miners enhance their understanding of GDPR compliance.

</details>

# Introduction

Subnet 13, Data Universe, is a data-scraping and sentiment analysis subnet owned by Macrocosmos.ai. It collects and stores large amounts of data from across a wide-range of sources for use by other Subnets and businesses. Data Universe scrapes real-time data from X (formerly known as Twitter), Reddit, and YouTube transcripts.

Data Universe is currently the world's largest open-source social media dataset, highlighting its efficiency and business use cases. Its datasets are easy to access, so businesses and other subnets can use them quickly and with low-friction.

Data Universe was built from the ground-up with a focus on decentralization and scalability. There is no centralized entity that controls the data; the data is stored across all Miners on the network, and is queryable via the Validators. At launch, Data Universe is able to support up to 50 Petabytes of data across 200 miners, while only requiring ~10GB of storage on the Validator.

# Products

Macrocosmos has launched its own products built with subnet 13:

## Gravity

This flagship product allows you to send data-scraping jobs to the network, choosing specific key terms and phrases to collect. This functions on both X and Reddit, with YouTube scraping support coming soon. Gravity is a highly accessible no-code tool that can be used by anyone, regardless of technical background. It's also a low-cost solution to data-scraping and sentiment analysis, undercutting many centralized and decentralized alternatives on the market.

- Try Gravity here: https://app.macrocosmos.ai/gravity
- Learn more about Gravity here: https://docs.macrocosmos.ai/constellation-user-guides/gravity 
- Access Gravity via the API: https://docs.macrocosmos.ai/developers/api-documentation/sn13-data-universe

## Mission Commander

This is an agentic LLM chatbot built into Gravity, designed to help you pick the right terms and phrases for your data-scraping needs. Simply tell it what information you want, and it'll offer suggestions and help with brainstorming. Mission Commander is built with Subnet 1, Apex, also owned by Macrocosmos.

- Try Mission Commander via the Gravity app: https://app.macrocosmos.ai/gravity

## Nebula

This is our data-scraping visualizer. It allows you to see your scraped data mapped out on a 3-dimensional plane. It also shows sentiment analysis. Nebula is especially valuable for people who prefer visual aids when looking through data.

- Try Nebula here: https://nebula.macrocosmos.ai/
- Learn more about Nebula here: https://docs.macrocosmos.ai/constellation-user-guides/nebula

## MCP

You can integrate subnet 13, Data-Universe, directly into Claude and Cursor via our MCP. This allows you to access our data-scraping information about X and Reddit, rather than only from our website.

- Try the MCP by following our guide here: https://docs.macrocosmos.ai/developers/tools/macrocosmos-mcp

# Team

Subnet 13 is owned by Macrocosmos.ai, one of the largest companies on Bittensor. Key team members include:

- **Will Squires, Co-Founder and CEO**: Will is an entrepreneur with experience building AI startups. He has a Master of Engineering degree in civil engineering and sustainability from the University of Warwick.

- **Steffen Cruz, Co-Founder and CTO**: Steffen formerly worked at OpenTensor Foundation (the organization behind Bittensor) as their CTO, and is the original architect of Subnet 1, Apex's codebase. He has a PhD in Experimental Nuclear Physics.

- **[Arrmlet](https://github.com/Arrmlet)**: Our subnet lead is the leading designer for Data-Universe with a Masters in Physics from the University of Lviv, Ukraine.

- **[ewekazoo](https://github.com/ewekazoo), Software Engineer**: With a B.S. in Computer Science from Columbia University, our subnet engineer works on maintaining, expanding, and strengthening subnet 13.

# Market and Partnerships

Subnet 13 has a clear product-market fit, with Gravity functioning as a service that allows businesses to scrape specific social media data for market research purposes at a low cost. Companies can also get sentiment analysis included within their scraping tasks, helping to quickly gauge public perception, identify emerging trends, and make data-driven decisions.

# Overview

The Data Universe documentation assumes you are familiar with basic Bittensor concepts: Miners, Validators, and incentives. If you need a primer, please check out https://docs.bittensor.com/learn/bittensor-building-blocks.

In the Data Universe, Miners scrape data from a defined set of sources, called DataSources. Each piece of data (e.g. a webpage, BTC prices), called a DataEntity, is stored in the miner's database. Each DataEntity belongs to exactly one DataEntityBucket, which is uniquely identified by its DataEntityBucketId, a tuple of: where the data came from (DataSource), when it was created (TimeBucket), and a classification of the data (DataLabel, e.g. a stock ticker symbol). The full set of DataEntityBuckets on a Miner is referred to as its MinerIndex.

Validators periodically query each Miner to fetch their latest MinerIndexes and store them in a local database. This gives the Validator a complete understanding of all data that's stored on the network, as well as which Miners to query for specific types of data. Validators also periodically verify the correctness of the data stored on Miners and reward Miners based on the amount of [valuable data](#data-value) the Miner has. Validators log to [wandb](https://wandb.ai/macrocosmos/data-universe-validators) anonymously by default.

Miners upload their local stores to S3-compatible storage for public dataset access. This data is anonymized for privacy purposes to comply with the Terms of Service per each data source.

- For S3 storage setup, see the [S3 Storage documentation](https://github.com/macrocosm-os/data-universe-api).

The S3 storage option provides improved scalability, security, and performance, especially for large datasets. It uses a folder-based structure with blockchain authentication to ensure miners can only access their own data while validators can efficiently access all miners' data.

See the [Miner](docs/miner.md) and [Validator](docs/validator.md) docs for more information about how they work, as well as setup instructions.

# Incentive Mechanism

As described above, each Miner reports its MinerIndex to the Validator. The MinerIndex details how much and what type of data the Miner has. The Miner is then scored based on 2 dimensions:
1. How much data the Miner has and how valuable that data is.
1. How credible the Miner is.

## Data Value

Not all data is equally valuable! There are several factors used to determine data value:

### 1) Data Freshness

Fresh data is more valuable than old data, and data older than a certain threshold is not scored.

As of Dec 11th, 2023 data older than 30 days is not scored. This may increase in future.

### 2) Data Desirability

Data Universe maintains a [Dynamic Desirability List](docs/dynamic_desirability.md) that defines which types of data are desirable according to user requests from [Gravity](https://app.macrocosmos.ai/), the Data Universe product. Data deemed desirable is scored more highly. Unspecified labels get the default_scale_factor of 0.3, meaning they score less than half value in comparison.

The DataDesirabilityLookup will evolve over time, but each change will be announced ahead of time to give Miners adequate time to prepare for the update.

### 3) Duplication Factor

Data that's stored by many Miners is less valuable than data stored by only a few. The value of a piece of data is decreases proportional to the number of Miners storing it.

## Miner Credibility

Validators remain suspicious of Miners and so they periodically check a sample of data from each Miner's MinerIndex, to verify the data correctness. The Validator uses these checks to track a Miner's credibility, which it then uses to scale a Miner's score. The scaling is done in such a way that it is **always** worse for a Miner to misrepresent what types and how much data it has.

# Data Universe Dashboard

As you can see from the above, Data Universe rewards diversity of data (storing 200 copies of the same data isn't exactly beneficial!)

To help understand the current data on the Subnet, the Data Universe team hosts a dashboard (https://sn13-dashboard.api.macrocosmos.ai/), showing the amount of each type of data (by DataEntityBucketId) on the Subnet. Miners are strongly encouraged to use this dashboard to customize their [Miner Configuration](./docs/miner.md#configuring-the-miner), to maximize their rewards.

# Use Cases

Subnet 13, Data Universe, has a range of use-cases:

- Low-cost data-scraping that undercuts centralized and decentralized alternatives.
- Aid in market research in any industry by collecting social media results on pressing topics.
- Help boost the accuracy of predictive tools by injecting them with freshly scraped social media data.
- Understand global online sentiment on any topic, to help inform businesses on their goals.
- Track sports analytics on social media (used by Subnet 44, Score).
- Forecast performance of assets (used by Subnet 64, Chutes, via Squad.ai).

Data-Universe's potential is extensive, as anybody can use it to build datasets on whatever topics are meaningful to them. By tapping into Bittensor's miner and validator communities, people can access data scraping and analysis at a fast and affordable rate.

# Getting Started

See [Miner Setup](docs/miner.md#miner_setup) to learn how to setup a Miner.

See [Validator Setup](docs/validator.md#validator_setup) to learn how to setup a Validator.

# Upcoming Features

1. Private storage
2. More datasources feel free to add by yourself in [this repo](https://github.com/macrocosm-os/scrapers)! 
3. Youtube multilanguage support

# Terminology

**DataDesirabilityLookup:** A [defined list of rules](https://github.com/RusticLuftig/data-universe/blob/main/rewards/data_desirability_lookup.py) that determine how desirable data is, based on its DataSource and DataLabel.

**DataEntity:** A single "item" of data collected by a Miner. Each DataEntity has a URI, that the Validators can use to retrieve the item from its DataSource.

**DataEntityBucket:** A logical grouping of DataEntities, based on its DataEntityBucketId.

**DataEntityBucketId:** The unique identifier for a DataEntityBucket. It contains the TimeBucket, DataSource, and DataLabel.

**DataLabel:** A label associated with a DataEntity. Precisely what the label represents is unique to the DataSource. For example, for a Yahoo finance DataSource, the label is the stock ticker of the finance data.

**DataSource:** A source from which Miners scrape data.

**Miner Credibility**: A per-miner rating, based on how often they pass data validation checks. Used to heavily penalize Miner's who misrepresent their MinerIndex.

**Miner Index**: A summary of how much and what types of data a Miner has. Specifically, it's a list of DataEntityBuckets.

# Feedback

We welcome feedback!

If you have a suggestion, please reach out to @arrmlet, @ewekazoo or any of the broader Macrocosmos Team on the Discord channel, or file an Issue.

data-universe

data-universe Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

data-universe

data-universe Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README