datavines

KO | EN

gitlite — search

#nodejs #javascript #typescript #react #python #api #hacktoberfest #mysql #python3 #blockchain #reactjs #neural-network

★ 757 Open GitHub ↗

Know your data better！Datavines is Next-gen Data Observability Platform, support metadata manage and data quality.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Datavines
[![EN doc](https://img.shields.io/badge/document-English-blue.svg)](https://raw.githubusercontent.com/datavane/datavines/dev/README.md)
[![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](https://raw.githubusercontent.com/datavane/datavines/dev/README.zh-CN.md)
---

Data quality is used to ensure the accuracy of data in the process of integration and processing. It is also the core component of DataOps. DataVines is an easy-to-use data quality service platform that supports multiple metric.

## Architecture Design
![DataVinesArchitecture](https://raw.githubusercontent.com/datavane/datavines/dev/docs/img/architecture.jpg)

## Install

Need: Maven 3.6.1 and later
```sh
$ mvn clean package -Prelease -DskipTests
```

## Metadata Database

- The default `datavines-server` configuration uses `PostgreSQL`
- Initialize a PostgreSQL metadata database with `scripts/sql/datavines-postgresql.sql`
- If you want to run with `MySQL`, switch to the `mysql` Spring profile and initialize with `scripts/sql/datavines-mysql.sql`
- The release package now contains both scripts in `${DATAVINES_HOME}/scripts/`

## Features

### Data Catalog

- Obtain **data source metadata** regularly to construct data directory 
- Regular monitoring of **metadata changes**
- **Tag management** with support for metadata

![Data Catalog](https://raw.githubusercontent.com/datavane/datavines/dev/docs/img/data-catalog.jpg)

### Data Quality

- Built-in **27** data quality check rules
- Support **4** data quality check rule types
    - Single Table-Column Check
    - Single Table Custom `SQL` check 
    - Cross Table Accuracy Check
    - Two Table Value Comparison Check
- Support schedule tasks for check
- Support `SLA` for **check result alert**

![Data Quality](https://raw.githubusercontent.com/datavane/datavines/dev/docs/img/data-quality.jpg)

### Data Profile

- Support timing execution of data detection, output **data profile report**
- Support **automatically identify** column types to automatically match appropriate data profile indicators
- Support **table row number trend** monitoring
- Support **data distribution** view

![数据目录](https://raw.githubusercontent.com/datavane/datavines/dev/docs/img/data-profile.jpg)

### Plug-in Design

The platform is based on plug-in design, and the following modules support user-defined plug-ins to expand

- **Data Source**: `MySQL`, `Impala`, `StarRocks`, `Doris`, `Presto`, `Trino`, `ClickHouse`, `PostgreSQL` are already supported
- **Check Rules**: 27 check rules such as built-in null value check, non-null check, enumeration check, etc.
- **Job Execution Engine**: Two execution engines `Spark` and `Local` have been supported. The `Spark` engine currently only supports the `Spark2.4` version, and the `Local` engine is a local execution engine developed based on `JDBC`, without relying on other execution engines.
- **Alert Channel**: Supported **Email**
- **Error Data Storage**: `MySQL` and **local files** are already supported (only `Local` execution engine is supported)
- **Registry**: Already supports `MySQL`, `PostgreSQL` and `ZooKeeper`

### Multiple Execute Modes

- Provide **Web page** to configure check jobs, run jobs, view job execution logs, view error data and check results
- Support **online generation** job running scripts, submit jobs through `datavines-submit.sh`, can be used in conjunction with the scheduling system

![作业脚本](https://raw.githubusercontent.com/datavane/datavines/dev/docs/img/data-job-script.jpg)

### Easy Deployment & High Availability

- Less platform dependency, easy to deploy
- Minimal only rely on `MySQL` to start the project and complete the check of data quality operations
- Support horizontal expansion, automatic fault tolerance
- **Decentralized design**, `Server` node supports horizontal expansion to improve performance
- Job **Automatic Fault Tolerance**, to ensure that jobs are not lost or repeated

## Environmental Dependency

1. java runtime environment: jdk8
2. If the data volume is small, or the goal is merely for functional verification, you can use JDBC engine
3. If you want to run DataVines based on Spark, you need to ensure that your server has spark installed
## Quick Start
Click [Document](https://datavane.github.io/datavines-website/docs/user-guide/quick-start) for more information

## Development

Click [Document](https://datavane.github.io/datavines-website/docs/development/environment-preparation) for more information

## Contribution

[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](https://github.com/datavane/datavines/pulls)

You can submit any ideas as [pull requests](https://github.com/datavane/datavines/pulls) or as [GitHub issues](https://github.com/datavane/datavines/issues/new/choose).

> If you're new to posting issues, we ask that you read [*How To Ask Questions The Smart Way*](http://www.catb.org/~esr/faqs/smart-questions.html) (**This guide does not provide actual support services for this project!**), [How to Report Bugs Effectively](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html) prior to posting. Well written bug reports help us help you!

Thank you to all the people who already contributed to Datavines!

[![contrib graph](https://contrib.rocks/image?repo=datavane/datavines)](https://github.com/datavane/datavines/graphs/contributors)

## License

Datavines is licensed under the [Apache License 2.0](LICENSE). Datavines relies on some third-party components, and their open source protocols are also Apache License 2.0 or compatible with Apache License 2.0. In addition, Datavines also directly references or modifies some codes in Apache DolphinScheduler, SeaTunnel and Dubbo, all of which are Apache License 2.0. Thanks for contributions to these projects.

## Social Media

- WeChat Official Account (in Chinese, scan the QR code to follow)

![wechat-qrcode](https://raw.githubusercontent.com/datavane/datavines/dev/docs/img/wechat-qrcode-en.jpg)

## Contact Author

- Notes "Datavines" When Adding Me On WeChat
 
![wechat-author-qrcode](https://raw.githubusercontent.com/datavane/datavines/dev/docs/img/wechat-author-qrcode.jpg)

## Donation

![wechat-donation-qrcode](https://raw.githubusercontent.com/datavane/datavines/dev/docs/img/wechat-donation-qrcode.jpg)

datavines

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README