dolphinbeat - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #react #ai #typescript #go #machine-learning #rust #linux #docker #nodejs #awesome #vue #open-source

dolphinbeat

(★ 177)

A server that pulls and parses MySQL binlog, pushs change data into different sinks like Kafka.

File Explorer

Download Latest Version (.zip)

- - position.go
  - progress.go
  - progress_test.go
- canal.go
- canal_test.go
- config.go
- dump.go
- gtid.go
- handler.go
- handler_mux.go
- hook.go
- master.go
- rows.go
- sync.go
- checkpoint.go
- config.go
- manager.go
- manager_test.go
- storage.go
- storage_file.go
- storage_file_test.go
- storage_mock.go
- storage_zookeeper.go
- - app.go
  - config.go
  - dolphinbeat.toml.sample
  - dolphinbeat.toml.template
  - hook.go
  - http.go
  - http_handler.go
  - log.go
  - main.go
  - metrics.go
  - sink.go
  - sink_logger.go
- dump.go
- dump_test.go
- parser.go
- go-mysql-LICENSE
- config.go
- definition.go
- storage.go
- storage_boltdb.go
- storage_mysql.go
- tracker.go
- - config.go
  - event_handler.go
  - kafka.go
  - operation.go
  - pkg.go
- - config.go
  - event_handler.go
  - operation.go
  - pkg.go
  - stdout.go
  - stdout_test.go
- config.go
- logger.go
- sink.go
- version.go
.gitignore
.travis.yml
docker-entrypoint.sh
Dockerfile
go.mod
go.sum
LICENSE
Makefile
README.md
README.zh-cn.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DolphinBeat [![Build Status](https://travis-ci.org/bytewatch/dolphinbeat.svg?branch=master)](https://travis-ci.org/bytewatch/dolphinbeat)

![](https://img.shields.io/github/license/bytewatch/dolphinbeat.svg)

> Other languages: [中文](./README.zh-cn.md)

This is a high available server that pulls MySQL binlog, parses binlog and pushs incremental update data into different sinks.

The types of sink supported currently and officially are [Kafka](#Kafka) and [Stdout](#Stdout).

Features:
* Supports MySQL and MariaDB.
* Supports GTID and not GTID.
* Supports MySQL failover: if using GTID, `dolphinbeat` can work smoothly even if MySQL failover.
* Supports MySQL DDL: `dolphinbeat` can parse DDL statement and replay DDL upon it's own schema data in memory.
* Supports breakpoint resume: `dolphinbeat` has persistent metadata, it can resume to work after crash recover.
* Supports standalone and election mode: if election enabled, `dolphinbeat` follower will take over dead leader.
* Supports filter rules base on database and table for each sink.
* Supports HTTP API to inspect `dolphinbeat`.
* Supports metrics in Prometheus style.

> The types of sink are scalable, you can implement your own sink if need, but I recommend you to use Kafka sink and let business consumes data from Kafka.

# Quick start
Prepare your MySQL source, trun on binlog with ROW format, and type following commands and you will see JSON printed by `dolphinbeat`'s Stdout sink.
```
docker run -e MYSQL_ADDR='8.8.8.8:3306' -e MYSQL_USER='root' -e MYSQL_PASSWORD='xxx' bytewatch/dolphinbeat
{
  "header": {
    "server_id": 66693,
    "type": "rotate",
    "timestamp": 0,
    "log_pos": 0
  },
  "next_log_name": "mysql-bin.000008",
  "next_log_pos": 4
}
...
...
```
The docker image above is for MySQL with GTID and only with Stdout sink enabled.
> If your source database is not GTID, please add `-e GTID_ENABLED='false'` arg.
> If your source database is MariaDB, please add `-e FLAVOR='mariadb'` arg.

If you want to have a deep test, type following commands and you will get a shell:
```
docker run -e MYSQL_ADDR='8.8.8.8:3306' -e MYSQL_USER='root' -e MYSQL_PASSWORD='xxx' sh
``` 
In this shell, you can modify configurations in /data directory, and then start `dolphinbeat` manually.

Configuration description is presented in [Wiki](https://github.com/bytewatch/dolphinbeat/wiki/Configuration).

# Compile from source
Type following commands and you will get builded binary distribution at build/dolphinbeat directory:
```
go get github.com/bytewatch/dolphinbeat
make 
```
# Documents
* [Configuration](https://github.com/bytewatch/dolphinbeat/wiki/Configuration)
* [HTTP API](https://github.com/bytewatch/dolphinbeat/wiki/HTTP-API)

# Sink
## Kafka
This is a sink used for production. `Dolphinbeat` write data encoded with Protobuf into Kafka and business consumes data from Kafka.

Business need use [client library](sink/kafka/client) to decode data in Kafka message, do stream processing on the binlog stream.

The Protobuf protocol is presented in [protocol.proto ](sink/kafka/protocol).

Kafka sink has following features:
* Strong-ordered delivery: business will receive events in the same order with MySQL binlog. 
* Exactly-once delivery: client library can dedup duplicated message with same sequence number which may caused by producer retry or Kafka failover.
* Unlimited event size: `dolphinbeat` use fragments algorithm like IPV4 if the binlog event is bigger than Kafka's max message size.

A small example using client library is presented in [kafka-consumer](cmd/tools/kafka-consumer).

`kafka-consumer` is a command tool to  decode data in Kafka message and print out with JSON.

## Stdout
This is a sink used for demonstration. `Dolphinbeat` write data encoded with JSON to Stdout.

> Stdout sink doesn't support breakpoint resume.

# Special thanks
Thank siddontang for his popular and powerful `go-mysql` library!

# License
[Apache License 2.0](https://github.com/bytewatch/dolphinbeat/blob/master/LICENSE)