kafka-backup - GitLite

KO | EN

gitlite — search

#go #golang #machine-learning #deep-learning #linux #docker #nodejs #java #awesome #vue #pytorch #security #openai #open-source

kafka-backup

(★ 174)

Backup and Restore for Apache Kafka

파일 탐색기

최종 버전 다운로드 (.zip)

- - workflow.yml
- backup-standalone.sh
- completed_segments.py
- partition-index.sh
- restore-standalone.sh
- segment-index.sh
- segment.sh
- - 2019-06_Introducing_Kafka_Backup.md
- Comparing_Kafka_Backup_Solutions.md
- FAQ.md
- Kafka_Backup_Architecture.md
- Tooling.md
- Usage.md
- - gradle-wrapper.jar
  - gradle-wrapper.properties
- - utils.sh
- - Dockerfile
  - runutil
  - utils.py
  - utils.sh
- .gitignore
- 01_simple_roundtrip_test.yaml
- 02_full_test.yaml
- 03_start_n_stop.yaml
- 04_delete_old_segments.yaml
- docker-compose.yml
- README.md
.gitignore
build.gradle
Dockerfile
gradlew
gradlew.bat
LICENSE
README.md
settings.gradle

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/itadventurer/kafka-backup

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd kafka-backup

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Docker

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Docker Desktop 컨테이너를 빌드하고 실행하려면 필요합니다. 설치 후 실행해서 백그라운드에 켜두세요.

docker run -d -v /path/to/backup-dir/:/kafka-backup/ --rm \

빌드된 이미지를 실제 컨테이너로 실행합니다.

docker run -v /path/to/backup/dir:/kafka-backup/ --rm \

빌드된 이미지를 실제 컨테이너로 실행합니다.

✅ 터미널에 docker compose ps 를 입력해 컨테이너들이 Up 상태인지 확인하세요. README에 포트 번호가 적혀있다면 브라우저에서 http://localhost:포트번호 로 접속해보세요.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

3. Gradle (Java/Kotlin)

보통

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
JDK (Java) Java/Kotlin 프로젝트를 빌드/실행하려면 필요합니다.
Gradle 레포에 포함된 gradlew(Gradle Wrapper)를 쓰면 Gradle을 따로 설치할 필요가 없습니다.

./gradlew build

Gradle로 빌드를 진행합니다.

✅ BUILD SUCCESSFUL 메시지가 뜨면 성공입니다. build/ 폴더에 결과물이 생성됩니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Kafka Backup

> **Update:** I am no longer maintaining the Kafka Backup project. As an alternative, I recommend [Kannika](https://kannika.io/?utm_source=github_anatoly), a commercial backup solution developed by my friends at [Cymo](https://cymo.eu/?utm_source=github_anatoly) (and don't forget to say hello from Anatoly 😊).
> [Disclosure: I am a business partner of Cymo and may receive compensation for referrals to Kannika]
> 
> Please contact me if you want to continue maintaining this project.

Kafka Backup is a tool to back up and restore your Kafka data
including all (configurable) topic data and especially also consumer
group offsets. To the best of our knowledge, Kafka Backup is the only
viable solution to take a cold backup of your Kafka data and restore
it correctly.

It is designed as two connectors for Kafka
Connect: A sink connector (backing data up) and a source connector
(restoring data).

Currently `kafka-backup` supports backup and restore to/from the file
system.

## Features

* Backup and restore topic data
* Backup and restore consumer-group offsets
* Currently supports only backup/restore to/from local file system
* Released as a jar file or packaged as a Docker image

# Getting Started

**Option A) Download binary**

Download the latest release [from GitHub](https://github.com/itadventurer/kafka-backup/releases) and unzip it.

**Option B) Use Docker image**

Pull the latest Docker image from [Docker Hub](https://hub.docker.com/repository/docker/itadventurer/kafka-backup/tags)

**DO NOT USE THE `latest` STAGE IN PRODUCTION**. `latest` are automatic builds of the master branch. Be careful!

**Option C) Build from source**

Just run `./gradlew shadowJar` in the root directory of Kafka Backup. You will find the CLI tools in the `bin` directory.

## Start Kafka Backup

```sh
backup-standalone.sh --bootstrap-server localhost:9092 \
    --target-dir /path/to/backup/dir --topics 'topic1,topic2'
```

In Docker:
```sh
docker run -d -v /path/to/backup-dir/:/kafka-backup/ --rm \
    kafka-backup:[LATEST_TAG] \
    backup-standalone.sh --bootstrap-server kafka:9092 \
    --target-dir /kafka-backup/ --topics 'topic1,topic2'
```

You can pass options via CLI arguments or using environment variables:

| Parameter                                   | Type/required? | Description                                                                                                          |
|---------------------------------------------|----------------|----------------------------------------------------------------------------------------------------------------------|
| `--bootstrap-server`<br/>`BOOTSTRAP_SERVER` | [REQUIRED]     | The Kafka server to connect to                                                                                       |
| `--target-dir`<br/>`TARGET_DIR`             | [REQUIRED]     | Directory where the backup files should be stored                                                                    |
| `--topics`<br/>`TOPICS`                     | <T1,T2,…>      | List of topics to be backed up. You must provide either `--topics` or `--topics-regex`. Not both                     |
| `--topics-regex`<br/>`TOPICS_REGEX`         |                | Regex of topics to be backed up. You must provide either `--topics` or `--topics-regex`. Not both                    |
| `--max-segment-size`<br/>`MAX_SEGMENT_SIZE` |                | Size of the backup segments in bytes DEFAULT: 1GiB                                                                   |
| `--command-config`<br/>`COMMAND_CONFIG`     | <FILE>         | Property file containing configs to be passed to Admin Client. Only useful if you have additional connection options |
| `--debug`<br/>`DEBUG=y`                     |                | Print Debug information                                                                                              |
| `--help`                                    |                | Prints this message                                                                                                  |

**Kafka Backup does not stop!** The Backup process is a continous background job that runs forever as Kafka models data as a stream without end. See [Issue 52: Support point-in-time snapshots](https://github.com/itadventurer/kafka-backup/issues/52) for more information.

## Restore data

```sh
restore-standalone.sh --bootstrap-server localhost:9092 \
    --target-dir /path/to/backup/dir --topics 'topic1,topic2'
```

In Docker:
```sh
docker run -v /path/to/backup/dir:/kafka-backup/ --rm \
    kafka-backup:[LATEST_TAG]
    restore-standalone.sh --bootstrap-server kafka:9092 \
    --source-dir /kafka-backup/ --topics 'topic1,topic2'
```

You can pass options via CLI arguments or using environment variables:

| Parameter                                   | Type/required? | Description                                                                                                                                                                                                                |
|---------------------------------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `--bootstrap-server`<br/>`BOOTSTRAP_SERVER` | [REQUIRED]     | The Kafka server to connect to                                                                                                                                                                                             |
| `--source-dir`<br/>`SOURCE_DIR`             | [REQUIRED]     | Directory where the backup files are found                                                                                                                                                                                 |
| `--topics`<br/>`TOPICS`                     | [REQUIRED]     | List of topics to restore                                                                                                                                                                                                  |
| `--batch-size`<br/>`BATCH_SIZE`             |                | Batch size (Default: 1MiB)                                                                                                                                                                                                 |
| `--offset-file`<br/>`OFFSET_FILE`           |                | File where to store offsets. THIS FILE IS CRUCIAL FOR A CORRECT RESTORATION PROCESS IF YOU LOSE IT YOU NEED TO START THE BACKUP FROM SCRATCH. OTHERWISE YOU WILL HAVE DUPLICATE DATA Default: [source-dir]/restore.offsets |
| `--command-config`<br/>`COMMAND_CONFIG`     | <FILE>         | Property file containing configs to be passed to Admin Client. Only useful if you have additional connection options                                                                                                       |
| `--help`<br/>`HELP`                         |                | Prints this message                                                                                                                                                                                                        |
| `--debug`<br/>`DEBUG`                       |                | Print Debug information (if using the environment variable, set it to 'y')                                                                                                                                                 |

## More Documentation

* [FAQ](./docs/FAQ.md)
* [High Level
  Introduction](./docs/Blogposts/2019-06_Introducing_Kafka_Backup.md)
* [Comparing Kafka Backup
  Solutions](./docs/Comparing_Kafka_Backup_Solutions.md)
* [Architecture](./docs/Kafka_Backup_Architecture.md)
* [Tooling](./docs/Tooling.md)

## License

This project is licensed under the Apache License Version 2.0 (see
[LICENSE](./LICENSE)).