clickhouse-kafka-connect - GitLite

KO | EN

gitlite — search

#javascript #python #hacktoberfest #machine-learning #deep-learning #linux #docker #nodejs #macos #cli #security #ios #mcp #openai

clickhouse-kafka-connect

(★ 209)

ClickHouse Kafka Connector

File Explorer

Download Latest Version (.zip)

- - bug_report.md
  - config.yml
  - feature_request.md
- - increment-version.yml
  - integrationTests.yaml
  - pre-release.yaml
  - release.yaml
  - test-head.yaml
  - tests.yaml
- CODEOWNERS
- dependabot.yml
- pull_request_template.md
- .gitattributes
- .gitignore
- build.gradle.kts
- gradle.properties
- gradlew
- gradlew.bat
- README.md
- settings.gradle.kts
- - manifest.json
- architecture.png
- cropping_batches.png
- deduplication.png
- DESIGN.md
- full_state_machine.png
- invalid_states.png
- start_state.png
- - gradle-wrapper.jar
  - gradle-wrapper.properties
- libs.versions.toml
- run-ch-version-client-matrix.sh
.gitattributes
.gitignore
AGENTS.md
AI_POLICY.md
build.gradle.kts
CHANGELOG.md
CLAUDE.md
ClickHouseKafkaConnectDashboard.json
CONTRIBUTING.md
Dockerfile
gradlew
gradlew.bat
jmx-export-connector.yml
LICENSE
README.md
SECURITY.md
settings.gradle.kts
Support Scripts.md
VERSION

# Installation Guide

1. Get the code

git clone https://github.com/ClickHouse/clickhouse-kafka-connect

Downloads the entire project code from GitHub to your computer.

cd clickhouse-kafka-connect

Moves into the project folder you just downloaded.

2. Docker

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Docker Desktop Needed to build and run containers. Install it and keep it running in the background.

docker compose -f src/testFixtures/docker/clickhouse/cluster/docker-compose.yml up -d --build

Runs the command against the services defined in the compose file.

✅ Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

3. Gradle (Java/Kotlin)

Medium

Prerequisites

Git Needed to download the project code from GitHub.
JDK (Java) Required to build and run Java/Kotlin projects.
Gradle If the repo includes gradlew (Gradle Wrapper), you don't need to install Gradle separately.

cd benchmark

This project's files live in a subfolder, so move into it first.

./gradlew build

Builds the project using Gradle.

✅ A BUILD SUCCESSFUL message means it worked. The output is created under build/.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ClickHouse Kafka Connect Sink

## About
clickhouse-kafka-connect is the official Kafka Connect sink connector for [ClickHouse](https://clickhouse.com/).

The Kafka connector delivers data from a Kafka topic to a ClickHouse table.
## Documentation

See the [ClickHouse website](https://clickhouse.com/docs/en/integrations/kafka/clickhouse-kafka-connect-sink) for the full documentation entry.

## Design
For a full overview of the design and how exactly-once delivery semantics are achieved, see the [design document](./docs/DESIGN.md).

## Help
For additional help, please [file an issue in the repository](https://github.com/ClickHouse/clickhouse-kafka-connect/issues) or raise a question in [ClickHouse public Slack](https://clickhouse.com/slack).

## KeyToValue Transformation
We've created a transformation that allows you to convert a Kafka message key into a value.
This is useful when you want to store the key in a separate column in ClickHouse - by default, the column is `_key` and the type is String.

```sql
CREATE TABLE your_table_name
(
    `your_column_name` String,
    ...
    ...
    ...
    `_key` String
) ENGINE = MergeTree()
```

Simply add the transformation to your connector configuration:
    
```properties
transforms=keyToValue
transforms.keyToValue.type=com.clickhouse.kafka.connect.transforms.KeyToValue
transforms.keyToValue.field=_key
```

## Performance Testing

There is a dedicated gradle project in this repo - `benchmark` for performance testing. 
Please see its [README](./benchmark/README.md) for more information and how to run.