go-kafka-connect

KO | EN

gitlite — search

#javascript #golang #go #machine-learning #rust #deep-learning #nodejs #macos #java #windows #awesome #cli #kubernetes #awesome-list

go-kafka-connect

(★ 24)

Go library providing bindings for the Kafka connect API

go-kafka-connect Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# go-kafka-connect
Go project containing two different sub-projects: a kafka-connect client library and a CLI to use it.

## Kafka-connect client library
This library is to be used as an abstraction layer for the kafka-connect REST API.
It currently implements the following API calls:
- Create a connector
- Update a connector
- Delete a connector
- Pause a connector
- Resume a connector
- Restart a connector
- Get a connector's details (overview, configuration, status or tasks list)

It also contains two 'bonus' features:
- Do synchronously: All calls to the REST API trigger an asynchronous function on kafka-connect.
  This feature lets the library check regularly if the action has taken effect on kafka-connect's side,
  and considers the request as completed only when the consequences of the command can be verified.
  It allows users of this library to use its functions in a synchronous way.
- Deploy connector, a function used to deploy a connector, or replace an existing one gracefully.
  This function checks if the target connector exists. If it exists, it will then be paused before being updated.
  Before being updating it check the current config. If it match the deployment's config, nothing will be done.
  The new connector is then deployed, and resumed. This function is always synchronous.
- Ignore internal config keys: On Confluent Platform 8.x+, Kafka Connect injects runtime
  `__internal.*` keys (e.g. `__internal.config.version`) into connector config responses.
  These keys are not part of user-submitted config and can cause false "out of date" results
  in `IsUpToDate` and `DeployConnector`. Call `SetDisableInternalConfig(true)` on the client
  to strip them before comparing.

# Running
download binary for your system:
- linux: `wget https://github.com/ricardo-ch/go-kafka-connect/releases/download/1.0.0/kccli`
- windows: `wget https://github.com/ricardo-ch/go-kafka-connect/releases/download/1.0.0/kccli.exe`
- mac: `wget https://github.com/ricardo-ch/go-kafka-connect/releases/download/1.0.0/kccli_mac`

set executable if needed
`chmod +x kccli`

enjoy!

# Example of command

- Get help

```bash
./kccli --help
```

- Deploy a connector which config is stored in a json:

On Confluent Platform 8.x+, pass `--ignore-internal-config` so runtime `__internal.*` keys
injected by Kafka Connect do not cause unnecessary redeploys.

```bash
./kccli deploy -u http://kafka-connect.local --ignore-internal-config -f my-connector-config.json
```

- Deploy a bunch of connector in parallel and wait for the end:

```bash
jobs=''

./kccli deploy -u http://kafka-connect.local -f my-connector-config.json & jobs="$jobs $!"
./kccli deploy -u http://kafka-connect.local -f my-connector-2-config.json & jobs="$jobs $!"

status=0
for job in $jobs; do
  wait $job
  if [ $? != '0' ]; then $status=1; fi
done
if [ $status != 0 ]; then exit $status; fi
```

- Get connector status

```bash
./kccli -u http://kafka-connect.local get --status -n my-connector
```

# Setup environment for development
Required:
 - Go 1.9
 - Docker (for testing purpose only)

run `go get -u github.com/ricardo-ch/go-kafka-connect`
then inside repo run: `make install` to install dependencies

# Testing
For now, only integration test are available.
run `make test-integration`

Right now, integration tests only run locally

note: you may also run `make rundep`, if you just want to run a kafka-connect cluster in background for manual testing.

go-kafka-connect

go-kafka-connect Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

go-kafka-connect

go-kafka-connect Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README