kafka-connect-github-source - GitLite

KO | EN

gitlite — search

#hacktoberfest #ai #rust #linux #macos #windows #awesome #cli #awesome-list #vue #pytorch #security #openai #ios

kafka-connect-github-source

(★ 463)

Get a stream of issues and pull requests for your chosen GitHub repository

File Explorer

Download Latest Version (.zip)

- GitHubSourceConnectorExample.properties
- worker.properties
- .gitignore
.gitignore
build.sh
Dockerfile
end-to-end.sh
LICENSE
pom.xml
README.md
run.sh

# Installation Guide

1. Get the code

git clone https://github.com/simplesteph/kafka-connect-github-source

Downloads the entire project code from GitHub to your computer.

cd kafka-connect-github-source

Moves into the project folder you just downloaded.

2. Docker

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Docker Desktop Needed to build and run containers. Install it and keep it running in the background.

docker build -t kafka-connect-github-source .

Builds a runnable image based on the Dockerfile.

docker run -p 8080:80 kafka-connect-github-source

Runs the built image as an actual container.

✅ Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

3. Maven (Java)

Medium

Prerequisites

Git Needed to download the project code from GitHub.
JDK (Java) Required to build and run Java projects.
Maven The build tool used for the mvn command.

mvn clean install

Installs dependencies and builds the project using Maven.

✅ A BUILD SUCCESS message means it worked. The output is created under target/.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Learning

This project is a companion repository to the [Apache Kafka Connect course on Udemy](https://links.datacumulus.com/kafka-connect-coupon).

https://links.datacumulus.com/kafka-connect-coupon

# Kafka Connect Source GitHub

This connector allows you to get a stream of issues and pull requests from your GitHub repository, using the GitHub Api: https://developer.github.com/v3/issues/#list-issues-for-a-repository

Issues are pulled based on `updated_at` field, meaning any update to an issue or pull request will appear in the stream.

The connector writes to topic that is great candidate to demonstrate *log compaction*. It's also a fun way to automate your GitHub workflow.

It's finally aimed to be an educative example to demonstrate how to write a Source Connector a little less trivial than the `FileStreamSourceConnector` provided in Kafka.

# Contributing

This connector is not perfect and can be improved, please feel free to submit any PR you deem useful.

# Configuration

```
name=GitHubSourceConnectorDemo
tasks.max=1
connector.class=com.simplesteph.kafka.GitHubSourceConnector
topic=github-issues
github.owner=kubernetes
github.repo=kubernetes
since.timestamp=2017-01-01T00:00:00Z
# I heavily recommend you set those two fields:
auth.username=your_username
auth.password=your_password
```

# Running in development

Note: Java 8 is required for this connector. 
Make sure `config/worker.properties` is configured to wherever your kafka cluster is

```
./build.sh
./run.sh 
```

The simplest way to run `run.sh` is to have docker installed. It will pull a Dockerfile and run the connector in standalone mode above it.

# Deploying

Note: Java 8 is required for this connector.

TODO