kafka-docker-playground

(โ˜… 787)

๐Ÿณโœจ Fully automated Apache Kafkaยฎ and Confluent Docker based examples // ๐Ÿ‘ทโ€โ™‚๏ธ Easily build examples or reproduction models

ํŒŒ์ผ ์ˆ˜๊ฐ€ ๋งŽ์•„ ์ผ๋ถ€๋งŒ ํ‘œ์‹œ๋ฉ๋‹ˆ๋‹ค. ์ „์ฒด ํŒŒ์ผ์€ ์œ„ ๋‹ค์šด๋กœ๋“œ ๋ฒ„ํŠผ์œผ๋กœ ํ™•์ธํ•ด ์ฃผ์„ธ์š”.

  • .gitignore
  • .gitmodules
  • CLAUDE.md
  • CONTRIBUTING.md
  • LICENSE
  • README.md

๐Ÿš€ ์„ค์น˜ ๊ฐ€์ด๋“œ

1. ์ฝ”๋“œ ๋‚ด๋ ค๋ฐ›๊ธฐ
git clone https://github.com/vdesabou/kafka-docker-playground

๊นƒํ—ˆ๋ธŒ์—์„œ ํ”„๋กœ์ ํŠธ ์ฝ”๋“œ ์ „์ฒด๋ฅผ ๋‚ด ์ปดํ“จํ„ฐ๋กœ ๋‚ด๋ ค๋ฐ›์Šต๋‹ˆ๋‹ค.

cd kafka-docker-playground

๋ฐฉ๊ธˆ ๋‚ด๋ ค๋ฐ›์€ ํ”„๋กœ์ ํŠธ ํด๋” ์•ˆ์œผ๋กœ ์ด๋™ํ•ฉ๋‹ˆ๋‹ค.

2. Docker

์‰ฌ์›€ ์ถ”์ฒœ
์‚ฌ์ „ ์ค€๋น„๋ฌผ
  • Git GitHub์—์„œ ํ”„๋กœ์ ํŠธ ์ฝ”๋“œ๋ฅผ ๋‚ด๋ ค๋ฐ›์œผ๋ ค๋ฉด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • Docker Desktop ์ปจํ…Œ์ด๋„ˆ๋ฅผ ๋นŒ๋“œํ•˜๊ณ  ์‹คํ–‰ํ•˜๋ ค๋ฉด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค. ์„ค์น˜ ํ›„ ์‹คํ–‰ํ•ด์„œ ๋ฐฑ๊ทธ๋ผ์šด๋“œ์— ์ผœ๋‘์„ธ์š”.
docker compose -f academy/connect-connect-aws-s3-sink/docker-compose.plaintext.backup-and-restore-assuming-iam-role-with-custom-aws-credential-provider.yml up -d --build

compose ์„ค์ • ํŒŒ์ผ์— ์ •์˜๋œ ์„œ๋น„์Šค๋“ค์„ ๋Œ€์ƒ์œผ๋กœ ๋ช…๋ น์„ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

โœ… ํ„ฐ๋ฏธ๋„์— docker compose ps ๋ฅผ ์ž…๋ ฅํ•ด ์ปจํ…Œ์ด๋„ˆ๋“ค์ด Up ์ƒํƒœ์ธ์ง€ ํ™•์ธํ•˜์„ธ์š”. README์— ํฌํŠธ ๋ฒˆํ˜ธ๊ฐ€ ์ ํ˜€์žˆ๋‹ค๋ฉด ๋ธŒ๋ผ์šฐ์ €์—์„œ http://localhost:ํฌํŠธ๋ฒˆํ˜ธ ๋กœ ์ ‘์†ํ•ด๋ณด์„ธ์š”.

3. .NET

๋ณดํ†ต
์‚ฌ์ „ ์ค€๋น„๋ฌผ
  • Git GitHub์—์„œ ํ”„๋กœ์ ํŠธ ์ฝ”๋“œ๋ฅผ ๋‚ด๋ ค๋ฐ›์œผ๋ ค๋ฉด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • .NET SDK dotnet ๋ช…๋ น์–ด๋ฅผ ์“ฐ๊ธฐ ์œ„ํ•œ SDK์ž…๋‹ˆ๋‹ค.
cd ccloud/client-dotnet

์ด ํ”„๋กœ์ ํŠธ์˜ ๊ด€๋ จ ํŒŒ์ผ์ด ํ•˜์œ„ ํด๋” ์•ˆ์— ์žˆ์–ด์„œ, ๋จผ์ € ๊ทธ ํด๋”๋กœ ์ด๋™ํ•ฉ๋‹ˆ๋‹ค.

dotnet restore

ํ”„๋กœ์ ํŠธ์— ํ•„์š”ํ•œ ํŒจํ‚ค์ง€๋ฅผ ๋‚ด๋ ค๋ฐ›์Šต๋‹ˆ๋‹ค.

dotnet run

ํ”„๋กœ์ ํŠธ๋ฅผ ๋นŒ๋“œํ•˜๊ณ  ๋ฐ”๋กœ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

โœ… dotnet run ์‹คํ–‰ ํ›„ ํ„ฐ๋ฏธ๋„์— ๋‚˜ํƒ€๋‚˜๋Š” ์•ˆ๋‚ด ๋ฉ”์‹œ์ง€๋‚˜ ์ฃผ์†Œ๋ฅผ ํ™•์ธํ•˜์„ธ์š”.

4. Maven (Java)

๋ณดํ†ต
์‚ฌ์ „ ์ค€๋น„๋ฌผ
  • Git GitHub์—์„œ ํ”„๋กœ์ ํŠธ ์ฝ”๋“œ๋ฅผ ๋‚ด๋ ค๋ฐ›์œผ๋ ค๋ฉด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • JDK (Java) Java ํ”„๋กœ์ ํŠธ๋ฅผ ๋นŒ๋“œ/์‹คํ–‰ํ•˜๋ ค๋ฉด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • Maven mvn ๋ช…๋ น์–ด๋ฅผ ์“ฐ๊ธฐ ์œ„ํ•œ ๋นŒ๋“œ ๋„๊ตฌ์ž…๋‹ˆ๋‹ค.
cd ccloud/fm-azure-event-hubs-source/simple-send

์ด ํ”„๋กœ์ ํŠธ์˜ ๊ด€๋ จ ํŒŒ์ผ์ด ํ•˜์œ„ ํด๋” ์•ˆ์— ์žˆ์–ด์„œ, ๋จผ์ € ๊ทธ ํด๋”๋กœ ์ด๋™ํ•ฉ๋‹ˆ๋‹ค.

mvn clean install

Maven์œผ๋กœ ์˜์กด์„ฑ ์„ค์น˜์™€ ๋นŒ๋“œ๋ฅผ ์ง„ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

โœ… BUILD SUCCESS ๋ฉ”์‹œ์ง€๊ฐ€ ๋œจ๋ฉด ์„ฑ๊ณต์ž…๋‹ˆ๋‹ค. target/ ํด๋”์— ๊ฒฐ๊ณผ๋ฌผ์ด ์ƒ์„ฑ๋ฉ๋‹ˆ๋‹ค.

5. Node.js

์‰ฌ์›€
์‚ฌ์ „ ์ค€๋น„๋ฌผ
  • Git GitHub์—์„œ ํ”„๋กœ์ ํŠธ ์ฝ”๋“œ๋ฅผ ๋‚ด๋ ค๋ฐ›์œผ๋ ค๋ฉด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • Node.js npm ๋ช…๋ น์–ด๋ฅผ ์“ฐ๋ ค๋ฉด Node.js๊ฐ€ ์„ค์น˜๋˜์–ด ์žˆ์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. LTS(์žฅ๊ธฐ์ง€์›) ๋ฒ„์ „์„ ์ถ”์ฒœํ•ฉ๋‹ˆ๋‹ค.
cd connect/connect-http-sink/httpserver-oauth2

์ด ํ”„๋กœ์ ํŠธ์˜ ๊ด€๋ จ ํŒŒ์ผ์ด ํ•˜์œ„ ํด๋” ์•ˆ์— ์žˆ์–ด์„œ, ๋จผ์ € ๊ทธ ํด๋”๋กœ ์ด๋™ํ•ฉ๋‹ˆ๋‹ค.

npm install

package.json์— ๋ช…์‹œ๋œ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋“ค์„ ๋‚ด๋ ค๋ฐ›์•„ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค.

npm start

๊ฐœ๋ฐœ/์‹คํ–‰ ์„œ๋ฒ„๋ฅผ ์ผญ๋‹ˆ๋‹ค.

โœ… ๋ช…๋ น์–ด ์‹คํ–‰ ํ›„ ํ„ฐ๋ฏธ๋„์— ๋‚˜ํƒ€๋‚˜๋Š” ์ฃผ์†Œ(๋ณดํ†ต http://localhost:3000 ํ˜•ํƒœ)๋ฅผ ๋ธŒ๋ผ์šฐ์ €์—์„œ ์—ด์–ด๋ณด์„ธ์š”.

6. Python

์‰ฌ์›€
์‚ฌ์ „ ์ค€๋น„๋ฌผ
  • Git GitHub์—์„œ ํ”„๋กœ์ ํŠธ ์ฝ”๋“œ๋ฅผ ๋‚ด๋ ค๋ฐ›์œผ๋ ค๋ฉด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • Python 3 ์„ค์น˜ ์‹œ 'Add Python to PATH' ์˜ต์…˜์„ ๊ผญ ์ฒดํฌํ•˜์„ธ์š” (Windows).
pip install -r ccloud/client-python/requirements.txt

requirements.txt ๋“ฑ์— ๋ช…์‹œ๋œ ํŒŒ์ด์ฌ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค.

jupyter notebook

๋ธŒ๋ผ์šฐ์ €์—์„œ ๋…ธํŠธ๋ถ(.ipynb) ํŒŒ์ผ๋“ค์„ ์—ด๋žŒํ•˜๊ณ  ์‹คํ–‰ํ•  ์ˆ˜ ์žˆ๋Š” Jupyter ํ™”๋ฉด์„ ์ผญ๋‹ˆ๋‹ค.

โœ… ์—๋Ÿฌ ๋ฉ”์‹œ์ง€ ์—†์ด ์‹คํ–‰๋˜๊ณ  ํ„ฐ๋ฏธ๋„์— ์•ˆ๋‚ด ๋ฌธ๊ตฌ๊ฐ€ ์ถœ๋ ฅ๋˜๋ฉด ์ •์ƒ์ž…๋‹ˆ๋‹ค.

7. Go

๋ณดํ†ต
์‚ฌ์ „ ์ค€๋น„๋ฌผ
  • Git GitHub์—์„œ ํ”„๋กœ์ ํŠธ ์ฝ”๋“œ๋ฅผ ๋‚ด๋ ค๋ฐ›์œผ๋ ค๋ฉด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • Go Go ์–ธ์–ด ์ปดํŒŒ์ผ๋Ÿฌ ๋ฐ ๋„๊ตฌ ๋ชจ์Œ์ž…๋‹ˆ๋‹ค.
cd ccloud/client-go

์ด ํ”„๋กœ์ ํŠธ์˜ ๊ด€๋ จ ํŒŒ์ผ์ด ํ•˜์œ„ ํด๋” ์•ˆ์— ์žˆ์–ด์„œ, ๋จผ์ € ๊ทธ ํด๋”๋กœ ์ด๋™ํ•ฉ๋‹ˆ๋‹ค.

go build ./...

Go ํ”„๋กœ๊ทธ๋žจ์„ ์ปดํŒŒ์ผํ•ด ์‹คํ–‰ ํŒŒ์ผ์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค.

go run .

๋”ฐ๋กœ ๋นŒ๋“œํ•˜์ง€ ์•Š๊ณ  ๋ฐ”๋กœ Go ํ”„๋กœ๊ทธ๋žจ์„ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

โœ… ๋นŒ๋“œ๊ฐ€ ์—๋Ÿฌ ์—†์ด ๋๋‚˜๋ฉด ์„ฑ๊ณต์ž…๋‹ˆ๋‹ค. go run . ์„ ์ผ๋‹ค๋ฉด ํ„ฐ๋ฏธ๋„ ์ถœ๋ ฅ์„ ํ™•์ธํ•˜์„ธ์š”.

8. Make

๋ณดํ†ต
์‚ฌ์ „ ์ค€๋น„๋ฌผ
  • Git GitHub์—์„œ ํ”„๋กœ์ ํŠธ ์ฝ”๋“œ๋ฅผ ๋‚ด๋ ค๋ฐ›์œผ๋ ค๋ฉด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • Make Linux/macOS๋Š” ๋ณดํ†ต ๊ธฐ๋ณธ ์„ค์น˜๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค. Windows๋Š” ๋ณ„๋„ ์„ค์น˜(์˜ˆ: MSYS2, WSL)๊ฐ€ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
cd connect/connect-hbase-sink/kerberos/hbase

์ด ํ”„๋กœ์ ํŠธ์˜ ๊ด€๋ จ ํŒŒ์ผ์ด ํ•˜์œ„ ํด๋” ์•ˆ์— ์žˆ์–ด์„œ, ๋จผ์ € ๊ทธ ํด๋”๋กœ ์ด๋™ํ•ฉ๋‹ˆ๋‹ค.

make

์ƒ์„ฑ๋œ ๋นŒ๋“œ ์„ค์ •์„ ๋ฐ”ํƒ•์œผ๋กœ ์‹ค์ œ ์ปดํŒŒ์ผ์„ ์ง„ํ–‰ํ•ด ์‹คํ–‰ ํŒŒ์ผ์„ ๋งŒ๋“ญ๋‹ˆ๋‹ค.

โœ… ์—๋Ÿฌ ์—†์ด ๋๋‚˜๋ฉด ์„ฑ๊ณต์ž…๋‹ˆ๋‹ค. ์ƒ์„ฑ๋œ ์‹คํ–‰ ํŒŒ์ผ์„ ์ง์ ‘ ์‹คํ–‰ํ•ด๋ณด์„ธ์š”.
// repository documentation