kafka-spark-streaming-integration

KO | EN

gitlite — search

#javascript #react #ai #typescript #android #linux #docker #macos #awesome #awesome-list #vue #pytorch #ios #openai

kafka-spark-streaming-integration

(★ 8)

End to end data pipeline implementation from scratch with Kafka Spark Streaming Integration

kafka-spark-streaming-integration Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# kafka-spark-streaming-integration

This code base are the part of YouTube Binod Suman Academy Channel for End to end data pipeline implementation from 
scratch with Kafka Spark Streaming Integration.

Kafka Spark Streaming Integration in java from scratch | Code walk through. https://youtu.be/UcWi3-FODjs

There are two Maven Project:
1. kafka  
2. Spark

One input file is there tet.csv

Before starting executing these code, you have should have already:
1. Kafka server running  
2. Kafka Topic (demo) shoud be created.  
3. kafka producer and consumer console started for testing purpose.

These YouTube Video will help you to undertand better Kafka, Spark and setup.

Kakfa Installation 
https://youtu.be/D7IY_CwXUKc

Kafka Introduction:
https://youtu.be/HtrKp6V9KuE

Kafka python
https://youtu.be/HIz0pUXhM3U

Kafka Java
https://youtu.be/5AENxG_Bvns

Spark installation on Windows | without Virtual box | Hands on
https://youtu.be/7QAveURI8ZI

Spark Introduction
https://youtu.be/cfiXvwnCz3E

----- Some Useful Kafka Commands -------- 
Start Zookeeper 
zookeeper-server-start.sh config\zookeeper.properties 
 
Start Kafka Broker 
kafka-server-start.sh config/server.properties 
 
Create topic 
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test 
Newer versions(2.2+) of Kafka no longer requires ZooKeeper connection string
./kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 
 
List topic 
kafka-topics.sh --list --zookeeper localhost:2181 
 ./kafka-topics.sh --list --bootstrap-server localhost:9092 
 
Start Producer 
kafka-console-producer.sh --broker-list localhost:9092 --topic test 
 
Send message 
How are you 
Binod Suman Academy 
 
Receive message 
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning 
How are you 
Binod Suman Academy

kafka-spark-streaming-integration

kafka-spark-streaming-integration Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

kafka-spark-streaming-integration

kafka-spark-streaming-integration Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README