kafka-demo

KO | EN

gitlite — search

#javascript #react #go #machine-learning #linux #nodejs #macos #windows #cli #kubernetes #vue #ios #mcp #openai

(★ 17)

Simple demo Python client scripts for Apache Kafka

kafka-demo 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Kafka demo scripts
==================

This is the demo kit that accompanies the talk I gave at PyConZA 2015 on Apache
Kafka. It contains scripts for demoing a producer and balanced consumers.

You can watch the talk here:

* https://www.youtube.com/watch?v=b8Cj5-LieH0

Dependencies
------------

This demo requires Kafka 0.8.2. If you do not have access to a Kafka cluster,
you can set it up in standalone mode, by downloading it here:

* http://kafka.apache.org/downloads.html

and installing it following the instructions here:

* http://kafka.apache.org/documentation.html#quickstart

You will also need the pykafka client library. I used version 2.0.0, which was
the latest version at the time of writing. Get it using

pip install pykafka

or from the source repo here:

*  https://github.com/Parsely/pykafka

Configuration
-------------

Edit `settings.conf` in this repo to connect to your Kafka and Zookeeper
hosts. By default, it is set up for a standalone Kafka installation on
localhost.

For the tests below to run properly, create a topic called "test" by executing

cd /opt/kafka  # This is the default install directory
    ./bin/kafka-topics.sh --topic test --partitions 6 --create \
        --zookeeper localhost:2181 --replication-factor 1

The test topic will have 6 partitions (more on this in Demo 4 below).

To check that your topic got created properly, you can execute

./bin/kafka-topics.sh --describe --zookeeper localhost:2181

This will display information about all Kafka topics.

Demo 1: Running a producer
--------------------------

python demo-producer.py

This will produce messages as fast as possible and display how many got
produced, once every 5 seconds.

Demo 2: Running a consumer
--------------------------

python demo-consumer.py

This will consume already produced messages as fast as possible and display how
many got consumed, once every 5 seconds. It will also show the list of
partitions from which messages got consumed. More on partitions in demo 4.

The consumer will block when it runs out of messages.

Demo 3: Clearing out your Kafka and Zookeeper data
--------------------------------------------------

To delete all Kafka logs and Zookeeper data, you can do the following. This is
for test purposes only and should obviously never be done in production.

1. Stop your Kafka server
  2. Stop your Zookeeper server
  3. `rm -Rf /tmp/zookeeper/*`
     This is the default data location for a standalone Zookeeper install. See
     the `dataDir` property in `config/zookeeper.properties` for where your
     installation is storing things.
  4. `rm -Rf /tmp/kafka-logs/*`
     This is the default data location for a standalone Kafka install. See the
     `log.dirs` property in `config/server.properties` for where your
     installation is storing its Kafka logs.
  5. Start your Zookeeper server
  6. Start your Kafka server
  7. Re-create your Kafka topics

Demo 4: Running a producer and multiple, balanced consumers
-----------------------------------------------------------

Run `demo-producer.py` and a few copies of `demo-consumer.py` simultaneously,
in different terminals. You should see that the different consumers
automatically read from different partitions. If a consumer is killed or added,
the partitions being read will automatically be rebalanced between the
consumers.

Note that this will work only if your topic has more than 1 partition. Nothing
useful happens if you have more consumers than you have partitions. See the
Configuration section above on how to create a topic with multiple partitions.

kafka-demo

kafka-demo 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

kafka-demo

kafka-demo 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README