Saturday, August 27, 2016

On Kafka

This blog captures steps to install and setup Kafka.

For instructions on how to install Kafka, visit: Installation Steps

After downloading:
untar bundle to /opt/apache-kafka
ln -s kafka_2.11-0.9.0.1 current
cd /opt/apache-kafka/current

At this point you can start Zookeper:

bin/zookeeper-server-start.sh config/zookeeper.properties&

To run Kafka on the current server, modify config/server.properties:

Uncomment:

port=9092
host.name=localhost

To allow for removing topics, add the following to config/server.properties:

# Enable topic deletion - only for development
delete.topic.enable=true

Now you can start Kafka:

bin/kafka-server-start.sh config/server.properties&

Kafka REST Proxy
In order to publish messages to Kafka via REST calls, you can use Kafka REST Proxy. To install it, visit Confluent Platform Quickstart.

Untar the downloaded bundle to /opt/confluent:

ln -s confluent-2.0.1 current
cd current
vi etc/kafka-rest/kafka-rest.properties

Uncomment

id=kafka-rest-test-server
schema.registry.url=http://localhost:8081
zookeeper.connect=localhost:2181

To run Kafka REST Proxy:
bin/schema-registry-start etc/schema-registry/schema-registry.properties &
bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties &

Some useful aliases to start/stop/view Kafka processes
alias zs="cd /opt/apache-kafka/current; nohup bin/zookeeper-server-start.sh config/zookeeper.properties 2>/dev/null &"
alias ks="cd /opt/apache-kafka/current; nohup bin/kafka-server-start.sh config/server.properties 2>/dev/null&"

alias krp1="cd /opt/confluent/current; nohup bin/schema-registry-start etc/schema-registry/schema-registry.properties 2>/dev/null&"
alias krp2="cd /opt/confluent/current; nohup bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties 2>/dev/null&"

alias pk="ps -aef|grep kafka"
alias topics="/opt/apache-kafka/current/bin/kafka-topics.sh  --list --zookeeper localhost:2181"

To create topics:
cd/opt/apache-kafka/current; bin/kafka-topics.sh --create --topic test --zookeeper YOUR.IP.ADDRESS.HERE:2181 --partition 1 --replication-factor 1

To list topics:
cd /opt/apache-kafka/current; bin/kafka-topics.sh --list --zookeeper YOUR.IP.ADDRESS.HERE:2181

To consume messages from a topic:
/opt/apache-kafka/current/bin/kafka-console-consumer.sh --zookeeper YOUR.IP.ADDRESS.HERE:2181 --topic MY_TOPIC 


No comments:

Post a Comment