Saturday, August 27, 2016

On Kafka

This blog captures steps to install and setup Kafka.

For instructions on how to install Kafka, visit: Installation Steps

After downloading:
untar bundle to /opt/apache-kafka
ln -s kafka_2.11- current
cd /opt/apache-kafka/current

At this point you can start Zookeper:

bin/ config/

To run Kafka on the current server, modify config/



To allow for removing topics, add the following to config/

# Enable topic deletion - only for development

Now you can start Kafka:

bin/ config/

Kafka REST Proxy
In order to publish messages to Kafka via REST calls, you can use Kafka REST Proxy. To install it, visit Confluent Platform Quickstart.

Untar the downloaded bundle to /opt/confluent:

ln -s confluent-2.0.1 current
cd current
vi etc/kafka-rest/



To run Kafka REST Proxy:
bin/schema-registry-start etc/schema-registry/ &
bin/kafka-rest-start etc/kafka-rest/ &

Some useful aliases to start/stop/view Kafka processes
alias zs="cd /opt/apache-kafka/current; nohup bin/ config/ 2>/dev/null &"
alias ks="cd /opt/apache-kafka/current; nohup bin/ config/ 2>/dev/null&"

alias krp1="cd /opt/confluent/current; nohup bin/schema-registry-start etc/schema-registry/ 2>/dev/null&"
alias krp2="cd /opt/confluent/current; nohup bin/kafka-rest-start etc/kafka-rest/ 2>/dev/null&"

alias pk="ps -aef|grep kafka"
alias topics="/opt/apache-kafka/current/bin/  --list --zookeeper localhost:2181"

To create topics:
cd/opt/apache-kafka/current; bin/ --create --topic test --zookeeper YOUR.IP.ADDRESS.HERE:2181 --partition 1 --replication-factor 1

To list topics:
cd /opt/apache-kafka/current; bin/ --list --zookeeper YOUR.IP.ADDRESS.HERE:2181

To consume messages from a topic:
/opt/apache-kafka/current/bin/ --zookeeper YOUR.IP.ADDRESS.HERE:2181 --topic MY_TOPIC 


  1. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your.
    Digital Marketing Company in India


  2. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

    Personal Installment Loans
    Payday Cash Advance loan
    Title Car loan
    Cash Advance Loan