in Uncategorized

Installing Kafka and Zookeeper on Vagrant Ubuntu 16.04

Step 1–Set up Ubuntu 16.04 Virtual Machine

First, follow steps 1 through 3 in my previous tutorial. NOTE: in your Vagrantfile you may need to use /usr/vagrant/api as the directory instead of /usr/ubuntu/api. I’m not sure why, but some installs use vagrant as the root user, and others use ubuntu. I’m sure there’s some reason why, but I haven’t figured it out.

Step 2–Install the latest JDK

Second, log into your box by typing vagrant ssh on the command line and follow these instructions for installing the latest JDK. (Or these.)

Step 3–Install Zookeeper

Third, install Apache Zookeeper with the following command:

$ sudo apt-get install zookeeperd

It should automatically start up on port 2181. You can verify this with the following command:

$ netstat -ant | grep :2181

Which should give you output like the following:

tcp6       0      0 :::2181                 :::*                    LISTEN

Zookeeper is a small, fast “coordination service” that is maintained by Apache. Rather than having everybody (like Hadoop, Kafta, Drill, etc.) develop and include their own coordination service into their distributed software package, Apache decided to build a really high quality, lightweight coordinator (Zookeeper) that can be used by all of these other distributed systems to manage configuration, assure message ordering, etc. for your entire cluster. Here’s an overview, if you’d like to know more.

Step 4

Fourth, go to the Kafka downloads page. Click on the first binary version available, next to where it says Scala 2.11 (currently version 1.0.1). At the top of the page that comes up there will be a link after the words: “We suggest the following mirror site for your download:” Right click and copy the link address for your download. Then in your terminal, type wget and then a space and then paste the URL you just copied, and then hit Enter. For example:

$ wget

Once it’s done, if you view the contents of the directory, it should look something like this:

vagrant@ubuntu-xenial:~$ ls -la
total 84444
drwxr-xr-x 6 vagrant vagrant     4096 Mar 11 05:17 .
drwxr-xr-x 4 root    root        4096 Mar 11 04:59 ..
-rw-r--r-- 1 vagrant vagrant      220 Mar  9 08:35 .bash_logout
-rw-r--r-- 1 vagrant vagrant     3771 Mar  9 08:35 .bashrc
drwx------ 2 vagrant vagrant     4096 Mar 11 05:00 .cache
drwxr-xr-x 1 vagrant vagrant       68 Mar 11 04:31 kafka
-rw-rw-r-- 1 vagrant vagrant 49766096 Mar  5 19:00 kafka_2.11-1.0.1.tgz
drwxrwxr-x 2 vagrant vagrant     4096 Mar 11 05:17 .oracle_jre_usage
-rw-r--r-- 1 vagrant vagrant      655 Mar  9 08:35 .profile
drwx------ 2 vagrant vagrant     4096 Mar 11 05:00 .ssh

Unzip the Kafka download as follows (using the filename for the version of Kafka that YOU downloaded–it might be different than the one listed in this tutorial):

$ tar -xzf kafka_2.11-1.0.1.tgz

This will unzip the contents of the file into a directory named kafka_2.11-1.0.1. cd into that directory and then you can start up a Kafka server instance as follows:

$ cd kafka_2.11-1.0.1
$ bin/ config/ &

Congratulations! Assuming all has gone well, you should now have a Kafka server up and running. If you’d like to do something basic with it, pick up at Step 3 of the Quickstart on the Kafka website.

I’m still learning about what Kafka is and what can be done with it. When I find out, I’ll blog about it here!

Photo Credit:

TED-Ed “What makes something ‘Kafkaesque’?” by Noah Tavlin on YouTube

Write a Comment