Install and Configure Apache Kafka and the Confluent Platform using Homebrew

If you’re on a Mac, among the many ways to install Apache Kafka and the Confluent Platform, there is Homebrew.

The Confluent Platform is available there and installing it is as easy as brew install confluent-platform.

Unfortunately, once you’ve run that command, all the Confluent components are installed on your machine but it’s not ready to start and there’s not much documentation to help you.

The main advantages with using Homebrew to manage your Confluent packages is that you can have multiple versions of Confluent/Kafka installed and easily switch between them and it uses a lot less resources than Docker.

Because of my work, I constantly have to alter my Kafka setup and try new things, so in this article, I’ll focus on starting and configuring things manually.

To install, run:

brew install confluent-platform

Once this is done, your install is not fully functional. You also have to create the appropriate /var/log folder:

sudo mkdir -p /var/log/kafka
sudo chown -R $(whoami) /var/log/kafka

Then you’re ready to start a zookeeper and a broker.

If you want to modify the configuration first, edit those files:

For zookeeper: /usr/local/etc/kafka/zookeeper.properties

For the broker: /usr/local/etc/kafka/server.properties

For connect: /usr/local/etc/kafka/connect-distributed.properties

Note that /usr/local/etc/kafka points to /usr/local/Cellar/confluent-platform/5.3.1/etc/kafka/ if you’re using 5.3.1.

When you’re happy with the config files, open up a bunch of terminal windows and run the various components:

First, zookeeper:

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

Then a broker:

kafka-server-start /usr/local/etc/kafka/server.properties

For a Connect in distributed mode:

connect-distributed /usr/local/etc/kafka/connect-distributed.properties`

Of course, in addition to the above, all the tools of the platform are installed:

$ ls /usr/local/Cellar/confluent-platform/5.3.1/bin/
confluent-hub				kafka-api-start				kafka-mqtt-run-class			kafka-streams-application-reset		schema-registry-start
confluent-rebalancer			kafka-avro-console-consumer		kafka-mqtt-start			kafka-topics				schema-registry-stop
connect-distributed			kafka-avro-console-producer		kafka-mqtt-stop				kafka-verifiable-consumer		schema-registry-stop-service
connect-standalone			kafka-broker-api-versions		kafka-preferred-replica-election	kafka-verifiable-producer		security-plugins-run-class
control-center-3_0_0-reset		kafka-configs				kafka-producer-perf-test		ksql					sr-acl-cli
control-center-3_0_1-reset		kafka-console-consumer			kafka-reassign-partitions		ksql-datagen				stub-api-run-class
control-center-console-consumer		kafka-console-producer			kafka-replica-verification		ksql-print-metrics			stub-api-start
control-center-export			kafka-consumer-groups			kafka-rest-run-class			ksql-run-class				support-metrics-bundle
control-center-reset			kafka-consumer-perf-test		kafka-rest-start			ksql-server-start			zookeeper-security-migration
control-center-run-class		kafka-delegation-tokens			kafka-rest-stop				ksql-server-stop			zookeeper-server-start
control-center-set-acls			kafka-delete-records			kafka-rest-stop-service			ksql-stop				zookeeper-server-stop
control-center-start			kafka-dump-log				kafka-run-class				ksql-test-runner			zookeeper-shell
control-center-stop			kafka-log-dirs				kafka-server-start			replicator
kafka-acls				kafka-mirror-maker			kafka-server-stop			schema-registry-run-class

You can use them locally, for example:

$ kafka-topics --zookeeper localhost:2181 --list
__confluent.support.metrics
__consumer_offsets
connect-configs
connect-offsets
connect-status
file_source
laurent
~ $ kafka-console-producer --broker-list localhost:9092 --topic laurent
>one more line
>
$

If you want more than one zookeeper and one broker (which is enough for many local development tasks or configuration tests), you have to duplicate the configuration files, alter them (at least, add a broker-id into the broker config and point the brokers to separate log.dirs folders) and run the above commands into additional terminal windows pointing to separate config files.

iTerm galore

And, yes, you’ll end up with many terminal windows, but who cares.

‹»Carl von Ossietzky« »Switch Between Apache Kafka and Confluent Versions with Homebrew«›