
Install and Configure Apache Kafka and the Confluent Platform using Homebrew
If you’re on a Mac, among the many ways to install Apache Kafka and the Confluent Platform, there is Homebrew.
The Confluent Platform is available there and installing it is as easy as brew install confluent-platform
.
Unfortunately, once you’ve run that command, all the Confluent components are installed on your machine but it’s not ready to start and there’s not much documentation to help you.
The main advantages with using Homebrew to manage your Confluent packages is that you can have multiple versions of Confluent/Kafka installed and easily switch between them and it uses a lot less resources than Docker.
Because of my work, I constantly have to alter my Kafka setup and try new things, so in this article, I’ll focus on starting and configuring things manually.
To install, run:
brew install confluent-platform
Once this is done, your install is not fully functional. You also have to create the appropriate /var/log
folder:
sudo mkdir -p /var/log/kafka
sudo chown -R $(whoami) /var/log/kafka
Then you’re ready to start a zookeeper and a broker.
If you want to modify the configuration first, edit those files:
For zookeeper: /usr/local/etc/kafka/zookeeper.properties
For the broker: /usr/local/etc/kafka/server.properties
For connect: /usr/local/etc/kafka/connect-distributed.properties
Note that /usr/local/etc/kafka
points to /usr/local/Cellar/confluent-platform/5.3.1/etc/kafka/
if you’re using 5.3.1.
When you’re happy with the config files, open up a bunch of terminal windows and run the various components:
First, zookeeper:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
Then a broker:
kafka-server-start /usr/local/etc/kafka/server.properties
For a Connect in distributed mode:
connect-distributed /usr/local/etc/kafka/connect-distributed.properties`
Of course, in addition to the above, all the tools of the platform are installed:
$ ls /usr/local/Cellar/confluent-platform/5.3.1/bin/
confluent-hub kafka-api-start kafka-mqtt-run-class kafka-streams-application-reset schema-registry-start
confluent-rebalancer kafka-avro-console-consumer kafka-mqtt-start kafka-topics schema-registry-stop
connect-distributed kafka-avro-console-producer kafka-mqtt-stop kafka-verifiable-consumer schema-registry-stop-service
connect-standalone kafka-broker-api-versions kafka-preferred-replica-election kafka-verifiable-producer security-plugins-run-class
control-center-3_0_0-reset kafka-configs kafka-producer-perf-test ksql sr-acl-cli
control-center-3_0_1-reset kafka-console-consumer kafka-reassign-partitions ksql-datagen stub-api-run-class
control-center-console-consumer kafka-console-producer kafka-replica-verification ksql-print-metrics stub-api-start
control-center-export kafka-consumer-groups kafka-rest-run-class ksql-run-class support-metrics-bundle
control-center-reset kafka-consumer-perf-test kafka-rest-start ksql-server-start zookeeper-security-migration
control-center-run-class kafka-delegation-tokens kafka-rest-stop ksql-server-stop zookeeper-server-start
control-center-set-acls kafka-delete-records kafka-rest-stop-service ksql-stop zookeeper-server-stop
control-center-start kafka-dump-log kafka-run-class ksql-test-runner zookeeper-shell
control-center-stop kafka-log-dirs kafka-server-start replicator
kafka-acls kafka-mirror-maker kafka-server-stop schema-registry-run-class
You can use them locally, for example:
$ kafka-topics --zookeeper localhost:2181 --list
__confluent.support.metrics
__consumer_offsets
connect-configs
connect-offsets
connect-status
file_source
laurent
~ $ kafka-console-producer --broker-list localhost:9092 --topic laurent
>one more line
>
$
If you want more than one zookeeper and one broker (which is enough for many local development tasks or configuration tests), you have to duplicate the configuration files, alter them (at least, add a broker-id
into the broker config and point the brokers to separate log.dirs
folders) and run the above commands into additional terminal windows pointing to separate config files.
And, yes, you’ll end up with many terminal windows, but who cares.