Switch Between Apache Kafka and Confluent Versions with Homebrew
Photo of Prague by Vlad Kiselov

Switch Between Apache Kafka and Confluent Versions with Homebrew

Once you’ve installed Apache Kafka and the Confluent Platform using Homebrew, there are many ways to maintain multiple versions of the package on your local computer.

(This is kinda part 2 of Install and Configure Apache Kafka and the Confluent Platform using Homebrew.)

To switch from a version to another with Homebrew (here: 5.3.1), simply do:

brew switch confluent-platform 5.3.1

This is great but it relies on the fact that the version you want to switch to is currently installed locally in your cellar. If it’s the case, Homebrew does the switch instantly by updating symlinks.

There are several ways to ensure that you have all the versions that you want in your local computer. At some point, they will collide with the fact that Homebrew does a cleanup every 30 days or each time you run an upgrade, unless you the HOMEBREW_NO_INSTALL_CLEANUP envvar.

You can:

  1. Use HOMEBREW_NO_INSTALL_CLEANUP when upgrading all Homebrew packages to keep the previous versions on your disk. Unless it is set, brew cleanup will be run for the formula that you’re upgrading or, every 30 days, for all formulae. It will fill your disk quite quickly if you use a lot of packages and unless you’re super careful, you’ll loose your previous versions sooner or later.

  2. Build you own keg and feed it with the versions that you want to keep using brew extract. Too much work IMHO.

  3. Use HOMEBREW_NO_INSTALL_CLEANUP when upgrading confluent-platform. If the version you want is gone, restore it using brew install referencing the specific commit number. (Also remember to backup the Confluent/Kafka config files in a folder outside of Homebrew.)

In this article, I’m describing Option 3.

Reference points:

First Install

To install for the first time, run:

brew install confluent-platform

Upgrades

To upgrade without removing the previous versions, run:

~ $ HOMEBREW_NO_INSTALL_CLEANUP=1 brew upgrade confluent-platform
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
.../...

==> Upgrading 1 outdated package:
confluent-platform 5.3.1 -> 5.4.0
==> Upgrading confluent-platform
==> Downloading https://packages.confluent.io/archive/5.4/confluent-5.4.0-2.12.tar.gz
######################################################################## 100.0%
🍺  /usr/local/Cellar/confluent-platform/5.4.0: 2,943 files, 951.4MB, built in 85 minutes 52 seconds
==> Checking for dependents of upgraded formulae...
==> No dependents found!

(To set the option permanently and for all the Homebrew packages, you can add export HOMEBREW_NO_INSTALL_CLEANUP=1 to your bash profile but I don’t recommend it.)

To check what you have locally (in this example, see line 7 and 9, I have 5.3.1 and 5.4.0):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
~ $ brew info confluent-platform
confluent-platform: stable 5.4.0
Developer-optimized distribution of Apache Kafka
https://www.confluent.io/product/confluent-platform/
Conflicts with:
 kafka (because kafka also ships with identically named Kafka related executables)
/usr/local/Cellar/confluent-platform/5.3.1 (2,499 files, 827.2MB)
 Built from source on 2020-02-26 at 14:22:54
/usr/local/Cellar/confluent-platform/5.4.0 (2,943 files, 951.4MB) *
 Built from source on 2020-02-28 at 08:08:33
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/confluent-platform.rb
==> Requirements
Required: java = 1.8 ✔
==> Analytics
install: 317 (30 days), 572 (90 days), 1,593 (365 days)
install-on-request: 316 (30 days), 570 (90 days), 1,578 (365 days)
build-error: 0 (30 days)

You can also check what on the disk:

~ $ ls /usr/local/Cellar/confluent-platform/
5.3.1	5.4.0

Re-install of a Previous Version

If you want to re-install a previous version:

  1. Get the commit number at https://github.com/Homebrew/homebrew-core/commits/master/Formula/confluent-platform.rb. If it says “this commit history is taking too long to generate”, then clone homebrew-core and check the commits there with git log master -- Formula/confluent-platform.rb.

From the above page, you can get the commit numbers of the previous versions. For your convenience, I am listing here some of the recent ones:

For prior versions, it’s a little more complicated as the formula was named confluent-oss before.

  1. Reference the commit number in the install command, for example to install 5.3.0:
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/6a76b682ff4969afec52cc34be58315cee3271fe/Formula/confluent-platform.rb

Now, check what you have (see above) and go crazy switching versions:

One last thing:

  • Don’t forget to stop running instances before upgrade/switch operations. (Ctrl-C on your terminals…)
  • To fully reset your data (delete your topics and log files) between version switches, find where the topics are stored (check the log.dirs config property, by default it’s in /tmp/kafka-logs):
rm -rf /tmp/kafka-logs
rm -rf /var/log/kafka/*

That’s all, folks!

Written by Ldom //