Deploy Kafka on Kubernetes

In this article, I will document steps requried to deploy Apache Kafka on Kubernetes (K8s).

Kafka cannot function without Apache Zookeeper. The Kafka service keeps restarting until a working Zookeeper deployment is detected.

Deploy Zookeeper

First, we need to create a deployment resource for ZooKeeper (zk). Create a zk-deploy.yaml file:

apiVersion: extensions/v1
kind: Deployment
metadata:
  name: zk-deploy
spec:
  selector:
    matchLabels:
      app: zk-1
  template:
    metadata:
      labels:
        app: zk-1
    spec:
      containers:
      - name: zk1
        image: zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zk1

• Enviroment variable ZOOKEEPER_SERVER_1 should be same as the container name
• The key-value pairs in spec.selectors.matchLabels and spec.templace.metadata.labels should be same because that's how deployment zk-deploy selects pods to manage.

Create deployment resource in K8s via:

$ kubectl create -f zk-deploy.yaml
deployment.apps/zk-deploy created

Next, create a zk-service.yaml file for creating ZK service resource:

apiVersion: v1
kind: Service
metadata:
  name: zk-s
  labels:
    app: zk-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zk-1

• Notice that the metadata.labels.app and specs.selector.app values should be same as the

Create zk-s service via:

$ kubectl apply -f zk-service.yaml
service/zk-s created

Deploying Kafka

Create following kafka-svc.yaml file:

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    app: kafka-broker
spec:
  selector:
    app: kafka-broker
  type: LoadBalancer
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP

and run the following command:

$ kubectl apply -f kafka-svc.yaml
service/kafka-service created

Next, create kafka-broker.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kafka-broker
  name: kafka-broker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker
  template:
    metadata:
      labels:
        app: kafka-broker
    spec:
      hostname: kafka-broker
      containers:
      - env:
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: 10.99.232.213:2181
        - name: KAFKA_LISTENERS
          value: PLAINTEXT://:9092
        - name: KAFKA_ADVERTISED_LISTENERS
          value: PLAINTEXT://kafka-broker:9092
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        name: kafka-broker
        ports:
        - containerPort: 9092

and create deployment resource via:

$ kubectl apply -f kafka-broker.yaml
deployment.apps/kafka-broker created

To ensure that Zookeeper and Kafka can communicate by using this hostname (kafka-broker), we need to add the following entry to the /etc/hosts (or C:\Windows\System32\drivers\etc\hosts) file on our local machine:

127.0.0.1   kafka-broker

Testing Kafka Topics

Get kafka pod name via kubectl command:

$ kubectl get pods | grep kafka
NAME                           READY   STATUS    RESTARTS      AGE
kafka-broker-7b6fd5666-4tjxb   1/1     Running   0             14m

In order to test that we can send and retrieve messages from a topic in Kafka, we will need to expose a port for Kafka to make it accessible from localhost:

$ kubectl port-forward kafka-broker-7b6fd5666-4tjxb 9092
Forwarding from 127.0.0.1:9092 -> 9092

Install kcat

To easily send and retrieve messages from Kafka, we’ll use a command-line tool named KCat (formerly Kafkacat). Install kcat command in Ubuntu (or WSL on Windows 11) via:

$ sudo apt-get install kafkacat

Test producer and consumer

To create a message and a topic named empire-of-the-sun, we create a producer via the following command:

$ echo "we are the people" | kafkacat -P -b localhost:9092 -t empire-of-the-sun

• The command should run without any error.

To create a consumer to read messages from the topic empire-of-the-sun:

$ kafkacat -C -b localhost:9092 -t empire-of-the-sun
we are the people
% Reached end of topic empire-of-the-sun [0] at offset 1

This proves that our kafka deployment is working OK.

© 2022 Sumeet Das