Portainer Templates logo

Portainer Templates

Apache Kafka Apache Kafka

Stack

Streaming Messaging

3-node Apache Kafka cluster in KRaft mode (no ZooKeeper) with Kafbat UI, deployable locally with plain Docker Compose or to Miget as a compose stack.

Configuration

Type
Compose
Platform
linux
Source
Repo

Template by deployable-sh·Source

Report issue

Services

kafka-1

Configuration

Image
apache/kafka:4.0.0
Ports
90929093
Volumes
/var/lib/kafka/data : kafkadata-1
Env vars
KAFKA_PROCESS_ROLES=broker,controllerKAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXTKAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLERKAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXTKAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093KAFKA_LOG_DIRS=/var/lib/kafka/dataKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=3KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=2KAFKA_DEFAULT_REPLICATION_FACTOR=3KAFKA_MIN_INSYNC_REPLICAS=2KAFKA_HEAP_OPTS=-Xms256m -Xmx1gCLUSTER_ID=5L6g3nShT-eMCtK--X86swKAFKA_NODE_ID=1KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-1:9092
Restart
unless-stopped

Image details

Pulls: 44.5M
User: dfoulks
Created: Nov 27, 2023
Updated: 2 days ago
Status: active

kafka-2

Configuration

Image
apache/kafka:4.0.0
Ports
90929093
Volumes
/var/lib/kafka/data : kafkadata-2
Env vars
KAFKA_PROCESS_ROLES=broker,controllerKAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXTKAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLERKAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXTKAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093KAFKA_LOG_DIRS=/var/lib/kafka/dataKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=3KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=2KAFKA_DEFAULT_REPLICATION_FACTOR=3KAFKA_MIN_INSYNC_REPLICAS=2KAFKA_HEAP_OPTS=-Xms256m -Xmx1gCLUSTER_ID=5L6g3nShT-eMCtK--X86swKAFKA_NODE_ID=2KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-2:9092
Restart
unless-stopped

Image details

Pulls: 44.5M
User: dfoulks
Created: Nov 27, 2023
Updated: 2 days ago
Status: active

kafka-3

Configuration

Image
apache/kafka:4.0.0
Ports
90929093
Volumes
/var/lib/kafka/data : kafkadata-3
Env vars
KAFKA_PROCESS_ROLES=broker,controllerKAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXTKAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLERKAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXTKAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093KAFKA_LOG_DIRS=/var/lib/kafka/dataKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=3KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=3KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=2KAFKA_DEFAULT_REPLICATION_FACTOR=3KAFKA_MIN_INSYNC_REPLICAS=2KAFKA_HEAP_OPTS=-Xms256m -Xmx1gCLUSTER_ID=5L6g3nShT-eMCtK--X86swKAFKA_NODE_ID=3KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-3:9092
Restart
unless-stopped

Image details

Pulls: 44.5M
User: dfoulks
Created: Nov 27, 2023
Updated: 2 days ago
Status: active

kafka-ui

Configuration

Image
ghcr.io/kafbat/kafka-ui:v1.5.0
Ports
5000:5000
Env vars
KAFKA_CLUSTERS_0_NAME=kafkaKAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka-1:9092,kafka-2:9092,kafka-3:9092SERVER_PORT=5000
Restart
unless-stopped

Image details

User: kafbat

Standalone Install

Select an install method, to see config/commands for deploying Apache Kafka

Installation method

Install on Portainer

Import all app templates into your Portainer instance, for easy 1-click deploys

  1. Ensure both Docker and Portainer are installed, and up-to-date
  2. Log into your Portainer web UI
  3. Under Settings → App Templates, paste the below URL
  4. Head to Home → App Templates, and the list of apps will show up
  5. Select Apache Kafka, fill in any config options, and hit Deploy

Template Import URL

https://raw.githubusercontent.com/Lissy93/portainer-templates/main/templates.json
Show Me demo
Original stackfile

The compose file this template deploys, straight from its repo:

name: kafka

x-kafka: &kafka
  image: apache/kafka:4.0.0
  restart: unless-stopped

x-kafka-env: &kafka-env
  KAFKA_PROCESS_ROLES: broker,controller
  KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
  KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
  KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
  KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
  KAFKA_LOG_DIRS: /var/lib/kafka/data
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "3"
  KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: "3"
  KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: "2"
  KAFKA_DEFAULT_REPLICATION_FACTOR: "3"
  KAFKA_MIN_INSYNC_REPLICAS: "2"
  KAFKA_HEAP_OPTS: -Xms256m -Xmx1g
  CLUSTER_ID: 5L6g3nShT-eMCtK--X86sw

services:
  kafka-1:
    <<: *kafka
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: "1"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9092
    ports:
      - "9092"
      - "9093"
    volumes:
      - kafkadata-1:/var/lib/kafka/data

  kafka-2:
    <<: *kafka
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: "2"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:9092
    ports:
      - "9092"
      - "9093"
    volumes:
      - kafkadata-2:/var/lib/kafka/data

  kafka-3:
    <<: *kafka
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: "3"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:9092
    ports:
      - "9092"
      - "9093"
    volumes:
      - kafkadata-3:/var/lib/kafka/data

  kafka-ui:
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:5000/actuator/health >/dev/null 2>&1"]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 60s
    image: ghcr.io/kafbat/kafka-ui:v1.5.0
    restart: unless-stopped
    environment:
      KAFKA_CLUSTERS_0_NAME: kafka
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka-1:9092,kafka-2:9092,kafka-3:9092
      SERVER_PORT: "5000"
    ports:
      - "5000:5000"
    depends_on:
      - kafka-1
      - kafka-2
      - kafka-3

volumes:
  kafkadata-1:
  kafkadata-2:
  kafkadata-3:

Or deploy it directly from the source:

git clone https://github.com/deployable-sh/stacks
cd stacks
docker compose -f kafka/compose.yaml up -d

More install options in our documentation.

Container Documentation

kafka-1 Documentation

Apache Kafka

kafka-2 Documentation

Apache Kafka

kafka-3 Documentation

Apache Kafka

Serve Apache Kafka on your own domain behind Caddy, Nginx or Traefik. Fill in your domain and copy the result. It's a starting point, some apps need their own base URL or extra headers set too.

Proxying apache-kafka.example.com to http://kafka-1:9092

Add this to your Caddyfile

apache-kafka.example.com {
	reverse_proxy http://kafka-1:9092
}

Check the logs first

Nine times out of ten the logs tell you exactly what went wrong.

  • In Portainer, go to Containers, click the container, then Logs. Or run docker logs <container>
  • Exit codes help too: 137 means killed, usually out of memory. 126 or 127 means the command inside the image is broken.

Port already in use

If deployment fails with "Bind for 0.0.0.0:5000 failed: port is already allocated", something else on your server is using that port.

  • Find what's using it: sudo ss -tlnp | grep :5000
  • Stop the other service, or pick a different host port. In 5000:5000 only the left number is yours to change, the right one belongs to the app.

Published on a random port

This template exposes 9092 without setting a host port, so Docker picks a random free one on every deploy.

  • Find it in the Ports column of Portainer's container list, or with docker port <container>

Running but the page won't load

The container is up but nothing appears in your browser.

  • Use your server's real IP: http://your-server-ip:5000. The 0.0.0.0 link Portainer shows isn't a real address.
  • Give it a minute after first deploy, kafka can take a while to initialise.
  • Make sure your firewall allows the port, e.g. sudo ufw allow 5000

Image won't pull

Test the pull directly on the host: docker pull apache/kafka:4.0.0

  • "manifest unknown" means the tag no longer exists.
  • "toomanyrequests" is the Docker Hub rate limit. Log in with docker login to raise it.
  • "no space left on device" means a full disk. Reclaim space with docker system prune

"exec format error"

This means the image was built for a different CPU architecture than your server.

  • Check yours with uname -m: x86_64 is amd64, aarch64 is arm64. Raspberry Pi and other ARM boards are the usual culprits.

Container keeps restarting

The unless-stopped restart policy relaunches the app after every crash, so the real error can scroll past.

  • Check the logs right after a restart, the last few lines before it died are the useful ones.
  • Get the exit code with docker inspect <container> --format '{{.State.ExitCode}}'
  • Still stuck? Redeploy once with the restart policy set to no so the failure stays visible.

Stack won't deploy

Compose stacks fail fast on small mistakes, and Portainer shows the reason just above the editor.

  • YAML only accepts spaces for indentation, a single tab breaks the whole file.

Raise an issue

Found something which isn't working as it should? Here's how to report it.

A Compose stack

Apache Kafka is a Compose stack, a set of containers (4 of them) defined in one file and brought up together by Portainer, then started and stopped as a single app.

The services

This stack is built from 4 containers that run side by side. Here's each one, with the image it runs and anything it waits for first:

  • kafka-1 runs apache/kafka:4.0.0
  • kafka-2 runs apache/kafka:4.0.0
  • kafka-3 runs apache/kafka:4.0.0
  • kafka-ui runs ghcr.io/kafbat/kafka-ui:v1.5.0, starts after kafka-1, kafka-2, kafka-3

Ports

A port is the door the app answers on. A mapping like 5000:5000 means it's reachable on port 5000 of your server, where the left number is yours to change and the right one belongs to the app. It opens:

  • 9092, published on a random host port
  • 9093, published on a random host port
  • 9092, published on a random host port
  • 9093, published on a random host port
  • 9092, published on a random host port
  • 9093, published on a random host port
  • 5000:5000

Volumes

A volume is where Apache Kafka keeps its files so they survive an update or a restart. Without one, anything it saves would sit inside the container and vanish the moment it's recreated. This template mounts:

  • /var/lib/kafka/data kept in the kafkadata-1 volume Docker manages
  • /var/lib/kafka/data kept in the kafkadata-2 volume Docker manages
  • /var/lib/kafka/data kept in the kafkadata-3 volume Docker manages

Environment variables

Environment variables are the settings you hand over when you deploy, things like a password or a timezone. Apache Kafka takes 19 of them, all with defaults you can leave alone or tweak:

  • KAFKA_PROCESS_ROLES, defaults to broker,controller
  • KAFKA_CONTROLLER_QUORUM_VOTERS, defaults to 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
  • KAFKA_LISTENER_SECURITY_PROTOCOL_MAP, defaults to CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
  • KAFKA_CONTROLLER_LISTENER_NAMES, defaults to CONTROLLER
  • KAFKA_INTER_BROKER_LISTENER_NAME, defaults to PLAINTEXT
  • KAFKA_LISTENERS, defaults to PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
  • KAFKA_LOG_DIRS, defaults to /var/lib/kafka/data
  • KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR, defaults to 3
  • KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR, defaults to 3
  • KAFKA_TRANSACTION_STATE_LOG_MIN_ISR, defaults to 2
  • KAFKA_DEFAULT_REPLICATION_FACTOR, defaults to 3
  • KAFKA_MIN_INSYNC_REPLICAS, defaults to 2
  • KAFKA_HEAP_OPTS, defaults to -Xms256m -Xmx1g
  • CLUSTER_ID, defaults to 5L6g3nShT-eMCtK--X86sw
  • KAFKA_NODE_ID, defaults to 3
  • KAFKA_ADVERTISED_LISTENERS, defaults to PLAINTEXT://kafka-3:9092
  • KAFKA_CLUSTERS_0_NAME, defaults to kafka
  • KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS, defaults to kafka-1:9092,kafka-2:9092,kafka-3:9092
  • SERVER_PORT, defaults to 5000

Restart policy

The restart policy here is unless-stopped, so Docker restarts Apache Kafka after a crash or reboot, but leaves it off when you stop it on purpose. You can change this on the deploy screen. The choices are no (never restart), on-failure (only after a crash), unless-stopped (restart unless you stop it), and always (bring it back no matter what).

Networking

Portainer puts these services on one shared private network, so they can find each other by name (like kafka-1) while only the ports above are open to you.

Platform

The platform is linux, the kind of system the container is built to run on. Docker and Portainer handle this on a normal Linux server.

Portainer app templates

Zooming out, this whole page comes from a Portainer app template: a short recipe telling Portainer how to set Apache Kafka up. Add the template list to Portainer once, then deploying Apache Kafka is a click rather than a wall of config.