qmp-testing-suite/kafka-stack

Qualtrics Messaging Platform Docker Quick Start Locally

Usage

Clone this repository:

git clone ssh://git@gitlab-app.eng.qops.net:10022/core/qmp/qmp-docker.git

Move into directory and set required environment variables:

cd qmp-docker/
export KAFKA_CREATE_TOPICS="my-topic-1:20:1;my-topic-2:20:1"
export SCHEMAS_LOCATION="<absolute path to .avsc schema files>"

KAFKA_CREATE_TOPICS is a list of topics separated by a ';'. Each topic consists of three fields separated by ':', topic name, number of partitions, number of replicas. Since this is for running locally, we are only starting one broker therefore the number of replicas is required to be 1. By default, QMP creates topics with 20 partitions which directly maps to the maximum number of consumers that can consume from this topic concurrently.

SCHEMAS_LOCATION is the path to the directory containing your avro schemas. The schema registry will automatically load in the schemas at this location on start up.

NOTE: The qmp.envelope.schema_id.avsc schema file is required for qmp. If you do not have this file, copy it from here.

To run specific QMP image versions, export the following:

export ZOOKEEPER_VERSION="<version of core zookeeper image>"
export KAFKA_VERSION="<version of core kafka image>"
export SCHEMA_REGISTRY_VERSION="<version of core schema registry image>"
export SCHEMA_REGISTRY_UI_VERSION="<version of core schema registry ui image>"
export KAFKA_MANAGER_VERSION="<version of core kafka manager image>"

Start the QMP images in order:

docker-compose up

This will start the following images on the following ports:

• Zookeeper: 2181
• Kafka: 9092
• Schema Registry: 8081
• Schema Registry UI: 8000
• Kafka Manager: 9000

To add the cluster in kafka manager, navigate to localhost:9000. If you are prompted for a username and password, simply input admin for the username and password for the password. Click on the Cluster dropdown and select Add Cluster. Input local for the cluster name, and core-zookeeper:2181/qmp/local for cluster zookeeper hosts. These are the only required fields, scroll to the bottom and hit Save.

The Schema Registry UI can be accessed at localhost:8000.

To tear down QMP locally, control+c the process, wait for graceful teardown, then execute:

docker-compose down

It's recommended to add the following entry to your /etc/hosts file:

127.0.0.1	core-kafka

This way you can access the kafka broker from the container name core-kafka.