Integrate Kpow with the Redpanda Streaming Platform

Overview

Redpanda offers a simple, powerful, and Kafka®-compatible streaming data platform. Kpow provides a rich, developer-focused UI to manage and monitor it. Together, they form a robust stack for building and operating real-time data pipelines.

This guide will walk you through the process of setting up Kpow with a local Redpanda cluster using Docker. We will cover launching the environment, using Kpow to create and manage an Avro schema in Redpanda's built-in registry, producing schema-governed data to a topic, and finally, inspecting that data in a human-readable format.

💡 Kpow supports Confluent-compatible schema registries out-of-the-box, which is why it works seamlessly with Redpanda. Learn how to connect other compatible registries by reading our guide on How to Integrate Confluent-compatible Registries with Kpow.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Prerequisites

This tutorial uses the Community Edition of Kpow, where the default user has all the necessary permissions for the tasks we will perform.

For users of Kpow Enterprise Edition with user authorization enabled, performing these actions would require the logged-in user to have the SCHEMA_CREATE permission for Role-Based Access Control (RBAC) or have ALLOW_SCHEMA_CREATE=true set for Simple Access Control. You can learn more in the Kpow User Authorization documentation.

Launch Redpanda and Kpow

We begin by creating a dedicated Docker network named factorhouse, which establishes a private communication channel for the Redpanda and Kpow containers. We then launch a single Redpanda broker and Kpow instance in dedicated containers.

Here's an overview of the containers:

• Redpanda (redpanda)
  • Image: redpandadata/redpanda:latest
  • Host Ports:
    • 19092: Exposes the Kafka API to the host machine.
    • 18081: Exposes the Schema Registry API to the host machine.
  • Configuration:
    • Mode: Runs in dev-container mode, optimized for a single-node development environment.
    • Kafka API: Listens for connections internally on port 9092 and externally on 19092. It advertises itself as redpanda:9092 within the Docker network and localhost:19092 to the host.
    • Schema Registry: The built-in schema registry is enabled, listening internally on 8081 and externally on 18081.
  • Network: Attached to the factorhouse network, making it reachable by other containers at the hostname redpanda.
• Kpow (kpow)
  • Image: factorhouse/kpow-ce:latest (Use factorhouse/kpow:latest for the enterprise edition)
  • Host Port: 3000 (for accessing the Kpow web UI from a browser at http://localhost:3000).
  • Configuration:
    • BOOTSTRAP: Configured to connect to the Redpanda cluster at redpanda:9092, using the internal Docker network for communication.
    • SCHEMA_REGISTRY_URL: Configured to connect to Redpanda's integrated schema registry at http://redpanda:8081.
    • ENVIRONMENT_NAME: Sets a descriptive name for the cluster that appears in the Kpow UI.
    • Licensing: The configuration is loaded from an environment file specified by the $KPOW_LICENSE_FILE shell variable, which is required to run the container.
  • Network: Attached to the factorhouse network, allowing it to resolve and connect to the redpanda container.

## Create a docker network to be shared
docker network create factorhouse

## Start a Redpanda cluster and schema registry
docker run -d -p 19092:19092 -p 18081:18081 --name redpanda --hostname redpanda --network factorhouse \
  redpandadata/redpanda:latest redpanda start \
    --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092 \
    --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092 \
    --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081 \
    --rpc-addr redpanda:33145 \
    --advertise-rpc-addr redpanda:33145 \
    --mode dev-container

## Start a Kpow instance
docker run -d -p 3000:3000 --name kpow --network factorhouse \
  -e ENVIRONMENT_NAME="Local Redpanda Cluster" \
  -e BOOTSTRAP="redpanda:9092" \
  -e SCHEMA_REGISTRY_NAME="Local Repanda Registry" \
  -e SCHEMA_REGISTRY_URL="http://redpanda:8081" \
  --env-file=$KPOW_LICENSE_FILE \
  factorhouse/kpow-ce:latest

Once the containers are running, navigate to http://localhost:3000 to access the Kpow UI. We can see that Kpow has automatically discovered and connected to the single Redpanda broker and its schema registry.

Schema Management

With our environment running, let's use Kpow to create a new schema subject in the Redpanda schema registry.

1. In the Schema menu, click Create subject.
   • Since we only have one registry configured, the Local Repanda Registry is selected by default.
2. Enter a subject name (e.g., demo-redpanda-value), choose AVRO as the type, and provide a schema definition. Click Create.

The Redpanda schema registry persists its schemas in an internal Kafka topic named _schemas. We can verify that our schema was created by inspecting the records of this topic directly within Kpow's Data tab.

Working with Avro Data

Finally, we'll produce and inspect an Avro record that is validated against the schema we just created.

First, create a new topic named demo-redpanda from the Kpow UI.

Now, to produce a record to the demo-redpanda topic:

1. Go to the Data menu, select the topic, and open the Produce tab.
2. Select String as the Key Serializer
3. Set the Value Serializer to AVRO.
4. Choose Local Repanda Registry as the Schema Registry.
5. Select the demo-redpanda-value subject.
6. Enter key/value data and click Produce.

To see the result, navigate back to the Data tab for the topic. In the deserializer options, select Local Repanda Registry. Kpow automatically fetches the correct schema version, deserializes the binary Avro message, and presents the data as easy-to-read JSON.

Shutdown Environment

The Docker containers and network can be removed using the commands below.

docker rm -f redpanda kpow
docker network rm factorhouse

Conclusion

In just a few minutes, we successfully launched a complete local streaming environment using Redpanda and Kpow. We demonstrated how this powerful combination simplifies the developer workflow by using Kpow's intuitive UI to manage schemas, produce Avro-encoded messages, and inspect topic data without writing any custom code. This setup proves how Redpanda's Kafka compatibility and integrated features, paired with Kpow's comprehensive toolkit, provide an efficient and accessible platform for developers building event-driven applications.