Overview

Kpow supports a wide range of Apache Kafka and Kafka API–compatible platforms, providing robust tools to manage, monitor, and secure data streaming workloads. In this guide, we'll walkthrough how to integrate Bufstream - a cloud-native, Kafka-compatible streaming solution - with Kpow, enabling seamless use of Bufstream's advanced schema management alongside Kpow's comprehensive Kafka management capabilities for an optimized streaming experience.

💡 Visit our earlier posts to learn how to set up Kpow with leading platforms such as Confluent Cloud, Amazon MSK, and Google Cloud Managed Service for Apache Kafka, as well as emerging solutions like Redpanda.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Prerequisites

This tutorial uses the Community Edition of Kpow, where the default user has all the necessary permissions for the tasks we will perform.

For users of Kpow Enterprise Edition with user authorization enabled, performing these actions would require the logged-in user to have the SCHEMA_CREATE permission for Role-Based Access Control (RBAC) or have ALLOW_SCHEMA_CREATE=true set for Simple Access Control. You can learn more in the Kpow User Authorization documentation.

Launch Bufstream and Kpow

We begin by creating a dedicated Docker network named factorhouse, which establishes a private communication channel for the Bufstream and Kpow containers. We then launch a single Bufstream broker and Kpow instance in dedicated containers.

Here's an overview of the containers:

• Bufstream (bufstream)
  • Image: bufbuild/bufstream:latest
  • Host Ports:
    • 9092: Exposes the Kafka API to the host machine.
  • Configuration:
    • Mode: Runs in inmemory mode, creating a temporary, single-node broker where all data is lost on shutdown.
    • Kafka API: Listens for connections internally on port 9092 and is also mapped to 9092 on the host. It advertises itself as bufstream:9092 within the Docker network.
  • Network: Attached to the factorhouse network, making it reachable by other containers at the hostname bufstream.
• Kpow (kpow)
  • Image: factorhouse/kpow-ce:latest
  • Host Ports:
    • 3000: Exposes the Kpow web UI to the host machine, accessible at http://localhost:3000.
  • Configuration:
    • ENVIRONMENT_NAME: UI friendly label for this cluster and resources.
    • BOOTSTRAP: Configured to connect to the Bufstream broker at bufstream:9092, using the internal Docker network for communication.
    • SCHEMA_REGISTRY_NAME: UI friendly label for a schema registry.
    • SCHEMA_REGISTRY_URL: Configured to connect to the Confluent-compatible API endpoint of Buf's public demo registry (https://demo.buf.dev/integrations/confluent/bufstream-demo). For details on connecting to a custom Confluent Schema Registry instance, see the official documentation.
    • Licensing: The configuration is loaded from an environment file specified by the $KPOW_LICENSE_FILE shell variable, which is required to run the container.
  • Network: Attached to the factorhouse network, which allows it to resolve and connect to bufstream:9092.

## Create a docker network to be shared
docker network create factorhouse

## Start a Bufstream broker
docker run -d -p 9092:9092 --name bufstream --network factorhouse \
  -e BUFSTREAM_KAFKA_HOST="0.0.0.0" \
  -e BUFSTREAM_KAFKA_PUBLIC_HOST="bufstream" \
  -e BUFSTREAM_KAFKA_PUBLIC_PORT="9092" \
  bufbuild/bufstream:latest serve --inmemory

## Start a Kpow instance
docker run -d -p 3000:3000 --name kpow --network factorhouse \
  -e ENVIRONMENT_NAME="Bufstream" \
  -e BOOTSTRAP="bufstream:9092" \
  -e SCHEMA_REGISTRY_NAME="Buf Registry" \
  -e SCHEMA_REGISTRY_URL="https://demo.buf.dev/integrations/confluent/bufstream-demo" \
  --env-file=$KPOW_LICENSE_FILE \
  factorhouse/kpow-ce:latest

Once the containers are running, navigate to http://localhost:3000 to access the Kpow UI. We will see that Kpow has automatically discovered and connected to the single Bufstream broker and Buf's public demo registry.

Schema Overview

In the Schema menu, we can see that there are four subjects in Buf's demo repository. All of them use the Protobuf format, and for this guide, we will focus on the workflow-dataset-nyc-taxis-value subject.

Clicking the menu next to a subject reveals options to view or edit the subject. By selecting View subject, we can examine the subject's schema definition.

The schema uses modern proto3 syntax and defines a detailed structure for a single New York City taxi ride via the main Ride message. It captures a comprehensive range of data, including nested fields for pickup and drop-off events, trip characteristics such as passenger count and distance, and a detailed breakdown of payment components-covering base fare, taxes, tolls, and tips.

// Copyright 2023-2025 Buf Technologies, Inc.
//
// All rights reserved.

syntax = "proto3";

package buf.bufstream.priv.dataset.nyctaxi.v1alpha1;

import "buf/confluent/v1/extensions.proto";
import "google/protobuf/timestamp.proto";

// Ride represents a single tax ride in New York City.
message Ride {
  option (buf.confluent.v1.subject) = {
    instance_name: "bufstream-demo"
    name: "workflow-dataset-nyc-taxis-value"
  };

  // ID of the vendor that provided the ride.
  optional int32 vendor_id = 1;
  // Details about the rider's pickup.
  Event pickup = 2;
  // Details about the rider's drop-off.
  Event drop_off = 3;
  // Other details of the trip.
  TripDetails details = 4;
  // Details of the fare and payment.
  Payment payment_info = 5;
  // Unclear what this is; it's in the Parquet schema though...
  optional string store_and_forward = 6;

  // Event includes properties for an event: time and location.
  message Event {
    // The time of the event.
    google.protobuf.Timestamp time = 1;
    // The location of the event.
    optional int32 location_id = 2;
  }

  // TripDetails describes properties of the trip.
  message TripDetails {
    // The number of passengers riding.
    optional int64 passenger_count = 1;
    // The number of miles travelled.
    optional double distance = 2;
    // The rate code for this trip.
    optional int32 rate_code_id = 3;
  }

  // Payment describes details about the payment and the amount charged.
  message Payment {
    // The type of instrument used for payment.
    optional int64 type = 1;
    // The total amount paid.
    optional double total_amount = 2;
    // Details of the amount charged.
    FareDetails details = 3;
  }
  // FareDetails breaks down all of the components of the amount charged.
  message FareDetails {
    // The fare, based on distance and time.
    optional double base_fare = 1;
    // Any extra fee agreed upon before the ride began.
    optional double extra = 2;
    // MTA taxes.
    optional double mta_tax = 3;
    // Optional tip.
    optional double tip = 4;
    // Fees for using toll bridges and roads.
    optional double tolls = 5;
    // Surcharge that vendor charges.
    optional double improvement_surcharge = 6;
    // Surcharge based on time and location/traffic conditions.
    optional double congestion_surcharge = 7;
    // Surcharge for trips to and from an airport.
    optional double airport_fee = 8;
  }
}

Below is an example JSON record that conforms to the schema:

{
  "vendorId": 2,
  "pickup": {
    "time": "2023-01-01T14:25:15Z",
    "locationId": 138
  },
  "dropOff": {
    "time": "2023-01-01T14:40:30Z",
    "locationId": 236
  },
  "details": {
    "passengerCount": 1,
    "distance": 3.45,
    "rateCodeId": 1
  },
  "paymentInfo": {
    "type": 1,
    "totalAmount": 24.8,
    "details": {
      "baseFare": 17.5,
      "extra": 1.25,
      "mtaTax": 0.5,
      "tip": 3.55,
      "tolls": 0,
      "improvementSurcharge": 0.3,
      "congestionSurcharge": 2.5,
      "airportFee": 0
    }
  },
  "storeAndForward": "N"
}

Working with Protobuf Data

With an understanding of the schema, let's use Kpow to interact with it by producing and consuming a Protobuf message. First, create a new topic named workflow-dataset-nyc-taxis from the Kpow UI.

Now, to produce a record to the workflow-dataset-nyc-taxis topic:

1. Go to the Data menu, select the topic, and open the Produce tab.
2. Set the Key Serializer to String.
3. Set the Value Serializer to Protobuf.
4. Choose Buf Registry as the Schema Registry.
5. Select the workflow-dataset-nyc-taxis-value subject.
6. Enter your key and a valid JSON value for the message, then click Produce.

To view the result, navigate to the Data tab and select the workflow-dataset-nyc-taxis topic. In the deserializer options, choose String as the Key deserializer and Protobuf as the Value deserializer, then select Buf Registry. Kpow will automatically fetch the correct schema version, deserialize the Protobuf message, and display the data as readable JSON.

Shutdown Environment

The Docker containers and network can be removed using the commands below.

docker rm -f bufstream kpow
docker network rm factorhouse

Conclusion

In this guide, we demonstrated how to integrate Bufstream with Kpow. We launched the services using Docker, connected Kpow to Buf's public Schema Registry, and explored a real-world Protobuf schema. Most importantly, we demonstrated how Kpow's UI simplifies the process of producing and consuming schema-validated Protobuf messages, providing immediate feedback and making data inspection effortless. This setup provides a powerful and streamlined workflow, significantly improving the development experience for any schema-driven application on Kafka.

‍

Overview

Redpanda offers a simple, powerful, and Kafka®-compatible streaming data platform. Kpow provides a rich, developer-focused UI to manage and monitor it. Together, they form a robust stack for building and operating real-time data pipelines.

This guide will walk you through the process of setting up Kpow with a local Redpanda cluster using Docker. We will cover launching the environment, using Kpow to create and manage an Avro schema in Redpanda's built-in registry, producing schema-governed data to a topic, and finally, inspecting that data in a human-readable format.

💡 Kpow supports Confluent-compatible schema registries out-of-the-box, which is why it works seamlessly with Redpanda. Learn how to connect other compatible registries by reading our guide on How to Integrate Confluent-compatible Registries with Kpow.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Prerequisites

This tutorial uses the Community Edition of Kpow, where the default user has all the necessary permissions for the tasks we will perform.

For users of Kpow Enterprise Edition with user authorization enabled, performing these actions would require the logged-in user to have the SCHEMA_CREATE permission for Role-Based Access Control (RBAC) or have ALLOW_SCHEMA_CREATE=true set for Simple Access Control. You can learn more in the Kpow User Authorization documentation.

Launch Redpanda and Kpow

We begin by creating a dedicated Docker network named factorhouse, which establishes a private communication channel for the Redpanda and Kpow containers. We then launch a single Redpanda broker and Kpow instance in dedicated containers.

Here's an overview of the containers:

• Redpanda (redpanda)
  • Image: redpandadata/redpanda:latest
  • Host Ports:
    • 19092: Exposes the Kafka API to the host machine.
    • 18081: Exposes the Schema Registry API to the host machine.
  • Configuration:
    • Mode: Runs in dev-container mode, optimized for a single-node development environment.
    • Kafka API: Listens for connections internally on port 9092 and externally on 19092. It advertises itself as redpanda:9092 within the Docker network and localhost:19092 to the host.
    • Schema Registry: The built-in schema registry is enabled, listening internally on 8081 and externally on 18081.
  • Network: Attached to the factorhouse network, making it reachable by other containers at the hostname redpanda.
• Kpow (kpow)
  • Image: factorhouse/kpow-ce:latest (Use factorhouse/kpow:latest for the enterprise edition)
  • Host Port: 3000 (for accessing the Kpow web UI from a browser at http://localhost:3000).
  • Configuration:
    • ENVIRONMENT_NAME: UI friendly label for this cluster and resources.
    • BOOTSTRAP: Configured to connect to the Redpanda cluster at redpanda:9092, using the internal Docker network for communication.
    • SCHEMA_REGISTRY_NAME: UI friendly label for a schema registry.
    • SCHEMA_REGISTRY_URL: Configured to connect to Redpanda's integrated schema registry at http://redpanda:8081.
    • Licensing: The configuration is loaded from an environment file specified by the $KPOW_LICENSE_FILE shell variable, which is required to run the container.
  • Network: Attached to the factorhouse network, allowing it to resolve and connect to the redpanda container.

## Create a docker network to be shared
docker network create factorhouse

## Start a Redpanda broker and schema registry
docker run -d -p 19092:19092 -p 18081:18081 --name redpanda --hostname redpanda --network factorhouse \
  redpandadata/redpanda:latest redpanda start \
    --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092 \
    --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092 \
    --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081 \
    --rpc-addr redpanda:33145 \
    --advertise-rpc-addr redpanda:33145 \
    --mode dev-container

## Start a Kpow instance
docker run -d -p 3000:3000 --name kpow --network factorhouse \
  -e ENVIRONMENT_NAME="Local Redpanda Cluster" \
  -e BOOTSTRAP="redpanda:9092" \
  -e SCHEMA_REGISTRY_NAME="Local Repanda Registry" \
  -e SCHEMA_REGISTRY_URL="http://redpanda:8081" \
  --env-file=$KPOW_LICENSE_FILE \
  factorhouse/kpow-ce:latest

Once the containers are running, navigate to http://localhost:3000 to access the Kpow UI. We can see that Kpow has automatically discovered and connected to the single Redpanda broker and its schema registry.

Schema Management

With our environment running, let's use Kpow to create a new schema subject in the Redpanda schema registry.

1. In the Schema menu, click Create subject.
   • Since we only have one registry configured, the Local Repanda Registry is selected by default.
2. Enter a subject name (e.g., demo-redpanda-value), choose AVRO as the type, and provide a schema definition. Click Create.

The Redpanda schema registry persists its schemas in an internal Kafka topic named _schemas. We can verify that our schema was created by inspecting the records of this topic directly within Kpow's Data tab.

Working with Avro Data

Finally, we'll produce and inspect an Avro record that is validated against the schema we just created.

First, create a new topic named demo-redpanda from the Kpow UI.

Now, to produce a record to the demo-redpanda topic:

1. Go to the Data menu, select the topic, and open the Produce tab.
2. Select String as the Key Serializer
3. Set the Value Serializer to AVRO.
4. Choose Local Repanda Registry as the Schema Registry.
5. Select the demo-redpanda-value subject.
6. Enter key/value data and click Produce.

To see the result, navigate back to the Data tab and select the demo-redpanda topic. In the deserializer options, choose String as the Key deserializer and AVRO as the Value deserializer, then select Local Repanda Registry. Kpow automatically fetches the correct schema version, deserializes the binary Avro message, and presents the data as easy-to-read JSON.

Shutdown Environment

The Docker containers and network can be removed using the commands below.

docker rm -f redpanda kpow
docker network rm factorhouse

Conclusion

In just a few minutes, we successfully launched a complete local streaming environment using Redpanda and Kpow. We demonstrated how this powerful combination simplifies the developer workflow by using Kpow's intuitive UI to manage schemas, produce Avro-encoded messages, and inspect topic data without writing any custom code. This setup proves how Redpanda's Kafka compatibility and integrated features, paired with Kpow's comprehensive toolkit, provide an efficient and accessible platform for developers building event-driven applications.

Overview

In modern data architectures built on Apache Kafka, a Schema Registry is an essential component for enforcing data contracts and supporting strong data governance. While the Confluent Schema Registry set the original standard, the ecosystem has expanded to include powerful Confluent-compatible alternatives such as Red Hat’s Apicurio Registry and Aiven’s Karapace.

Whether driven by a gradual migration, the need to support autonomous teams, or simply technology evaluation, many organizations find themselves running multiple schema registries in parallel. This inevitably leads to operational complexity and a fragmented view of their data governance.

This guide demonstrates how Kpow directly solves this challenge. We will integrate these popular schema registries into a single Kafka environment and show how to manage them all seamlessly through Kpow's single, unified interface.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Prerequisites

To create subjects in Kpow, the logged-in user must have the necessary permissions. If Role-Based Access Control (RBAC) is enabled, this requires the SCHEMA_CREATE action. For Simple Access Control, set ALLOW_SCHEMA_CREATE=true. For details, see the Kpow User Authorization docs.

Launch Kafka Environment

To accelerate the setup, we will use the Factor House Local repository, which provides a solid foundation with pre-built configurations for authentication and authorization.

First, clone the repository:

git clone https://github.com/factorhouse/factorhouse-local

Next, navigate into the project root and create a Docker Compose file named compose-kpow-multi-registries.yml. This file defines our entire stack: a 3-broker Kafka cluster, our three schema registries, and Kpow.

services:
  schema:
    image: confluentinc/cp-schema-registry:7.8.0
    container_name: schema_registry
    ports:
      - "8081:8081"
    networks:
      - factorhouse
    depends_on:
      - zookeeper
      - kafka-1
      - kafka-2
      - kafka-3
    environment:
      SCHEMA_REGISTRY_HOST_NAME: "schema"
      SCHEMA_REGISTRY_LISTENERS: http://schema:8081,http://${DOCKER_HOST_IP:-127.0.0.1}:8081
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "kafka-1:19092,kafka-2:19093,kafka-3:19094"
      SCHEMA_REGISTRY_AUTHENTICATION_METHOD: BASIC
      SCHEMA_REGISTRY_AUTHENTICATION_REALM: schema
      SCHEMA_REGISTRY_AUTHENTICATION_ROLES: schema-admin
      SCHEMA_REGISTRY_OPTS: -Djava.security.auth.login.config=/etc/schema/schema_jaas.conf
    volumes:
      - ./resources/kpow/schema:/etc/schema

  apicurio:
    image: apicurio/apicurio-registry:3.0.9
    container_name: apicurio
    ports:
      - "8080:8080"
    networks:
      - factorhouse
    environment:
      APICURIO_KAFKASQL_BOOTSTRAP_SERVERS: kafka-1:19092,kafka-2:19093,kafka-3:19094
      APICURIO_STORAGE_KIND: kafkasql
      APICURIO_AUTH_ENABLED: "true"
      APICURIO_AUTH_ROLE_BASED_AUTHORIZATION: "true"
      APICURIO_AUTH_STATIC_USERS: "admin=admin" # Format: user1=pass1,user2=pass2
      APICURIO_AUTH_STATIC_ROLES: "admin:sr-admin" # Format: user:role,user2:role2

  karapace:
    image: ghcr.io/aiven-open/karapace:develop
    container_name: karapace
    entrypoint:
      - python3
      - -m
      - karapace
    ports:
      - "8082:8081"
    networks:
      - factorhouse
    depends_on:
      - zookeeper
      - kafka-1
      - kafka-2
      - kafka-3
    environment:
      KARAPACE_KARAPACE_REGISTRY: true
      KARAPACE_ADVERTISED_HOSTNAME: karapace
      KARAPACE_ADVERTISED_PROTOCOL: http
      KARAPACE_BOOTSTRAP_URI: kafka-1:19092,kafka-2:19093,kafka-3:19094
      KARAPACE_PORT: 8081
      KARAPACE_HOST: 0.0.0.0
      KARAPACE_CLIENT_ID: karapace-0
      KARAPACE_GROUP_ID: karapace
      KARAPACE_MASTER_ELECTION_STRATEGY: highest
      KARAPACE_MASTER_ELIGIBILITY: true
      KARAPACE_TOPIC_NAME: _karapace
      KARAPACE_COMPATIBILITY: "BACKWARD"

  kpow:
    image: factorhouse/kpow:latest
    container_name: kpow-ee
    pull_policy: always
    restart: always
    ports:
      - "3000:3000"
    networks:
      - factorhouse
    depends_on:
      - schema
      - apicurio
      - karapace
    env_file:
      - resources/kpow/config/multi-registry.env
      - ${KPOW_TRIAL_LICENSE:-resources/kpow/config/trial-license.env}
    mem_limit: 2G
    volumes:
      - ./resources/kpow/jaas:/etc/kpow/jaas
      - ./resources/kpow/rbac:/etc/kpow/rbac

  zookeeper:
    image: confluentinc/cp-zookeeper:7.8.0
    container_name: zookeeper
    ports:
      - "2181:2181"
    networks:
      - factorhouse
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka-1:
    image: confluentinc/cp-kafka:7.8.0
    container_name: kafka-1
    ports:
      - "9092:9092"
    networks:
      - factorhouse
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka-1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
      KAFKA_LOG4J_ROOT_LOGLEVEL: INFO
      KAFKA_NUM_PARTITIONS: "3"
      KAFKA_DEFAULT_REPLICATION_FACTOR: "3"
    depends_on:
      - zookeeper

  kafka-2:
    image: confluentinc/cp-kafka:7.8.0
    container_name: kafka-2
    ports:
      - "9093:9093"
    networks:
      - factorhouse
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka-2:19093,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
      KAFKA_LOG4J_ROOT_LOGLEVEL: INFO
      KAFKA_NUM_PARTITIONS: "3"
      KAFKA_DEFAULT_REPLICATION_FACTOR: "3"
    depends_on:
      - zookeeper

  kafka-3:
    image: confluentinc/cp-kafka:7.8.0
    container_name: kafka-3
    ports:
      - "9094:9094"
    networks:
      - factorhouse
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka-3:19094,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
      KAFKA_LOG4J_ROOT_LOGLEVEL: INFO
      KAFKA_NUM_PARTITIONS: "3"
      KAFKA_DEFAULT_REPLICATION_FACTOR: "3"
    depends_on:
      - zookeeper

networks:
  factorhouse:
    name: factorhouse

Here's an overview of the three schema registries and Kpow:

• Confluent Schema Registry (schema)
  • Image: confluentinc/cp-schema-registry:7.8.0
  • Storage: It uses the connected Kafka cluster for durable storage, persisting schemas in an internal topic (named _schemas by default).
  • Security: This service is secured using BASIC HTTP authentication. Access requires a valid username and password, which are defined in the schema_jaas.conf file mounted via the volumes directive.
• Apicurio Registry (apicurio)
  • Image: apicurio/apicurio-registry:3.0.9
  • Storage: It's configured to use the kafkasql storage backend, and schemas are stored in a Kafka topic (kafkasql-journal).
  • Security: Authentication is enabled and managed directly through environment variables. This setup creates a static user (admin with password admin) and grants it administrative privileges.
  • API Endpoint: To align with the Kafka environment, we'll use /apis/ccompat/v7 as the Confluent Compatibility API endpoint.
• Karapace Registry (karapace)
  • Image: ghcr.io/aiven-open/karapace:develop
  • Storage: Like the others, it uses a Kafka topic (_karapace) to store its schema data.
  • Security: For simplicity, authentication is not configured, leaving the API openly accessible on the network. However, the logged-in Kpow user must still have the appropriate permissions to manage schema resources—highlighting one of the key access control benefits Kpow offers in enterprise environments.
• Kpow (kpow)
  • Image: factorhouse/kpow:latest
  • Host Port: 3000
  • Configuration:
    • env_file: Its primary configuration is loaded from external files. The multi-registry.env file is crucial, as it contains the connection details for the Kafka cluster and all three schema registries.
    • Licensing: The configuration also loads a license file. It uses a local trial-license.env by default, but this can be overridden by setting the KPOW_TRIAL_LICENSE environment variable to a different file path.
  • Volumes:
    • ./resources/kpow/jaas: This mounts authentication configuration (JAAS file) into Kpow.
    • ./resources/kpow/rbac: This mounts Role-Based Access Control (RBAC) file.

We also need to create the Kpow configuration file (resources/kpow/config/multi-registry.env). The environment variables in this file configures Kpow's own user security, the connection to the Kafka cluster, and the integration with all three schema registries.

## AauthN + AuthZ
JAVA_TOOL_OPTIONS="-Djava.awt.headless=true -Djava.security.auth.login.config=/etc/kpow/jaas/hash-jaas.conf"
AUTH_PROVIDER_TYPE=jetty
RBAC_CONFIGURATION_FILE=/etc/kpow/rbac/hash-rbac.yml

## Kafka environments
ENVIRONMENT_NAME=Multi-registry Integration
BOOTSTRAP=kafka-1:19092,kafka-2:19093,kafka-3:19094

SCHEMA_REGISTRY_RESOURCE_IDS=CONFLUENT,APICURIO,KARAPACE

CONFLUENT_SCHEMA_REGISTRY_URL=http://schema:8081
CONFLUENT_SCHEMA_REGISTRY_AUTH=USER_INFO
CONFLUENT_SCHEMA_REGISTRY_USER=admin
CONFLUENT_SCHEMA_REGISTRY_PASSWORD=admin

APICURIO_SCHEMA_REGISTRY_URL=http://apicurio:8080/apis/ccompat/v7
APICURIO_SCHEMA_REGISTRY_AUTH=USER_INFO
APICURIO_SCHEMA_REGISTRY_USER=admin
APICURIO_SCHEMA_REGISTRY_PASSWORD=admin

KARAPACE_SCHEMA_REGISTRY_URL=http://karapace:8081

We can start all services in the background using the Docker Compose file:

docker compose -f ./compose-kpow-multi-registries.yml up -d

Once the containers are running, navigate to http://localhost:3000 to access the Kpow UI (admin as both username and password). In the left-hand navigation menu under Schema, you will see all three registries - CONFLUENT, APICURIO, and KARAPACE.

Unified Schema Management

Now, we will create a schema subject in each registry directly from Kpow.

1. In the Schema menu, click Create subject.
2. Select CONFLUENT from the Registry dropdown.
3. Enter a subject name (e.g., demo-confluent-value), choose AVRO as the type, and provide a schema definition. Click Create.

Subject: demo-confluent-value

Following the same pattern, create subjects for the other two registries:

• Apicurio: Select APICURIO and create the demo-apicurio-value subject.
• Karapace: Select KARAPACE and create the demo-karapace-value subject.

Subject: demo-apicurio-value

Subject: demo-karapace-value

Each registry persists its schemas in an internal Kafka topic. We can verify this in Kpow's Data tab by inspecting the contents of their respective storage topics:

• CONFLUENT: _schemas
• APICURIO: kafkasql-journal (the default topic for its kafkasql storage engine)
• KARAPACE: _karapace

Produce and Inspect Records Across All Registries

Finally, we'll produce and inspect Avro records, leveraging the schemas from each registry.

First, create the topics demo-confluent, demo-apicurio, and demo-karapace in Kpow.

To produce a record for the demo-confluent topic:

1. Go to the Data menu, select the topic, and open the Produce tab.
2. Select String as the Key Serializer
3. Set the Value Serializer to AVRO.
4. Choose CONFLUENT as the Schema Registry.
5. Select the demo-confluent-value subject.
6. Enter key/value data and click Produce.

Topic: demo-confluent

Repeat this for the other topics, making sure to select the corresponding registry and subject for demo-apicurio and demo-karapace.

Topic: demo-apicurio

Topic: demo-karapace

To inspect the records, navigate back to the Data tab for each topic. Select the correct Schema Registry in the deserializer options. Kpow will automatically fetch the correct schema, deserialize the binary Avro data, and present it as human-readable JSON.

Topic: demo-confluent

Topic: demo-apicurio

Topic: demo-karapace

Conclusion

This guide has demonstrated that managing a heterogeneous, multi-registry Kafka environment does not have to be a fragmented or complex task. By leveraging the Confluent-compatible APIs of Apicurio and Karapace, we can successfully integrate them alongside the standard Confluent Schema Registry.

With Kpow providing a single pane of glass, we gain centralized control and visibility over all our schema resources. This unified approach simplifies critical operations like schema management, data production, and inspection, empowering teams to use the best tool for their needs without sacrificing governance or operational efficiency.