Set Up Kpow with Amazon Managed Streaming for Apache Kafka

Thursday, May 15th, 2025 by Jaehyeon Kim

Overview

Apache Kafka is a cornerstone of modern real-time data pipelines, facilitating high-throughput, low-latency event streaming.

Managing Kafka infrastructure, particularly at scale, presents significant operational challenges. To address this, Amazon Managed Streaming for Apache Kafka (MSK) provides a fully managed service - simplifying the provisioning, configuration, patching, and scaling of Kafka clusters. While MSK handles the infrastructure heavy lifting, effective management and control are still crucial for maintaining cluster health, performance, and reliability.

This article provides a comprehensive walkthrough to setting up an Amazon MSK cluster and integrating it with Kpow for Apache Kafka, a powerful tool for managing and monitoring Kafka environments. It walks through provisioning AWS infrastructure, configuring authentication with the OAUTHBEARER mechanism using AWS IAM, setting up a client EC2 instance within the same VPC, deploying Kpow via Docker Compose, and using Kpow's UI to monitor/manage brokers, topics, and messages.

Whether you manage production Kafka workloads or are evaluating management solutions, this guide provides practical steps for effectively managing and monitoring Kafka clusters on AWS.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions forw Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

image

Set up an EC2 instance

This demo is set up on an Ubuntu-based EC2 instance. Because the MSK cluster will be configured to accept traffic only from within the same VPC, this instance will serve as our primary access point for interacting with the cluster. To ensure connectivity and control, the instance must:

  • Be launched in the same VPC as the MSK cluster
  • Allow inbound HTTP (port 80) and SSH (port 22) traffic via its security group

We use the AWS Command Line Interface (CLI) to provision and manage AWS resources throughout the demo. If the CLI is not already installed, follow the official AWS CLI user guide for setup and configuration instructions.

As Kpow is designed to manage/monitor Kafka clusters and associated resources, we can give administrative privileges to it. For more fine-grained access control within a Kafka cluster, we can rely on Apache Kafka ACLs, and the Enterprise Edition of Kpow provides robust support for it - see Kpow's ACL management documentation for more details.

Below shows example policies that can be attached.

Option 1: Admin Access to ALL MSK Clusters in the Region/Account

This policy allows listing/describing all clusters and performing any data-plane action (kafka-cluster:*) on any cluster within the specified region and account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "kafka-cluster:*",
            "Resource": "arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka:ListClusters",
                "kafka:DescribeCluster",
                "kafka:GetBootstrapBrokers"
            ],
            "Resource": "*"
        }
    ]
}

Option 2: Admin Access to a Specific LIST of MSK Clusters

This policy allows listing/describing all clusters but restricts the powerful kafka-cluster:* data-plane actions to only the specific clusters listed in the Resource array.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "kafka-cluster:*",
            "Resource": [
                "arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/<CLUSTER-NAME-1>/<GUID-1>",
                "arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/<CLUSTER-NAME-2>/<GUID-2>"
                // Add more cluster ARNs here as needed following the same pattern
                // "arn:aws:kafka:<REGION>:<ACCOUNT-ID>:cluster/<CLUSTER-NAME-3>/<GUID-3>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka:ListClusters",
                "kafka:DescribeCluster",
                "kafka:GetBootstrapBrokers"
            ],
            "Resource": "*"
        }
    ]
}

Create a MSK Cluster

While Kpow supports both provisioned and serverless MSK clusters, in this post we'll use an MSK Serverless cluster.

First, create a security group for the Kafka cluster. This security group allows traffic on Kafka port 9098 from:

  • Itself (for intra-cluster communication), and
  • The EC2 instance's security group (for Kpow access within the same VPC).
VPC_ID=<vpc-ic>
SUBNET_ID1=<subnet-id-1>
SUBNET_ID2=<subnet-id-2>
SUBNET_ID3=<subnet-id-3>
EC2_SG_ID=<ec2-security-group-id>
CLUSTER_NAME=<cluster-name>
REGION=<aws-region>

SG_ID=$(aws ec2 create-security-group \
  --group-name ${CLUSTER_NAME}-sg \
  --description "Security group for $CLUSTER_NAME" \
  --vpc-id "$VPC_ID" \
  --region "$REGION" \
  --query 'GroupId' --output text)

## Allow traffic from itself
aws ec2 authorize-security-group-ingress \
  --group-id "$SG_ID" \
  --protocol tcp \
  --port 9098 \
  --source-group $SG_ID \
  --region "$REGION"

## Allow traffic from EC2 instance
aws ec2 authorize-security-group-ingress \
  --group-id "$SG_ID" \
  --protocol tcp \
  --port 9098 \
  --source-group $EC2_SG_ID \
  --region "$REGION"

Next, create an MSK serverless cluster. We use the aws kafka create-cluster-v2 command with a JSON configuration that specifies:

  • VPC subnet and security group,
  • SASL/IAM-based client authentication.
read -r -d '' SERVERLESS_JSON <<EOF
{
    "VpcConfigs": [
        {
          "SubnetIds": ["$SUBNET_ID1", "$SUBNET_ID2", "$SUBNET_ID3"],
          "SecurityGroupIds": ["$SG_ID"]
        }
    ],
    "ClientAuthentication": {
      "Sasl": {
        "Iam": { "Enabled": true }
        }
    }
}
EOF

aws kafka create-cluster-v2 \
    --cluster-name "$CLUSTER_NAME" \
    --serverless "$SERVERLESS_JSON" \
    --region "$REGION"

Launch a Kpow Instance

Kpow's Docker Compose environment (kpow-local), provides resources for setting up Kpow instances. In this post, we use some of the existing configurations to simplify the deployment process. Before proceeding, it's essential to have Docker Engine and the Docker Compose plugin installed; if these are not yet set up, refer to the official installation and post-installation guides.

To get started, clone the GitHub repository and create a docker-compose.yml file in the project's root directory with the following content. This Compose file defines a service that runs a Kpow instance, exposing the UI on port 80 of the host by mapping it to port 3000 inside the container. As a result, we can access the Kpow UI in the browser at http://<ec2-public-ip> without specifying a port. Kpow's authentication and authorization are configured using mounted JAAS and RBAC files, while other key settings, such as Kafka connection details, are managed through an .env file referenced in the env_file section.

services:
  kpow:
    image: factorhouse/kpow:latest
    pull_policy: always
    restart: always
    ports:
      - "80:3000"
    env_file:
      - resources/kpow/aws-trial.env
    mem_limit: 2G
    volumes: ## Enterprise edition only
      - ./resources/jaas:/etc/kpow/jaas
      - ./resources/rbac:/etc/kpow/rbac

Next, create a configuration file at resources/kpow/aws-trial.env to define Kpow’s main settings for connecting to the MSK cluster. This file includes authentication and authorization settings, Kafka connection details, licensing, and AWS credentials.

Kpow Enterprise Edition supports authentication and authorization out of the box. In this demo, we use the pre-configured Jetty authentication provider and a role-based access control (RBAC) file to manage user permissions.

The second section defines how Kpow connects to the MSK cluster:

  • ENVIRONMENT_NAME: A human-readable name for the Kafka environment shown in the Kpow UI.
  • BOOTSTRAP: The Kafka bootstrap server URL for the MSK Serverless cluster (e.g., boot-xxxxxxxx.c2.kafka-serverless.<region>.amazonaws.com:9098).
  • KAFKA_VARIANT: Set this to MSK_SERVERLESS to ensure Kpow creates its internal topics with the constrained topic configuration properties and service limitations specific to MSK Serverless.

Secure communication with the cluster is established using SASL over SSL:

  • SECURITY_PROTOCOL: Set to SASL_SSL to enable encrypted client-server communication.
  • SASL_MECHANISM: Set to AWS_MSK_IAM to use AWS IAM for Kafka client authentication.
  • SASL_JAAS_CONFIG: Specifies the use of the IAMLoginModule provided by Amazon for secure authentication.
  • SASL_CLIENT_CALLBACK_HANDLER_CLASS: Points to IAMClientCallbackHandler, which automates the process of retrieving and refreshing temporary credentials via IAM.

Finally, the configuration includes Kpow license details and AWS credentials, which are required to activate and run Kpow, as well as to access the Kafka cluster.

## AuthN & AuthZ - Enterprise Edition Only
JAVA_TOOL_OPTIONS=-Djava.security.auth.login.config=/etc/kpow/jaas/hash-jaas.conf
AUTH_PROVIDER_TYPE=jetty
RBAC_CONFIGURATION_FILE=/etc/kpow/rbac/hash-rbac.yml

## Managed Service for Apache Kafka Cluster Configuration
ENVIRONMENT_NAME=MSK Serverless
BOOTSTRAP=boot-<cluster-identifier>.c2.kafka-serverless.<aws-region>.amazonaws.com:9098
KAFKA_VARIANT=MSK_SERVERLESS
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=AWS_MSK_IAM
SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;
SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler

## Your License Details
LICENSE_ID=<license-id>
LICENSE_CODE=<license-code>
LICENSEE=<licensee>
LICENSE_EXPIRY=<license-expiry>
LICENSE_SIGNATURE=<license-signature>

## AWS Credentials
AWS_ACCESS_KEY_ID=<aws-access-key>
AWS_SECRET_ACCESS_KEY=<aws-secret-access-key>
AWS_SESSION_TOKEN=<aws-session-token> # Optional
AWS_REGION=<aws-region>

To deploy the Kpow instance, run the following command using Docker Compose:

docker compose up -d

Monitor and Manage Resources

After logging in with one of the default credentials (e.g., admin for both username and password), we can follow the example below to connect a managed Kafka cluster to Kpow. It walks through monitoring brokers, creating a topic and producing a message, then reading that message through the Kpow UI.

Kpow Demo

Conclusion

In summary, this guide walked through setting up a fully managed Kafka environment on AWS using Amazon MSK Serverless and Kpow. By leveraging MSK Serverless for Kafka infrastructure and Kpow for observability and control, we can streamline operations while gaining deep insight into our data pipelines. The process included provisioning AWS resources, configuring a secure cluster with IAM-based authentication, and deploying Kpow via Docker Compose with environment-specific and security-conscious settings.

Once connected, Kpow provides an intuitive interface to monitor brokers, manage topics, produce and consume messages, and track consumer lag in real time. Beyond the basics, it offers advanced features like schema inspection, Kafka Connect monitoring, RBAC enforcement, and audit visibility - helping teams shift from reactive troubleshooting to proactive, insight-driven operations. Together, Amazon Managed Streaming for Apache Kafka (MSK) and Kpow form a robust foundation for building and managing high-performance, secure, real-time streaming applications on AWS.

Kafka
Kpow
How-To
Amazon Web Service (AWS)
Amazon Managed Streaming for Apache Kafka (MSK)