Developer
Knowledge Center

Empowering engineers with everything they need to build, monitor, and scale real-time data pipelines with confidence.

Subscribe to Newsletter

Join our Slack

Article

xx min read

Case Study

xx min read

How To

xx min read

Deploy Kpow on EKS via AWS Marketplace using Helm

Streamline your Kpow deployment on Amazon EKS with our guide, fully integrated with the AWS Marketplace. We use eksctl to automate IAM Roles for Service Accounts (IRSA), providing a secure integration for Kpow's licensing and metering. This allows your instance to handle license validation via AWS License Manager and report usage for hourly subscriptions, enabling a production-ready deployment with minimal configuration.

Overview

This guide provides a comprehensive walkthrough for deploying Kpow, a powerful toolkit for Apache Kafka, onto an Amazon EKS (Elastic Kubernetes Service) cluster. We will cover the entire process from start to finish, including provisioning the necessary AWS infrastructure, deploying a Kafka cluster using the Strimzi operator, and finally, installing Kpow using a subscription from the AWS Marketplace.

The guide demonstrates how to set up both Kpow Annual and Kpow Hourly products, highlighting the specific integration points with AWS services like IAM for service accounts, ECR for container images, and the AWS License Manager for the annual subscription. By the end of this tutorial, you will have a fully functional environment running Kpow on EKS, ready to monitor and manage your Kafka cluster.

The source code and configuration files used in this guide can be found in the features/eks-deployment folder of this GitHub repository.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Prerequisites

To follow along the guide, you need:

CLI Tools:
- Recent versions of the following command-line interface (CLI) tools must be installed:
  - eksctl: For creating and managing Amazon EKS clusters.
  - kubectl: For interacting with Kubernetes resources.
  - Helm: The package manager for Kubernetes.
AWS Infrastructure:
- VPC: A Virtual Private Cloud (VPC) that has both public and private subnets is required.
- IAM Permissions: A user with the necessary IAM permissions to create an EKS cluster with a service account.
Kpow Subscription:
- A subscription to a Kpow product through the AWS Marketplace is required. After subscribing, you will receive access to the necessary components and deployment instructions.
- The specifics of accessing the container images and Helm chart depend on the chosen Kpow product:
  - Kpow Annual product:
    - Subscribing to the annual product provides access to the ECR (Elastic Container Registry) image and the corresponding Helm chart.
  - Kpow Hourly product:
    - For the hourly product, access to the ECR image will be provided and deployment utilizes the public Factor House Helm repository for installation.

Deploy an EKS cluster

We will use eksctl to provision an Amazon EKS cluster. The configuration for the cluster is defined in the manifests/eks/cluster.eksctl.yaml file within the repository.

Before creating the cluster, you must open this file and replace the placeholder values for <VPC-ID>, <PRIVATE-SUBNET-ID-* >, and <PUBLIC-SUBNET-ID-* > with your actual VPC and subnet IDs.

⚠️ The provided configuration assumes the EKS cluster will be deployed in the us-east-1 region. If you intend to use a different region, you must update the metadata.region field and ensure the availability zone keys under vpc.subnets (e.g., us-east-1a, us-east-1b) match the availability zones of the subnets in your chosen region.

Here is the content of the cluster.eksctl.yaml file:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: fh-eks-cluster
  region: us-east-1

vpc:
  id: "<VPC-ID>"
  subnets:
    private:
      us-east-1a:
        id: "<PRIVATE-SUBNET-ID-1>"
      us-east-1b:
        id: "<PRIVATE-SUBNET-ID-2>"
    public:
      us-east-1a:
        id: "<PUBLIC-SUBNET-ID-1>"
      us-east-1b:
        id: "<PUBLIC-SUBNET-ID-2>"

iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: kpow-annual
        namespace: factorhouse
      attachPolicyARNs:
        - "arn:aws:iam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy"
    - metadata:
        name: kpow-hourly
        namespace: factorhouse
      attachPolicyARNs:
        - "arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage"

nodeGroups:
  - name: ng-dev
    instanceType: t3.medium
    desiredCapacity: 4
    minSize: 2
    maxSize: 6
    privateNetworking: true

This configuration sets up the following:

Cluster Metadata: A cluster named fh-eks-cluster in the us-east-1 region.
VPC: Specifies an existing VPC and its public/private subnets where the cluster resources will be deployed.
IAM with OIDC: Enables the IAM OIDC provider, which allows Kubernetes service accounts to be associated with IAM roles. This is crucial for granting AWS permissions to your pods.
Service Accounts:
- kpow-annual: Creates a service account for the Kpow Annual product. It attaches the AWSLicenseManagerConsumptionPolicy, allowing Kpow to validate its license with the AWS License Manager service.
- kpow-hourly: Creates a service account for the Kpow Hourly product. It attaches the AWSMarketplaceMeteringRegisterUsage policy, which is required for reporting usage metrics to the AWS Marketplace.
Node Group: Defines a managed node group named ng-dev with t3.medium instances. The worker nodes will be placed in the private subnets (privateNetworking: true).

Once you have updated the YAML file with your networking details, run the following command to create the cluster. This process can take 15-20 minutes to complete.

eksctl create cluster -f cluster.eksctl.yaml

Once the cluster is created, eksctl automatically updates your kubeconfig file (usually located at ~/.kube/config) with the new cluster's connection details. This allows you to start interacting with your cluster immediately using kubectl.

kubectl get nodes
# NAME                              STATUS   ROLES    AGE     VERSION
# ip-192-168-...-21.ec2.internal   Ready    <none>   2m15s    v1.32.9-eks-113cf36
# ...

Launch a Kafka cluster

With the EKS cluster running, we will now launch an Apache Kafka cluster into it. We will use the Strimzi Kafka operator, which simplifies the process of running Kafka on Kubernetes.

Install the Strimzi operator

First, create a dedicated namespace for the Kafka cluster.

kubectl create namespace kafka

Next, download the Strimzi operator installation YAML. The repository already contains the file manifests/kafka/strimzi-cluster-operator-0.45.1.yaml, but the following commands show how it was downloaded and modified for this guide.

## Define the Strimzi version and download URL
STRIMZI_VERSION="0.45.1"
DOWNLOAD_URL=https://github.com/strimzi/strimzi-kafka-operator/releases/download/$STRIMZI_VERSION/strimzi-cluster-operator-$STRIMZI_VERSION.yaml

## Download the operator manifest
curl -L -o manifests/kafka/strimzi-cluster-operator-$STRIMZI_VERSION.yaml ${DOWNLOAD_URL}

## Modify the manifest to install the operator in the 'kafka' namespace
sed -i 's/namespace: .*/namespace: kafka/' manifests/kafka/strimzi-cluster-operator-$STRIMZI_VERSION.yaml

Now, apply the manifest to install the Strimzi operator in your EKS cluster.

kubectl apply -f manifests/kafka/strimzi-cluster-operator-0.45.1.yaml -n kafka

Deploy a Kafka cluster

The configuration for our Kafka cluster is defined in manifests/kafka/kafka-cluster.yaml. It describes a simple, single-node cluster suitable for development, using ephemeral storage, meaning data will be lost if the pods restart.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: fh-k8s-cluster
spec:
  kafka:
    version: 3.9.1
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
# ... (content truncated for brevity)

Deploy the Kafka cluster with the following command:

kubectl create -f manifests/kafka/kafka-cluster.yaml -n kafka

Verify the deployment

After a few minutes, all the necessary pods and services for Kafka will be running. You can verify this by listing all resources in the kafka namespace.

kubectl get all -n kafka -o name

The output should look similar to this, showing the pods for Strimzi, Kafka, Zookeeper, and the associated services. The most important service for connecting applications is the Kafka bootstrap service.

# pod/fh-k8s-cluster-entity-operator-...
# pod/fh-k8s-cluster-kafka-0
# ...
# service/fh-k8s-cluster-kafka-bootstrap <-- Kafka bootstrap service
# ...

Deploy Kpow

Now that the EKS and Kafka clusters are running, we can deploy Kpow. This guide covers the deployment of both Kpow Annual and Kpow Hourly products. Both deployments will use a common set of configurations for connecting to Kafka and setting up authentication/authorization.

First, ensure you have a namespace for Kpow. The eksctl command we ran earlier already created the service accounts in the factorhouse namespace, so we will use that. If you hadn't created it, you would run kubectl create namespace factorhouse.

Create ConfigMaps

We will use two Kubernetes ConfigMaps to manage Kpow's configuration. This approach separates the core configuration from the Helm deployment values.

kpow-config-files: This ConfigMap holds file-based configurations, including RBAC policies, JAAS configuration, and user properties for authentication.
kpow-config: This ConfigMap provides environment variables to the Kpow container, such as the Kafka bootstrap address and settings to enable our authentication provider.

The contents of these files can be found in the repository at manifests/kpow/config-files.yaml and manifests/kpow/config.yaml.

manifests/kpow/config-files.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kpow-config-files
  namespace: factorhouse
data:
  hash-rbac.yml: |
    # RBAC policies defining user roles and permissions
    admin_roles:
      - "kafka-admins"
    # ... (content truncated for brevity)
  hash-jaas.conf: |
    # JAAS login module configuration
    kpow {
      org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
      file="/etc/kpow/jaas/hash-realm.properties";
    };
    # ... (content truncated for brevity)
  hash-realm.properties: |
    # User credentials (username: password, roles)
    # admin/admin
    admin: CRYPT:adpexzg3FUZAk,server-administrators,content-administrators,kafka-admins
    # user/password
    user: password,kafka-users
    # ... (content truncated for brevity)

manifests/kpow/config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kpow-config
  namespace: factorhouse
data:
  # Environment Configuration
  BOOTSTRAP: "fh-k8s-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
  REPLICATION_FACTOR: "1"
  # AuthN + AuthZ
  JAVA_TOOL_OPTIONS: "-Djava.awt.headless=true -Djava.security.auth.login.config=/etc/kpow/jaas/hash-jaas.conf"
  AUTH_PROVIDER_TYPE: "jetty"
  RBAC_CONFIGURATION_FILE: "/etc/kpow/rbac/hash-rbac.yml"

Apply these manifests to create the ConfigMaps in the factorhouse namespace.

kubectl apply -f manifests/kpow/config-files.yaml \
  -f manifests/kpow/config.yaml -n factorhouse

You can verify their creation by running:

kubectl get configmap -n factorhouse
# NAME                DATA   AGE
# kpow-config         5      ...
# kpow-config-files   3      ...

Deploy Kpow Annual

Download the Helm chart

The Helm chart for Kpow Annual is in a private Amazon ECR repository. First, authenticate your Helm client.

# Enable Helm's experimental support for OCI registries
export HELM_EXPERIMENTAL_OCI=1

# Log in to the AWS Marketplace ECR registry
aws ecr get-login-password \
    --region us-east-1 | helm registry login \
    --username AWS \
    --password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com

Next, pull and extract the chart.

# Create a directory, pull the chart, and extract it
mkdir -p awsmp-chart && cd awsmp-chart
# Pull the latest version of the Helm chart from ECR (add --version <x.x.x> to specify a version)
helm pull oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/factor-house/kpow-aws-annual
tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete
cd ..

Launch Kpow Annual

Now, install Kpow using Helm. We will reference the service account kpow-annual that was created during the EKS cluster setup, which has the required IAM policy for license management.

helm install kpow-annual ./awsmp-chart/kpow-aws-annual/ \
    -n factorhouse \
    --set serviceAccount.create=false \
    --set serviceAccount.name=kpow-annual \
    --values ./values/eks-annual.yaml

The Helm values for this deployment are in values/eks-annual.yaml. It mounts the configuration files from our ConfigMaps and sets resource limits.

# values/eks-annual.yaml
env:
  ENVIRONMENT_NAME: "Kafka from Kpow Annual"
envFromConfigMap: "kpow-config"
volumeMounts:
  - name: kpow-config-volumes
    mountPath: /etc/kpow/rbac/hash-rbac.yml
    subPath: hash-rbac.yml
  - name: kpow-config-volumes
    mountPath: /etc/kpow/jaas/hash-jaas.conf
    subPath: hash-jaas.conf
  - name: kpow-config-volumes
    mountPath: /etc/kpow/jaas/hash-realm.properties
    subPath: hash-realm.properties
volumes:
  - name: kpow-config-volumes
    configMap:
      name: "kpow-config-files"
resources:
  limits:
    cpu: 1
    memory: 0.5Gi
  requests:
    cpu: 1
    memory: 0.5Gi

Note: The CPU and memory values are intentionally set low for this guide. For production environments, check the official documentation for recommended capacity.

Verify and access Kpow Annual

Check that the Kpow pod is running successfully.

kubectl get all -l app.kubernetes.io/instance=kpow-annual -n factorhouse
# NAME                                              READY   STATUS    RESTARTS   AGE
# pod/kpow-annual-kpow-aws-annual-c6bc849fb-zw5ww   0/1     Running   0          46s

# NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
# service/kpow-annual-kpow-aws-annual   ClusterIP   10.100.220.114   <none>        3000/TCP   47s
# ...

To access the UI, forward the service port to your local machine.

kubectl -n factorhouse port-forward service/kpow-annual-kpow-aws-annual 3000:3000

You can now access Kpow by navigating to http://localhost:3000 in your browser.

Deploy Kpow Hourly

Configure the Kpow Helm repository

The Helm chart for Kpow Hourly is available in the Factor House Helm repository. First, add the Helm repository.

helm repo add factorhouse https://charts.factorhouse.io

Next, update Helm repositories to ensure you install the latest version of Kpow.

helm repo update

Launch Kpow Hourly

Install Kpow using Helm, referencing the kpow-hourly service account which has the IAM policy for marketplace metering.

helm install kpow-hourly factorhouse/kpow-aws-hourly \
  -n factorhouse \
  --set serviceAccount.create=false \
  --set serviceAccount.name=kpow-hourly \
  --values ./values/eks-hourly.yaml

The Helm values are defined in values/eks-hourly.yaml.

# values/eks-hourly.yaml
env:
  ENVIRONMENT_NAME: "Kafka from Kpow Hourly"
envFromConfigMap: "kpow-config"
volumeMounts:
# ... (volume configuration is the same as annual)
volumes:
# ...
resources:
# ...

Verify and access Kpow Hourly

Check that the Kpow pod is running.

kubectl get all -l app.kubernetes.io/instance=kpow-hourly -n factorhouse
# NAME                                               READY   STATUS    RESTARTS   AGE
# pod/kpow-hourly-kpow-aws-hourly-68869b6cb9-x9prf   0/1     Running   0          83s

# NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
# service/kpow-hourly-kpow-aws-hourly   ClusterIP   10.100.221.36   <none>        3000/TCP   85s
# ...

To access the UI, forward the service port to a different local port (e.g., 3001) to avoid conflicts.

kubectl -n factorhouse port-forward service/kpow-hourly-kpow-aws-hourly 3001:3000

You can now access Kpow by navigating to http://localhost:3001 in your browser.

Delete resources

To avoid ongoing AWS charges, clean up all created resources in reverse order.

Delete Kpow and ConfigMaps

helm uninstall kpow-annual kpow-hourly -n factorhouse
kubectl delete -f manifests/kpow/config-files.yaml \
  -f manifests/kpow/config.yaml -n factorhouse

Delete the Kafka cluster and Strimzi operator

STRIMZI_VERSION="0.45.1"
kubectl delete -f manifests/kafka/kafka-cluster.yaml -n kafka
kubectl delete -f manifests/kafka/strimzi-cluster-operator-$STRIMZI_VERSION.yaml -n kafka

Delete the EKS cluster

This command will remove the cluster and all associated resources.

eksctl delete cluster -f manifests/eks/cluster.eksctl.yaml

Conclusion

In this guide, we have successfully deployed a complete, production-ready environment for monitoring Apache Kafka on AWS. By leveraging eksctl, we provisioned a robust EKS cluster with correctly configured IAM roles for service accounts, a critical step for secure integration with AWS services. We then deployed a Kafka cluster using the Strimzi operator, demonstrating the power of Kubernetes operators in simplifying complex stateful applications.

Finally, we walked through the deployment of both Kpow Annual and Kpow Hourly from the AWS Marketplace. This showcased the flexibility of Kpow's subscription models and their seamless integration with AWS for licensing and metering. You are now equipped with the knowledge to set up, configure, and manage Kpow on EKS, unlocking powerful insights and operational control over your Kafka ecosystem.

‍

Release

xx min read

Article

xx min read

Case Study

xx min read

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

Item 1
Item 2
Item 3

Unordered list

Item A
Item B
Item C

Text link

Bold text

Emphasis

^Superscript

_Subscript

How-to

xx min read

Deploy Kpow on EKS via AWS Marketplace using Helm

Overview

The source code and configuration files used in this guide can be found in the features/eks-deployment folder of this GitHub repository.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Prerequisites

To follow along the guide, you need:

CLI Tools:
- Recent versions of the following command-line interface (CLI) tools must be installed:
  - eksctl: For creating and managing Amazon EKS clusters.
  - kubectl: For interacting with Kubernetes resources.
  - Helm: The package manager for Kubernetes.
AWS Infrastructure:
- VPC: A Virtual Private Cloud (VPC) that has both public and private subnets is required.
- IAM Permissions: A user with the necessary IAM permissions to create an EKS cluster with a service account.
Kpow Subscription:
- A subscription to a Kpow product through the AWS Marketplace is required. After subscribing, you will receive access to the necessary components and deployment instructions.
- The specifics of accessing the container images and Helm chart depend on the chosen Kpow product:
  - Kpow Annual product:
    - Subscribing to the annual product provides access to the ECR (Elastic Container Registry) image and the corresponding Helm chart.
  - Kpow Hourly product:
    - For the hourly product, access to the ECR image will be provided and deployment utilizes the public Factor House Helm repository for installation.

Deploy an EKS cluster

We will use eksctl to provision an Amazon EKS cluster. The configuration for the cluster is defined in the manifests/eks/cluster.eksctl.yaml file within the repository.

⚠️ The provided configuration assumes the EKS cluster will be deployed in the us-east-1 region. If you intend to use a different region, you must update the metadata.region field and ensure the availability zone keys under vpc.subnets (e.g., us-east-1a, us-east-1b) match the availability zones of the subnets in your chosen region.

Here is the content of the cluster.eksctl.yaml file:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: fh-eks-cluster
  region: us-east-1

vpc:
  id: "<VPC-ID>"
  subnets:
    private:
      us-east-1a:
        id: "<PRIVATE-SUBNET-ID-1>"
      us-east-1b:
        id: "<PRIVATE-SUBNET-ID-2>"
    public:
      us-east-1a:
        id: "<PUBLIC-SUBNET-ID-1>"
      us-east-1b:
        id: "<PUBLIC-SUBNET-ID-2>"

iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: kpow-annual
        namespace: factorhouse
      attachPolicyARNs:
        - "arn:aws:iam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy"
    - metadata:
        name: kpow-hourly
        namespace: factorhouse
      attachPolicyARNs:
        - "arn:aws:iam::aws:policy/AWSMarketplaceMeteringRegisterUsage"

nodeGroups:
  - name: ng-dev
    instanceType: t3.medium
    desiredCapacity: 4
    minSize: 2
    maxSize: 6
    privateNetworking: true

This configuration sets up the following:

Cluster Metadata: A cluster named fh-eks-cluster in the us-east-1 region.
VPC: Specifies an existing VPC and its public/private subnets where the cluster resources will be deployed.
IAM with OIDC: Enables the IAM OIDC provider, which allows Kubernetes service accounts to be associated with IAM roles. This is crucial for granting AWS permissions to your pods.
Service Accounts:
- kpow-annual: Creates a service account for the Kpow Annual product. It attaches the AWSLicenseManagerConsumptionPolicy, allowing Kpow to validate its license with the AWS License Manager service.
- kpow-hourly: Creates a service account for the Kpow Hourly product. It attaches the AWSMarketplaceMeteringRegisterUsage policy, which is required for reporting usage metrics to the AWS Marketplace.
Node Group: Defines a managed node group named ng-dev with t3.medium instances. The worker nodes will be placed in the private subnets (privateNetworking: true).

Once you have updated the YAML file with your networking details, run the following command to create the cluster. This process can take 15-20 minutes to complete.

eksctl create cluster -f cluster.eksctl.yaml

kubectl get nodes
# NAME                              STATUS   ROLES    AGE     VERSION
# ip-192-168-...-21.ec2.internal   Ready    <none>   2m15s    v1.32.9-eks-113cf36
# ...

Launch a Kafka cluster

With the EKS cluster running, we will now launch an Apache Kafka cluster into it. We will use the Strimzi Kafka operator, which simplifies the process of running Kafka on Kubernetes.

Install the Strimzi operator

First, create a dedicated namespace for the Kafka cluster.

kubectl create namespace kafka

## Define the Strimzi version and download URL
STRIMZI_VERSION="0.45.1"
DOWNLOAD_URL=https://github.com/strimzi/strimzi-kafka-operator/releases/download/$STRIMZI_VERSION/strimzi-cluster-operator-$STRIMZI_VERSION.yaml

## Download the operator manifest
curl -L -o manifests/kafka/strimzi-cluster-operator-$STRIMZI_VERSION.yaml ${DOWNLOAD_URL}

## Modify the manifest to install the operator in the 'kafka' namespace
sed -i 's/namespace: .*/namespace: kafka/' manifests/kafka/strimzi-cluster-operator-$STRIMZI_VERSION.yaml

Now, apply the manifest to install the Strimzi operator in your EKS cluster.

kubectl apply -f manifests/kafka/strimzi-cluster-operator-0.45.1.yaml -n kafka

Deploy a Kafka cluster

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: fh-k8s-cluster
spec:
  kafka:
    version: 3.9.1
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
# ... (content truncated for brevity)

Deploy the Kafka cluster with the following command:

kubectl create -f manifests/kafka/kafka-cluster.yaml -n kafka

Verify the deployment

After a few minutes, all the necessary pods and services for Kafka will be running. You can verify this by listing all resources in the kafka namespace.

kubectl get all -n kafka -o name

# pod/fh-k8s-cluster-entity-operator-...
# pod/fh-k8s-cluster-kafka-0
# ...
# service/fh-k8s-cluster-kafka-bootstrap <-- Kafka bootstrap service
# ...

Deploy Kpow

Create ConfigMaps

We will use two Kubernetes ConfigMaps to manage Kpow's configuration. This approach separates the core configuration from the Helm deployment values.

kpow-config-files: This ConfigMap holds file-based configurations, including RBAC policies, JAAS configuration, and user properties for authentication.
kpow-config: This ConfigMap provides environment variables to the Kpow container, such as the Kafka bootstrap address and settings to enable our authentication provider.

The contents of these files can be found in the repository at manifests/kpow/config-files.yaml and manifests/kpow/config.yaml.

manifests/kpow/config-files.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kpow-config-files
  namespace: factorhouse
data:
  hash-rbac.yml: |
    # RBAC policies defining user roles and permissions
    admin_roles:
      - "kafka-admins"
    # ... (content truncated for brevity)
  hash-jaas.conf: |
    # JAAS login module configuration
    kpow {
      org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
      file="/etc/kpow/jaas/hash-realm.properties";
    };
    # ... (content truncated for brevity)
  hash-realm.properties: |
    # User credentials (username: password, roles)
    # admin/admin
    admin: CRYPT:adpexzg3FUZAk,server-administrators,content-administrators,kafka-admins
    # user/password
    user: password,kafka-users
    # ... (content truncated for brevity)

manifests/kpow/config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: kpow-config
  namespace: factorhouse
data:
  # Environment Configuration
  BOOTSTRAP: "fh-k8s-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092"
  REPLICATION_FACTOR: "1"
  # AuthN + AuthZ
  JAVA_TOOL_OPTIONS: "-Djava.awt.headless=true -Djava.security.auth.login.config=/etc/kpow/jaas/hash-jaas.conf"
  AUTH_PROVIDER_TYPE: "jetty"
  RBAC_CONFIGURATION_FILE: "/etc/kpow/rbac/hash-rbac.yml"

Apply these manifests to create the ConfigMaps in the factorhouse namespace.

kubectl apply -f manifests/kpow/config-files.yaml \
  -f manifests/kpow/config.yaml -n factorhouse

You can verify their creation by running:

kubectl get configmap -n factorhouse
# NAME                DATA   AGE
# kpow-config         5      ...
# kpow-config-files   3      ...

Deploy Kpow Annual

Download the Helm chart

The Helm chart for Kpow Annual is in a private Amazon ECR repository. First, authenticate your Helm client.

# Enable Helm's experimental support for OCI registries
export HELM_EXPERIMENTAL_OCI=1

# Log in to the AWS Marketplace ECR registry
aws ecr get-login-password \
    --region us-east-1 | helm registry login \
    --username AWS \
    --password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com

Next, pull and extract the chart.

# Create a directory, pull the chart, and extract it
mkdir -p awsmp-chart && cd awsmp-chart
# Pull the latest version of the Helm chart from ECR (add --version <x.x.x> to specify a version)
helm pull oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/factor-house/kpow-aws-annual
tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete
cd ..

Launch Kpow Annual

Now, install Kpow using Helm. We will reference the service account kpow-annual that was created during the EKS cluster setup, which has the required IAM policy for license management.

helm install kpow-annual ./awsmp-chart/kpow-aws-annual/ \
    -n factorhouse \
    --set serviceAccount.create=false \
    --set serviceAccount.name=kpow-annual \
    --values ./values/eks-annual.yaml

The Helm values for this deployment are in values/eks-annual.yaml. It mounts the configuration files from our ConfigMaps and sets resource limits.

# values/eks-annual.yaml
env:
  ENVIRONMENT_NAME: "Kafka from Kpow Annual"
envFromConfigMap: "kpow-config"
volumeMounts:
  - name: kpow-config-volumes
    mountPath: /etc/kpow/rbac/hash-rbac.yml
    subPath: hash-rbac.yml
  - name: kpow-config-volumes
    mountPath: /etc/kpow/jaas/hash-jaas.conf
    subPath: hash-jaas.conf
  - name: kpow-config-volumes
    mountPath: /etc/kpow/jaas/hash-realm.properties
    subPath: hash-realm.properties
volumes:
  - name: kpow-config-volumes
    configMap:
      name: "kpow-config-files"
resources:
  limits:
    cpu: 1
    memory: 0.5Gi
  requests:
    cpu: 1
    memory: 0.5Gi

Note: The CPU and memory values are intentionally set low for this guide. For production environments, check the official documentation for recommended capacity.

Verify and access Kpow Annual

Check that the Kpow pod is running successfully.

kubectl get all -l app.kubernetes.io/instance=kpow-annual -n factorhouse
# NAME                                              READY   STATUS    RESTARTS   AGE
# pod/kpow-annual-kpow-aws-annual-c6bc849fb-zw5ww   0/1     Running   0          46s

# NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
# service/kpow-annual-kpow-aws-annual   ClusterIP   10.100.220.114   <none>        3000/TCP   47s
# ...

To access the UI, forward the service port to your local machine.

kubectl -n factorhouse port-forward service/kpow-annual-kpow-aws-annual 3000:3000

You can now access Kpow by navigating to http://localhost:3000 in your browser.

Deploy Kpow Hourly

Configure the Kpow Helm repository

The Helm chart for Kpow Hourly is available in the Factor House Helm repository. First, add the Helm repository.

helm repo add factorhouse https://charts.factorhouse.io

Next, update Helm repositories to ensure you install the latest version of Kpow.

helm repo update

Launch Kpow Hourly

Install Kpow using Helm, referencing the kpow-hourly service account which has the IAM policy for marketplace metering.

helm install kpow-hourly factorhouse/kpow-aws-hourly \
  -n factorhouse \
  --set serviceAccount.create=false \
  --set serviceAccount.name=kpow-hourly \
  --values ./values/eks-hourly.yaml

The Helm values are defined in values/eks-hourly.yaml.

# values/eks-hourly.yaml
env:
  ENVIRONMENT_NAME: "Kafka from Kpow Hourly"
envFromConfigMap: "kpow-config"
volumeMounts:
# ... (volume configuration is the same as annual)
volumes:
# ...
resources:
# ...

Verify and access Kpow Hourly

Check that the Kpow pod is running.

kubectl get all -l app.kubernetes.io/instance=kpow-hourly -n factorhouse
# NAME                                               READY   STATUS    RESTARTS   AGE
# pod/kpow-hourly-kpow-aws-hourly-68869b6cb9-x9prf   0/1     Running   0          83s

# NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
# service/kpow-hourly-kpow-aws-hourly   ClusterIP   10.100.221.36   <none>        3000/TCP   85s
# ...

To access the UI, forward the service port to a different local port (e.g., 3001) to avoid conflicts.

kubectl -n factorhouse port-forward service/kpow-hourly-kpow-aws-hourly 3001:3000

You can now access Kpow by navigating to http://localhost:3001 in your browser.

Delete resources

To avoid ongoing AWS charges, clean up all created resources in reverse order.

Delete Kpow and ConfigMaps

helm uninstall kpow-annual kpow-hourly -n factorhouse
kubectl delete -f manifests/kpow/config-files.yaml \
  -f manifests/kpow/config.yaml -n factorhouse

Delete the Kafka cluster and Strimzi operator

STRIMZI_VERSION="0.45.1"
kubectl delete -f manifests/kafka/kafka-cluster.yaml -n kafka
kubectl delete -f manifests/kafka/strimzi-cluster-operator-$STRIMZI_VERSION.yaml -n kafka

Delete the EKS cluster

This command will remove the cluster and all associated resources.

eksctl delete cluster -f manifests/eks/cluster.eksctl.yaml

Conclusion

‍

Release

xx min read

Article

xx min read

Case Study

xx min read

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Block quote

Ordered list

Item 1
Item 2
Item 3

Unordered list

Item A
Item B
Item C

Text link

Bold text

Emphasis

^Superscript

_Subscript

How-to

xx min read

Release

xx min read

Release 94.6: Factor Platform, Ververica Integration, and kJQ Enhancements

The first Factor Platform release candidate is here, a major milestone toward a unified control plane for real-time data streaming technologies. This release also introduces Ververica Platform integration in Flex, plus support for Kafka Clients 4.1 / Confluent 8.0.0 and new kJQ operators for richer stream inspection.

Factor Platform release candidate: Early access to unified streaming control

For organisations operating streaming at scale, the challenge has never been about any one technology. It's about managing complexity across regions, tools, and teams while maintaining governance, performance, and cost control.

We've spent years building tools that bring clarity to Apache Kafka and Apache Flink. Now, we're taking everything we've learned and building something bigger: Factor Platform, a unified control plane for real-time data infrastructure.

Factor Platform delivers complete visibility and federated control across hundreds of clusters, multiple clouds, and distributed teams from a single interface. Engineers gain deep operational insight into jobs, topics, and lineage. Business and compliance teams benefit from native catalogs, FinOps intelligence, and audit-ready transparency.

The first release candidate is live. It's designed for early adopters exploring large-scale, persistent streaming environments, and it's ready to be shaped by the teams who use it.

Interested in early access? Contact sales@factorhouse.io

Unlocking native Flink management with Ververica Platform

Our collaboration with Ververica (the original creators of Apache Flink), enters a new phase with the introduction of Flex + Ververica Platform integration. This brings Flink’s enterprise management and observability capabilities directly into the Factor House ecosystem.

Flex users can now connect to Ververica Platform (Community or Enterprise v2) and instantly visualize session clusters, job deployments, and runtime performance. The current release provides a snapshot view of Ververica resources at startup, with live synchronization planned for future updates. It's a huge step toward true end-to-end streaming visibility—from data ingestion, to transformation, to delivery.

Configuration is straightforward: point to your Ververica REST API, authenticate via secure token, and your Flink environments appear right alongside your clusters.

This release represents just the beginning of our partnership with Ververica. Together, we’re exploring deeper integrations across the Flink ecosystem, including OpenShift and Amazon Managed Service for Apache Flink, to make enterprise-scale stream processing simpler and more powerful.

Read the full Ververica Platform integration guide →

Advancing Kafka support with Kafka Clients 4.1.0 and Confluent Schema SerDes 8.0.0

We’ve upgraded to Kafka Clients 4.1.0 / Confluent Schema SerDes 8.0.0, aligning Kpow with the latest Kafka ecosystem updates. Teams using custom Protobuf Serdes should review potential compatibility changes.

Read the full post →

Data Inspect gets more powerful with kJQ enhancements

Data Inspect in Kpow has been upgraded with improvements to kJQ, our lightweight JSON query language for streaming data. The new release introduces map() and select() functions, expanding the expressive power of kJQ for working with nested and dynamic data. These additions make it possible to iterate over collections, filter elements based on complex conditions, and compose advanced data quality or anomaly detection filters directly in the browser. Users can now extract specific values from arrays, filter deeply nested structures, and chain logic with built-in functions like contains, test, and is-empty.

For example, you can now write queries like:

.value.correctingProperty.names | map(.localeLanguageCode) | contains("pt")

Or filter and validate nested collections:

.value.names | map(select(.languageCode == "pt-Pt")) | is-empty | not

These updates make Data Inspect far more powerful for real-time debugging, validation, and exploratory data analysis. Explore the full range of examples and interactive demos in the kJQ documentation.

See map() and select() in action in the kJQ Playground →

Schema Registry performance improvements

We’ve greatly improved Schema Registry performance for large installations. The observation process now cuts down on the number of REST calls each schema observation makes by an order of magnitude. Kpow now defaults to SCHEMA_REGISTRY_OBSERVATION_VERSION=2, meaning all customers automatically benefit from these performance boosts.

Learn more about Schema Registry observation in the docs →

Article

xx min read

Kpow Custom Serdes and Protobuf v4.31.1

This post explains an update in the version of protobuf libraries used by Kpow, and a possible compatibility impact this update may cause to user defined Custom Serdes.

Kpow Custom Serdes and Protobuf v4.31.1

Note: The potential compatibility issues described in this post only impacts users who have implemented Custom Serdes that contain generated protobuf classes.

Resolution: If you encounter these compatibility issues, resolve them by re-generating any generated protobuf classes with protoc v31.1.

In the upcoming v94.6 release of Kpow, we're updating all Confluent Serdes dependencies to the latest major version 8.0.1.

In io.confluent/kafka-protobuf-serializer:8.0.1 the protobuf version is advanced from 3.25.5 to 4.31.1, and so the version of protobuf used by Kpow changes.

Confluent protobuf upgrade PR: https://github.com/confluentinc/schema-registry/pull/3569
Related Github issue: https://github.com/confluentinc/schema-registry/issues/3047

This is a major upgrade of the underlying protobuf libraries, and there are some breaking changes related to generated code.

Protobuf 3.26.6 introduces a breaking change that fails at runtime (deliberately) if the makeExtensionsImmutable method is called as part of generated protobuf code.

The decision to break at runtime was taken because earlier versions of protobuf were found to be vulnerable to the footmitten CVE.

Protobuf footmitten CVE and breaking change announcement: https://protobuf.dev/news/2025-01-23/
Apache protobuf discussion thread: https://lists.apache.org/thread/87osjw051xnx5l5v50dt3t81yfjxygwr
Comment on a Schema Registry ticket: https://github.com/confluentinc/schema-registry/issues/3360

We found that when we advanced to the 8.0.1 version of the libraries; we encountered issues with some test classes generated by 3.x protobuf libraries.

Compilation issues:

Compiling 14 source files to /home/runner/work/core/core/target/kpow-enterprise/classes
/home/runner/work/core/core/modules/kpow/src-java-dev/factorhouse/serdes/MyRecordOuterClass.java:129: error: cannot find symbol
        makeExtensionsImmutable();
        ^
  symbol:   method makeExtensionsImmutable()
  location: class MyRecord

Runtime issues:

Bad type on operand stack
Exception Details:
Location:
io/confluent/kafka/schemaregistry/protobuf/ProtobufSchema.toMessage(Lcom/google/protobuf/DescriptorProtos$FileDescriptorProto;Lcom/google/protobuf/DescriptorProtos$DescriptorProto;)Lcom/squareup/wire/schema/internal/parser/MessageElement; : invokestatic
Reason:
Type 'com/google/protobuf/DescriptorProtos$MessageOptions' (current frame, stack[1]) is not assignable to 'com/google/protobuf/GeneratedMessage$ExtendableMessage'
Current Frame:
bci:
flags: { }
locals: { 'com/google/protobuf/DescriptorProtos$FileDescriptorProto', 'com/google/protobuf/DescriptorProtos$DescriptorProto', 'java/lang/String', 'com/google/common/collect/ImmutableList$Builder', 'com/google/common/collect/ImmutableList$Builder', 'com/google/common/collect/ImmutableList$Builder', 'com/google/common/collect/ImmutableList$Builder', 'java/util/LinkedHashMap', 'java/util/LinkedHashMap', 'java/util/List', 'com/google/common/collect/ImmutableList$Builder' }
stack: { 'com/google/common/collect/ImmutableList$Builder', 'com/google/protobuf/DescriptorProtos$MessageOptions' }
Bytecode:
0000000: 2bb6 0334 4db2 0072 1303 352c b903 3703
0000010: 00b8 0159 4eb8 0159 3a04 b801 593a 05b8
0000020: 0159 3a06 bb02 8959 b702 8b3a 07bb 0289

If you encounter these compatibility issues, resolve them by re-generating any generated protobuf classes with protoc v31.1.

Case Study

xx min read

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Block quote

Ordered list

Item 1
Item 2
Item 3

Unordered list

Item A
Item B
Item C

Text link

Bold text

Emphasis

^Superscript

_Subscript

How-to

xx min read

Release

xx min read

Article

xx min read

Enhanced Under-Replicated Partition Detection in Kpow

Kpow now offers enhanced under-replicated partition (URP) detection for more accurate Kafka health monitoring. Our improved calculation correctly identifies URPs even when brokers are offline, providing a true, real-time view of your cluster's fault tolerance. This helps you proactively mitigate risks and ensure data durability.

Overview

In a distributed system like Apache Kafka, data is partitioned and replicated across multiple brokers to ensure high availability and fault tolerance. A partition is considered an under-replicated partition (URP) when the number of in-sync replicas (ISRs) falls below the configured replication factor. This scenario can arise from various issues, including broker failures, network partitions, or high load on specific brokers.

The presence of URPs is a significant concern as it indicates a degradation in your topics' fault tolerance. If another broker fails before the cluster recovers, you risk permanent data loss. A key challenge in Kafka management is accurately detecting these URPs in real-time, especially during common operational events like a broker failure. Standard monitoring methods can sometimes lag, creating a temporary but dangerous blind spot where a cluster appears healthy even though its resilience has been compromised.

This is precisely the challenge that Kpow's enhanced URP detection is designed to solve. By providing a more accurate and immediate assessment of your cluster's true fault tolerance, this feature delivers significant benefits. It gives you the confidence to act quickly on reliable data, the ability to proactively mitigate risks before they escalate, and ultimately, the power to ensure the resilience and durability of your critical data pipelines.

💡 Enhancement of URP detection is implemented in Release 94.5. For an overview of all the changes, check out the release note: Release 94.5: New Factor House docs, enhanced data inspection, and URP & KRaft improvements.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Enhanced calculation for more accurate health monitoring

Ensuring the fault tolerance of your Kafka clusters requires a precise and accurate count of under-replicated partitions. A challenging but common operational scenario can arise where a broker becomes unavailable, yet the overall cluster health status does not immediately reflect this change. This can mask a critical degradation in data durability, leading operations teams to believe their cluster is healthier than it actually is. Making decisions based on this incomplete information can delay necessary interventions.

To provide a more reliable and trustworthy view, Kpow has enhanced its calculation for under-replicated partitions. Instead of calculating replication status by iterating through each broker—a method that can be incomplete if a broker is offline and unreachable—our new calculation iterates directly through every topic-partition defined in the cluster.

This partition-centric approach provides a more comprehensive and authoritative view of the cluster's state. It is precisely this change that allows the system to correctly detect partitions with fewer in-sync replicas than the configured replication factor, even when brokers are offline and not reported by the AdminClient.

This enhancement ensures that Kpow's health monitoring is a precise reflection of your cluster's real-time condition. It gives you the confidence to trust the metrics you see and act decisively to investigate and resolve replication issues, thereby maintaining a resilient and robust system.

Surfacing URP details in Kpow

This vital health information, now powered by our more accurate calculation, continues to be clearly presented on both the Brokers and Topics pages of the user interface. We've retained this dual perspective as it remains essential for diagnosing problems from different angles—whether you're investigating a single problematic broker or assessing the health of a critical application's topic.

On both pages, summary statistics displays the total number of under-replicated partitions. If this count is greater than zero, it serves as an immediate visual alert. A detailed table automatically appears, listing all affected topics along with their specific URPs. This allows you to quickly identify which topics are at risk and gather the necessary context to restore full replication.

On the Brokers Page:

On the Topics Page:

To further strengthen monitoring and alerting capabilities, new Prometheus metrics have been introduced to track under-replicated partitions. These metrics integrate seamlessly with your existing observability stack and provide more granular insights for automated alerting and historical trend analysis:

broker_urp: The total number of under replicated topic partitions belonging to this broker.
topic_urp: The total number of under replicated partitions belonging to this topic.
topic_urp_total: The total number of under replicated partitions of all topics in the Kafka cluster.

Conclusion

Accurate and timely detection of under-replicated partitions (URPs) is fundamental to maintaining a resilient and reliable Apache Kafka cluster. With its enhanced calculation, Kpow provides a more precise and immediate understanding of your cluster's health by correctly identifying replication issues, particularly in scenarios involving broker failures. This enhanced detection, combined with detailed visibility in the Kpow UI and new Prometheus metrics for automated alerting, empowers you to proactively address replication issues, mitigate the risk of data loss, and ensure the continuous high performance of your real-time data pipelines. This feature update reaffirms Kpow's commitment to providing comprehensive and intuitive tooling for Kafka management and monitoring.

Case Study

xx min read

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Block quote

Ordered list

Item 1
Item 2
Item 3

Unordered list

Item A
Item B
Item C

Text link

Bold text

Emphasis

^Superscript

_Subscript

How-to

xx min read

Release

xx min read

Article

xx min read

Introducing Factor House Docs

We're excited to launch the new Factor House Docs, a unified hub for all our product documentation. Discover key improvements like a completely new task-based structure, interactive kJQ examples, and powerful search, all designed to help you find the information you need, faster than ever. Explore the new home for all things Kpow, Flex, Factor Platform and more.

Overview

We are excited to announce the launch of our new, unified documentation site: Factor House Docs. This new hub brings all our product documentation together into a single, streamlined resource, making it easier than ever to find the information you need.

Previously, our documentation was split between community and enterprise editions. Now, with Factor House Docs, all of that content is in one place with a simplified structure. This move is designed to provide a more cohesive and user-friendly experience for everyone.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Key hightlights

Unified product content

No more switching between different sites for community and enterprise documentation. All documentation for Kpow, Flex, and the upcoming Factor Platform is now in one centralized location. This consolidation simplifies the user experience by providing a single source of truth for all product-related information, regardless of your edition.

Clear feature availability

To help you quickly identify which features are available in your edition, we've introduced clear COMMUNITY, TEAM, and ENTERPRISE badges throughout the documentation. This makes it easy to understand the capabilities of your current plan and see what additional features are available if you choose to upgrade.

Improved organization

One of the most impactful changes in the new Factor House Docs is the move from a dense, reference-style list to a clear, task-oriented structure. This isn't just a redesign; it's a fundamental improvement in how you find and use our documentation.

A good example of this is how we've consolidated administrative features.

In the previous documentation, getting a complete view of your administrative capabilities required you to hunt through completely different sections. Core admin controls like Staged mutations and Temporary policies were buried under the User authorisation section. Meanwhile, other powerful administrative tools like Bulk actions and Data governance were located in the general Features list, mixed in with dozens of other unrelated items. To perform your duties, you had to already know which features we considered part of "authorization" and which were just "features."

The new documentation eliminates this guesswork by introducing a single, dedicated Administrative workflows section.

Content is now organized to reflect a natural user journey:

A clear onboarding path: Instead of a jumble of topics, you are now greeted with a clear, sequential path: Getting started, followed by dedicated Installation and Configuration sections. This guides you logically from initial setup to a fully running instance.
Topic-based grouping: The monolithic "Features" section is gone. Related capabilities are now grouped into intuitive, high-level categories. For instance, powerful data tools are now consolidated under the Query languages & Data management section. This creates a focused hub for anyone responsible for the data itself, bringing together everything from querying with kJQ to inspecting topics.
Enhanced discoverability: This new structure makes it much easier to discover the full range of Kpow's capabilities. By browsing top-level sections like Integration & Kafka management, you can quickly get a sense of the available tools without having to read through a long, unstructured list.

This streamlined layout is designed to get you to your answers faster, reducing search time and making the entire platform more accessible and easier to master.

Powerful search, instant answers

Finding the exact information you need is now faster and easier than ever. Using Algolia DocSearch, our documentation site delivers instant, relevant results across all sections. No more navigating multiple pages to locate a specific configuration property or function.

Whether you’re looking for an environment variable, a kJQ example, or details on Role-Based Access Control, the global search guides you directly to the content you need. This improvement ensures you spend less time searching and more time building.

Interactive kJQ examples

Our new documentation includes a dedicated section for fully interactive kJQ examples. You can now modify both the kJQ filter and the JSON data in real-time to see how different filters work and experiment with the syntax. This hands-on approach allows you to test everything from basic filters and function calls to complex, nested logic right in your browser.

Hands-on playground

To further enhance your learning experience, we've added a new "Playground" section. This area features a collection of labs and projects designed to help you explore the full range of product capabilities in a hands-on environment.

Ready for the future

The new Factor House Docs site is built to be a long-lasting resource. The documentation for the new Factor Platform will be added and expanded upon its release, ensuring that this hub will remain the most up-to-date and comprehensive resource for all product information now and in the future.

Conclusion

We are confident that the new Factor House Docs will be a valuable resource for our users. We encourage you to explore the new site and update your bookmarks. The previous documentation site will be retired.

‍

Case Study

xx min read

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Block quote

Ordered list

Item 1
Item 2
Item 3

Unordered list

Item A
Item B
Item C

Text link

Bold text

Emphasis

^Superscript

_Subscript

How-to

xx min read

Release

xx min read

Article

xx min read

Data Inspect Enhancements in Kpow 94.5

Kpow 94.5 enhances data inspection with comma-separated kJQ Projection expressions, in-browser search, and flexible deserialization options. This release also adds high-performance streaming for large datasets and expands kJQ with new transforms and functions—testable on our new interactive examples page. These updates provide deeper insights and more granular control over your Kafka data streams.

Overview

Following our recent improvements in Kpow release 94.3, which introduced AI-powered filtering and automatic deserialization, release 94.5 continues to enhance the data inspect experience. This latest version introduces several key additions, including comma-separated kJQ Projection expressions, in-browser search, multiple deserialization options, and attribute sorting. It also features a significant optimization with high-performance streaming and an enhancement to our query language with expanded kJQ capabilities. You can try out the new transforms and functions on the new interactive examples page on our Factor House docs. These updates provide more granular control and deeper insights into your Kafka topics, empowering developers to navigate and analyze their data with greater efficiency and precision.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Targeted data views with comma-separated kJQ Projection expressions

Kpow 94.5 introduces support for comma-separated projection expressions in kJQ, such as .value.base, .value.rates. This powerful feature allows you to extract multiple fields from Kafka records in a single query. Now you can create targeted data views without unnecessary information, streamlining your workflow and keeping your output clean. This functionality is available for both key and value sub-paths, offering greater flexibility in how you inspect your data.

Faster navigation with in-browser search

You can now use your browser's native search functionality (Ctrl + F) to quickly find records by JSON path or value when using kJQ filters. This eliminates the need to re-run queries, saving you time and effort. The results component is fully keyboard-friendly, adhering to the Listbox pattern for improved accessibility. This ensures a smoother and more predictable navigation experience for all users, including those who rely on screen readers.

Deeper insights into schemas and deserialization

Data Inspect now offers detailed schema metadata for each message, including schema IDs and deserializer types. This makes it easier to identify misaligned schemas and poison messages. To handle these problematic records, Kpow 94.5 provides several deserialization options:

Drop record (default): This option ignores erroneous records, displaying only the well-formatted ones.
Retain record: This includes both well-formatted and erroneous records. Problematic records are flagged with a 'Deserialization exception' message instead of displaying the raw, poisonous value.
Poison only: This option displays only the erroneous records, with the value recorded as 'Deserialization exception'.

Improved readability with attribute sorting

When you select the 'Pretty printed (sorted)' display option, the attributes of the key or value are now sorted alphabetically by name. This improves the readability and consistency of your data during inspection, making it easier to compare records and identify specific fields.

High-performance streaming for large datasets

Kpow 94.5 is optimized for performance, allowing you to stream over 500,000 records without any UI lag. This enables the efficient analysis of large datasets, so you can work with high-volume topics without compromising on speed or responsiveness.

Expanded kJQ capabilities with new transforms and functions

The kJQ language has been significantly expanded with a host of new transforms, including parse-json, floor, ceil, upper-case, lower-case, trim, ltrim, rtrim, reverse, sort, unique, first, last, keys, values, is-empty, and flatten. Additionally, new functions such as within, split, and join have been added to enable more complex data manipulation directly within your kJQ queries.

For more details on these new features, please refer to the updated kJQ manual. Also, be sure to visit the new interactive examples page on our new Factor House docs site—it's a great way to quickly verify your kJQ queries.

Conclusion

Kpow 94.5 builds upon the foundation of previous releases to deliver a more powerful and user-friendly data inspection experience. The latest additions—including kJQ Projection expressions, in-browser search, and flexible deserialization options—provide more granular control over your data. Together with high-performance streaming and expanded kJQ capabilities, these enhancements solidify Kpow as an essential tool for any developer working with Apache Kafka. Be sure to test out the new kJQ features on the interactive examples page on our Factor House docs. Ultimately, this release is designed to streamline your workflows, reduce debugging time, and empower you to unlock the full potential of your data.

‍

Case Study

xx min read

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Block quote

Ordered list

Item 1
Item 2
Item 3

Unordered list

Item A
Item B
Item C

Text link

Bold text

Emphasis

^Superscript

_Subscript

How-to

xx min read

Release

xx min read

Article

xx min read

Case Study

xx min read

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Block quote

Ordered list

Item 1
Item 2
Item 3

Unordered list

Item A
Item B
Item C

Text link

Bold text

Emphasis

^Superscript

_Subscript

How-to

xx min read

From Batch to Real-Time: A Hands-On CDC Project with Debezium, Kafka, and theLook eCommerce Data

This project transforms the static "theLook" eCommerce dataset into a live data stream. It uses a Python generator to simulate user activity in PostgreSQL, while Debezium captures every database change and streams it to Kafka. This creates a hands-on environment for building and testing real-time CDC pipelines.

Overview

The theLook eCommerce dataset is a valuable resource for data professionals. It provides a realistic, comprehensive schema for testing analytics queries and BI tools. However, it has one major limitation: it's static. It's a snapshot in time, designed for traditional batch workloads.

Modern data applications thrive on live, event-driven data. From real-time dashboards to responsive microservices, the ability to react to data as it changes is essential. How can we practice building these systems with a dataset that feels real?

To solve this, we've re-engineered theLook eCommerce data into a real-time, streaming data source. This project transforms the classic batch dataset into a dynamic environment for building and testing Change Data Capture (CDC) pipelines with Debezium and Kafka.

💡 The complete project, including all source code and setup instructions, is available on GitHub.

⚡ Looking for more hands-on labs and projects? Check out our previous posts: Introduction to Factor House Local and Building a Real-Time Leaderboard with Kafka and Flink to level up your streaming data skills.

What is Change Data Capture (CDC) with Debezium?

Change Data Capture is a design pattern for tracking row-level changes in a database (INSERT, UPDATE, DELETE) and making them available as an event stream. Instead of repeatedly querying a database for updates, CDC systems read the database's transaction log directly, capturing every committed change as it happens.

Debezium is a leading open-source, distributed platform for CDC. It provides a library of connectors that turn your existing databases into event streams. In this project, we use the Debezium PostgreSQL connector, which works by reading the database's write-ahead log (WAL). To enable this, the PostgreSQL server's wal_level is set to logical, which enriches the log with the detailed information needed for logical decoding.

With the Debezium PostgreSQL connector, we can use PostgreSQL's built-in pgoutput logical decoding plugin to stream the sequence of changes from the WAL. It operates on a push-based model, where the database actively sends changes to the Debezium connector as they are committed. The connector then processes these changes and pushes them as events to Kafka topics, ensuring low-latency data streaming.

Project Architecture: A Live eCommerce Store in a Box

This project combines a dynamic data generator with a complete CDC pipeline, allowing you to see the end-to-end flow of data.

Real-Time Data Generator

At the heart of the project is a Python-based simulator that brings theLook eCommerce dataset to life. It:

Simulates continuous user activity, including new user sign-ups, product browsing, purchases, and even order updates like cancellations or returns.
Writes this data directly into a PostgreSQL database, creating a constantly changing, realistic data source.
Models complex user journeys, from anonymous browsing sessions to multi-item orders.

This component transforms PostgreSQL from a static warehouse into a transactional database that mirrors a live application.

CDC Pipeline with Debezium and Kafka

With data flowing continuously into PostgreSQL, we can now capture it in real-time.

The PostgreSQL database is prepared with a PUBLICATION for all tables in our eCommerce schema. This publication acts as a logical grouping of tables whose changes should be made available to subscribers, in this case, the Debezium connector.
A Debezium PostgreSQL connector is deployed and configured to monitor all tables within the schema.
As the data generator writes new records, Debezium reads the WAL, captures every INSERT, UPDATE, and DELETE operation.
It then serializes these change events into Avro format and streams them into distinct Kafka topics for each table (e.g., ecomm.demo.users, ecomm.demo.orders).

The result is a reliable, low-latency stream of every single event happening in your e-commerce application, ready for consumption.

Why is This a Good Way to Learn?

This project provides a sandbox that is both realistic and easy to manage. You get hands-on experience with:

Realistic schema: Work with interconnected tables for users, orders, products, and events—not just a simple demo table.
Industry standard stack: Get familiar with the tools that power modern data platforms: PostgreSQL, Debezium, Kafka, and Docker.
End-to-end environment: The entire pipeline is runnable on your local machine, giving you a complete picture of how data flows from source to stream.

What Can You Build With This?

A real-time stream of eCommerce events in Kafka opens up many possibilities for development. This project is the perfect starting point for:

🔍 Building real-time analytics dashboards with tools like Apache Flink or Apache Pinot to monitor sales and user activity as it happens.
🧊 Ingesting data into a lakehouse (e.g., Apache Iceberg) with Apache Flink to keep your warehouse continuously updated with real-time data.
⚙️ Developing event-driven microservices that react to business events. For example, you could build a NotificationService that listens to the ecomm.demo.orders topic and sends a confirmation email when an order's status changes to Shipped.

Get Started in Minutes

The entire project is containerized and easy to set up.

Clone the factorhouse/examples repository from GitHub.
Start the infrastructure (Kafka, PostgreSQL, etc.) using Docker Compose.
Run the data generator via Docker Compose to populate the database.
Deploy the Debezium connector and monitor Kafka topics as they are created and populated with real-time data.

We'd love to see what you build with this. Join the Factor House Community Slack and share what you're working on.

Conclusion

This project bridges the gap between static, batch-oriented datasets and the dynamic, real-time world of modern data engineering. It provides a practical, hands-on environment to learn, experiment, and build production-ready CDC pipelines with confidence.

Release

xx min read

Article

xx min read

Kafka 4.1 Release: Queues, Stream Groups, and More

Apache Kafka 4.1 has landed: with queue support in preview, improved Kafka Streams coordination, and new security and metrics features, this release marks a major milestone for the future of real-time data systems.

At Factor House, we’re always tracking what’s new in the Apache Kafka ecosystem, both for our own products and to support the growing community of stream-first developers we work with every day. Here’s a quick rundown of what’s exciting in the 4.1.0 release.

Queues for Kafka move to preview (KIP-932)

After a long wait, Kafka’s new native queueing model is moving from Early Access to Preview. That means KIP-932, which introduces Share Consumer Groups, is stabilising. And while it’s not yet production-ready, it’s getting much closer.

So what does this mean? In short:

It brings queue-like semantics to Kafka, enabling multiple consumers to read from the same partition in parallel.
It supports out-of-order processing, with individual message acknowledgements and retries.
It introduces a more flexible model for building event-driven architectures that straddle the line between pub/sub and traditional messaging queues.

This is a big shift for teams building scalable consumer architectures, and one we’re watching very closely.

Kafka Streams gets smarter with Stream Groups (KIP-1071)

Kafka Streams applications just got a coordination upgrade.

KIP-1071 introduces a new rebalance protocol for Streams apps, based on KIP-848’s consumer group protocol. This update:

Streamlines how stream tasks are assigned and rebalanced
Makes scaling Kafka Streams applications smoother
Adds transparency and predictability to the rebalance process

If you’ve ever wrangled a cluster of Kafka Streams apps and found yourself wondering “why did that rebalance happen?”, this one’s for you.

Other noteworthy improvements

A few other highlights from the release that caught our eye:

KIP-877: A standardised API for plugin metrics — more visibility into Kafka internals, especially custom components.
KIP-891: Kafka Connect now supports multiple plugin versions, making upgrades and rollbacks less painful.
KIP-1050: Improved error handling for Transactional Producers, with clear exception categories that should simplify recovery strategies.
KIP-1139: Adds support for JWT Bearer OAuth 2.0, making it easier to manage secure access without static secrets.

All told, Kafka 4.1 includes contributions from 167 engineers across the globe. That's a testament to the strength and growth of the open source streaming community.

Our take at Factor House

“At Factor House, we’re already preparing to update our clients to Kafka 4.x in an upcoming product release. Features like improved plugin management and transactional error clarity are going to make life easier for developers, and we’re excited about what the queueing model means for the future of real-time stream consumption.” — Derek Troy-West, CEO, Factor House

Whether you’re running Kafka locally or at scale in production, Kafka 4.1 is a milestone release that makes the platform more powerful, more flexible, and more secure.

We’ll be diving deeper into some of these changes in future blog posts, particularly around how they affect real-world streaming workloads using tools like Kafka Connect, Kafka Streams, and our own product stack at Factor House.

Resources

Case Study

xx min read

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Block quote

Ordered list

Item 1
Item 2
Item 3

Unordered list

Item A
Item B
Item C

Text link

Bold text

Emphasis

^Superscript

_Subscript

How-to

xx min read

Release

xx min read

Article

xx min read

Melbourne Kafka x Flink July Meetup Recap: Real-time Data Hosted by Factor House & Confluent

From structuring data streams to spinning up full pipelines locally, our latest Kafka x Flink meetup in Melbourne was packed with hands-on demos and real-time insights. Catch the highlights and what's next.

On July 31st, we welcomed the Melbourne data streaming community to our Factor House HQ in Northcote for an evening of real-time insights, local dev tools, and excellent company, cohosted with our friends at Confluent.

The event, From Real-Time Data to Insights & Local Development with Kafka and Flink, was packed with engineers, data practitioners, and curious minds looking to deepen their knowledge of stream processing and hands-on development workflows.

Pizza, drinks, and chatter kicked things off, and the vibes didn't disappoint. The crowd was buzzing with conversation, from real-time architectures to favorite CLI tools.

Talks That Delivered

Olena Kutsenko, Staff Developer Advocate at Confluent

See Olena's presentation

Olena Kutsenko, Staff Developer Advocate at Confluent, took the stage first with The Art of Structuring Real-time Data Streams into Actionable Insights. Her talk offered a compelling walkthrough of the Kafka–Flink–Iceberg stack, showing how to tame messy real-time data and prepare it for analytics and AI. Attendees praised her clarity and technical depth:

“Olena managed to communicate her ideas in a way that was comprehensible to non-technical people, whilst still articulating the value for more experienced attendees.”

Jaehyeon Kim, Developer Experience Engineer at Factor House

Read Jae's blog post about Factor House Local & Labs

Next up, Jaehyeon Kim, Developer Experience Engineer at Factor House, gave a tour of Hands-On Local Development: Kafka, Flink, Docker, and more. Jae introduced Factor House Labs — a collection of open-source Docker environments designed to get engineers experimenting with modern data platforms fast.

“Jae’s passion for the product was infectious. His demo showed just how easy it is to spin up a full end-to-end data pipeline locally.”

Community, Connection, and What’s Next

Beyond the talks, a big highlight was the crowd itself. We saw first-timers, veterans, and everyone in between sharing ideas and chatting about use cases.

This meetup also marked the launch of the Factor House Community on Slack, a space to keep the conversation going, swap tips, and collaborate on all things real-time.

Next Stop: Sydney!

Couldn’t make it to Melbourne? This meetup will happen again in Sydney on September 16. Sydney Apache Kafka x Flink Meetup

Stay Connected

Want to hear about future events, tools, and hands-on learning experiences?

Join the Factor House Community on Slack

Follow Factor House on LinkedIn

Case Study

xx min read

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Block quote

Ordered list

Item 1
Item 2
Item 3

Unordered list

Item A
Item B
Item C

Text link

Bold text

Emphasis

^Superscript

_Subscript

How-to

xx min read

Release

xx min read

Events & Webinars

Stay plugged in with the Factor House team and our community.

Join the Factor Community

27 November 2025

[MELBOURNE, AUS] Apache Kafka and Apache Flink Meetup, 27 November

Melbourne, we’re making it a double feature. Workshop by day, meetup by night - same location, each with valuable content for data and software engineers, or those working with Data Streaming technologies. Build the backbone your apps deserve, then roll straight into the evening meetup.

Join Us

Event

26 November 2025

[SYDNEY, AUS] Apache Kafka and Apache Flink Meetup, 26 November

Sydney, we’re making it a double feature. Workshop by day, meetup by night - same location, each with valuable content for data and software engineers, or those working with Data Streaming technologies. Build the backbone your apps deserve, then roll straight into the evening meetup.

Join Us

Join the Factor Community

We’re building more than products, we’re building a community. Whether you're getting started or pushing the limits of what's possible with Kafka and Flink, we invite you to connect, share, and learn with others.