Kpow v88.6 brings Protobuf support for AWS Glue, easy configuration of JSON application logs, improved orphan schema detection, a JDK17 docker image, and improves integration with both Confluent Cloud and Confluent Schema Registry.

JDK17 Docker Image

Recently we discovered an issue with Amazon Corretto 11 that impacts memory management in a containerised environment. Although this issue will be resolved shortly, we have also expedited our plans to move to JDK17.

We provide a Corretto 17 container image of Kpow tagged with jdk17 in Dockerhub. In a later release we will adopt Corretto 17 as our standard base image.

AWS Glue Protobuf Support

Kpow now supports Protobuf schema in AWS Glue.

JSON Application Logs

Switch to JSON application logs by setting the environment variable:

LOG_FORMAT=json

Orphan Schema Detection

Kpow has an improved UX for orphan schema detection and deletion.

Kpow v88.5 brings improved integration with the Confluent Cloud Metrics API for large cloud clusters, and a number of minor improvements and bug fixes.

Confluent Cloud Metrics API

Kpow provides an optional Confluent Cloud Metrics API Integration.

When enabled Kpow will show full disk usage metrics in both graphs and tables just like a standard Kafka cluster.

Recent feedback identified improvements to this API integration in the case of large Kafka clusters (+1000 topics). These improvements are implemented in Kpow v88.5 and require no further configuration, just update and restart.

Kpow v88.4 brings nice new UI features including resizable columns and easy JSON export of tabular data. The results of a recent security audit bring improved websocket security and user session management configuration. Finally, we have reduced the local topic disk usage and CPU consumption.

This release blog provides details of both the 88.3 and 88.4 releases.

Table UI Improvements

Sometimes it's the little things.

Two often requested features of Kpow are the ability to download tabular information easily and to be able to resize columns for easy access to data.

Security Improvements

A recent security audit provided valuable feedback, with some minor improvements incorporated into Kpow v88.4.

Websocket Same-Origin Protection

The Kpow UI makes use of websockets and when a user session starts a websocket connection upgrade occurs in your browser.

It is now possible to restrict the websocket upgrade request with a same-origin policy.

To apply same-origin header checking to the websocket upgrade request configure the user-facing scheme, host, and (optional) port of the Kpow instance like so:

WS_ALLOWED_ORIGINS=https://kpow.mycorp:3000

Kpow websockets are also protected by CSRF tokens, same-origin checking is an optional extra setting. The default allowed origins are '*', in part because Kpow is often deployed in a manner that it is impossible for Kpow to determine the user-facing host of its own instance.

Concurrent User Login Controls

Kpow v88.4 introduces the ability to optionally detect and block concurrent logins to the same user account.

ALLOW_CONCURRENT_LOGIN=false

Concurrent sessions are allowed by default.

User Session Max Age

Kpow v88.4 provides a new configuration parameter to control the maximum user session age.

SESSION_MAX_AGE=-1

The default max age is -1, which means no expiry. You may manage this session age via the identity provide integrated with Kpow (e.g. Okta, etc). However if you are using Kpow with LDAP this new parameter allows you to evict session after a set period in seconds.

Performance Improvements

We continuously monitor, stress, and record metrics about Kpow's performance in multi-cluster, multi-connect, multi-schema environment.

Recent observations let us to performance improvements that reduce Kpow local topic disk usage by 75% and CPU usage by 50%.

Confluent Cloud Disk Usage

We recently detected that our internal topics were not being compressed in Confluent Cloud due to limitations on configuration that we rely on as default.

Kpow 88.4 brings performance improvements that drop disk usage in a Confluent Cloud environment by 95%.

Kpow v88.2 introduces improved support for monitoring Flink consumers, the ability to encrypt your Kpow configuration to avoid passwords in plaintext, new configuration options for connecting to Confluent Schema Registries that require mutual TLS for authentication, and more.

Simple Consumers

Kpow provides great insight into Kafka Consumer Groups.

Until v88.2 Kpow provided little insight into the earlier implementation of Kafka consumers - Simple Consumers. That's because simple consumers have no group membership or assignment information, often the only information we have on them is a name and an understanding of their consumer offsets.

Simple consumers are often anonymous, ad-hoc, or bespoke consumption workloads that manage their own state. One prominent user of simple consumers is Flink, the popular stream processing framework.

In Kpow v88.2 we introduce a new 'Simple Consumers' tab in the Consumers UI, under this tab you will find information about any simple consumer that we can identify consuming from your cluster - including Flink workloads.

Note: If hundreds of ad-hoc or simple groups degrade your Kpow performance, turn this feature off with SNAPSHOT_SIMPLE_GROUPS=false.

Simple Consumer UI

Simple consumers metrics are held separately from our existing consumer group metrics as simple consumers have a different lifecycle from consumer groups. They don't have a group state and tend to exist for as long as their offsets are commit to the consumer_offsets topic. That means old/idle simple consumers can have a prolonged period where they are considered lagging - as such we don't intermingle aggregate simple and consumer group metrics.

The following simple consumer metrics have been added to Kpow's Prometheus endpoint:

• "simple.group.offset.lag.total" - the total lag of all simple consumers
• "simple.group.offset.delta.total" - the total read/s of all simple consumers
• "simple.topic.consumption.inactive.mins" - the total minutes a topic has seen zero consumption by simple consumers
• "simple.topic.offset.delta" - the total read/s of all simple consumers to a topic
• "simple.topic.offset.lag" - the total lag of all simple consumers to a topic
• "simple.broker.offset.delta" - the total read/s of all simple consumers to a broker
• "simple.broker.offset.lag" - the total lag of all simple consumers to a broker

In a future release we will introduce aggregate simple consumer metrics to the Overview UI.

Kpow Encrypted Configuration

Kpow is deployed by our customers in every cloud and on premises. Often times sensitive configuration is protected by vaults or secret managers.

Kpow requires standard Kafka configuration to connect to and manage your Kafka resources, this can include plaintext passwords for keys, keystores, truststores, and even the bindPassword for LDAP if you are using our JAAS LDAP integration. Kpow v88.2 includes support for obfuscating and encrypting plaintext configuration.

For more information on how to secure Kpow configuration see the Kpow Secure Configuration Guide. Use the Java API to secure your own client configuration.

We have chosen to open-source our implementation as the kpow-secure project, available under the Apache 2.0 license.

Confluent Schema TLS Configuration

In this release we introduce support for connecting to Confluent Schema Registry with mutual TLS authentication.

The following environment variables can be used to configure registry connections with mutual TLS:

• SCHEMA_REGISTRY_SSL_KEYSTORE_LOCATION
• SCHEMA_REGISTRY_SSL_KEYSTORE_PASSWORD
• SCHEMA_REGISTRY_SSL_KEY_PASSWORD
• SCHEMA_REGISTRY_SSL_KEYSTORE_TYPE
• SCHEMA_REGISTRY_SSL_KEYMANAGER_ALGORITHM
• SCHEMA_REGISTRY_SSL_TRUSTSTORE_LOCATION
• SCHEMA_REGISTRY_SSL_TRUSTSTORE_PASSWORD
• SCHEMA_REGISTRY_SSL_TRUSTSTORE_TYPE
• SCHEMA_REGISTRY_SSL_TRUSTMANAGER_ALGORITHM
• SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM
• SCHEMA_REGISTRY_SSL_PROVIDER
• SCHEMA_REGISTRY_SSL_CIPHER_SUITES
• SCHEMA_REGISTRY_SSL_PROTOCOL
• SCHEMA_REGISTRY_SSL_ENABLED_PROTOCOLS
• SCHEMA_REGISTRY_SSL_SECURE_RANDOM_IMPLEMENTATION
• SCHEMA_REGISTRY_SSL_KEYSTORE_KEY
• SCHEMA_REGISTRY_SSL_KEYSTORE_CERTIFICATE_CHAIN
• SCHEMA_REGISTRY_SSL_TRUSTSTORE_CERTIFICATES
• SCHEMA_REGISTRY_SSL_ENGINE_FACTORY_CLASS

Kpow v88.1 is the first release to follow our new Major.Minor release model.

Major.Minor Releases

We're changing Kpow's release tag to give you more information about the significance of different releases.

88: The major version. This will change with each significant feature or security related release.

1:: The minor version. This will change with each intermediary release that should occur with our normal delivery frequency of every 1-2 weeks.

You should keep up to date with minor releases as they will include bug fixes, dependency updates, and any number of minor improvements to Kpow. Being behind on major releases is an indicator to you that you're missing out on significant new features or security improvements.

Kpow v88 features support for producing with message headers, Protobuf referenced schemas, a new topic partition increase function, and much more.

Message Header Production

Kpow now provides the ability to quickly and easily produce messages with headers, and reproduce existing messages with edited or updated headers.

Protobuf Referenced Schema

If your Protobuf schema includes referenced schema, Kpow will now resolve those referenced schema correctly on both Data Inspect and Data Produce.

Increase Topic Partitions

In the topic action menu you will find a new function to increase the number of partitions in a topic - use with caution!