This minor version release from Factor House:

• Improves protobuf rendering
• Sharpens light-mode
• Simplifies community edition setup
• Resolves a number of small bugs, and,
• Bumps Kafka client dependencies to v3.7.0.

Advance Apache Kafka Clients to 3.7.0

Client reports have lead us to observe unusual consumer behaviour in limited cases when using Kpow's data inspect feature.

We believe there may be some hard to reproduce bugs in the underlying fetch mechanism of the Kafka consumer within the standard Java libraries.

This area of code is currently being refactored within the Apache Kafka project and there are bug tickets that imply bugs are being squashed in due course.

We observed consumer issues being resolved in a client installation of Kpow as the Kafka version bumped above 3.6.2. We will keep an eye this area for the time being and produce a more meaningful blog post with details at some point in the future.

Improve Protobuf rendering

Data inspect now shows default properties for Protobuf messages for both AWS Glue and Confluent Schema Registry.

In below screenshot occupied has a value of false which is the default value for a boolean.

It is now displayed in the results also it can be used in the kJQ filter.

Read more details in the original issue here

Improve light mode

We have improved the light mode in Flex and Kpow, elements have more contrast and visible borders.

This is an area of continuous improvement for us as we take accessibility very seriously. If you want to see further improvements in this space then please give us feedback

Import properties files in Kpow Community Setup wizard

Users can now import .properties file in Kpow Community Setup wizard to set the connection details for Apache Kafka by clicking on Import properties file button.

Factor House release v93.3 brings full support for schema references in Confluent Schema Registry.

With this new release you can now create and edit AVRO, JSONSchema, and Protobuf schema with references, as well as consume and produce messages with those schema.

This release also includes improved general performance by removing producer and transaction observation, improved visualisation of Kafka Streams, consumer group, and Flink topologies, a better responsive UI for smaller screens, along with plenty of minor bugfixes.

Confluent Schema Registry References

Confluent schema registry provides support for the notion of schema references, the ability of a schema to refer to other schemas.

Each implementation is slightly different depending on the source format, but broadly speaking you can pass a json payload when creating and editing schema that defines precisely how to resolve referenced schema in the registry. See the Confluent documentation on schema references for more information.

Kpow now provides full support for managing schema references when creating and editing schema as well as producing and consuming messages with those schema.

General Performance Improvements

User feedback led us to understand that out approach to observing active producers and transactions in a Kafka cluster was performing sub-optimally. We have taken the decision to remove the recently introduced Producers UI in the short-term while we reconsider this piece of work.

Improved Kafka Streams, Consumer, and Flink Topology Viz

As we move towards the launch of the Factor Platform - our Unified Platform for Data in Motion - some UI improvements are finding their way into Kpow and Flex.

Flink topology viewing just got a lot easier on the eye!

Factor House release v93.2 brings a new minor version to our suite of products for Apache Kafka and Apache Flink.

This is the 117th release of Factor House products, and marks more than 2.5M downloads of Kpow and Flex from Dockerhub in the past five years.

The major new feature in v93.2 is the connector auto-restart which provides a hassle-free way for organizations to make sure that their connectors are always up and running.

This release also includes:

• Improve ksqlDB UI: import sql files, better error handling and timing metrics
• Allow RBAC wildcards at both start and end of resource, such as *foo*
• Introduce download capabilities in krepl
• Preserve the last visited page when navigating between resources
• Add keybindings table to data inspect + ksqldb forms
• Fix bug starting Kpow with PERSISTENCE_MODE=audit
• Improve mutation notification UX
• Fix bug with tenancy resource handling for fine-grained Schema resources

Connector Auto-restart

Kpow has supported the ability to restart connectors including bulk-restart since its inception, but we now offers the ability to automatically restart specified connectors and failing tasks, providing enhanced reliability and uptime for your data integration workflows.

To enable this feature, simply set the CONNECT_AUTO_RESTART environment variable in your configuration.

When activated, Kpow will monitor for failed connectors at one-minute intervals. After attempting to restart a connector, Kpow will wait for the user-configured amount of time before trying again. By default, this interval is set to 10 minutes, but you can adjust it using the CONNECT_AUTO_RESTART_WINDOW_MS environment variable.

Additionally, Kpow allows you to specify a limit on the number of connectors that can be restarted automatically. The default limit is 50 connectors, but you can modify this setting using the CONNECT_AUTO_RESTART_LIMIT environment variable. This feature helps prevent excessive restart requests being sent to the Kafka Connect cluster.

All restart attempts will be logged in the audit log as actions performed by the kpow_system user.

If you have configured Kpow's Slack integration, all restart attempts will also be sent to a Slack webhook. This integration enhances real-time monitoring and alerting capabilities, ensuring that your team is promptly notified of any restart actions taken by Kpow.

Note: If the connector is failing more frequently than CONNECT_AUTO_RESTART_WINDOW_MS limit then it may require manual intervention.

Example configuration

CONNECT_AUTO_RESTART="*" # restart all failed connectors
CONNECT_AUTO_RESTART="dbz-connector-1" # restart **only** the connector dbz-connector-1 when it enters a failed state
CONNECT_AUTO_RESTART="mysql-prod-us*" # restart any connector that matches the wildcard (mysql-prod-us-east1, mysql-prod-us-west2, etc)
CONNECT_AUTO_RESTART="payments-*,*-stage" # you can specify many filters by providing a comma-separated list

ksqlDB UI improvements

ksqlDB UI has been streamlined. Users can now import a file into the query editor by simply clicking at Load SQL from file. Error messages has been improved. Also, you can now see metrics about timing.

‍

Our mission at Factor House is to empower every engineer in the streaming tech space with superb tooling.

We are pleased to report that Kpow for Apache Kafka has achieved WCAG 2.1 AA accessibility compliance, confirmed and audited by an independent accessibility consultancy.

Kpow is the first Kafka UI to achieve WCAG 2.1 AA compliance and the only Kafka UI with a published VPAT (Voluntary Product Accessibility Template) report.

Kpow Community Edition achieves the same high standard of accessibility compliance as our commercially available products, and is free to use both by individuals and organisations, making accessible Kafka tooling available to everyone!

Release 92.4 also introduces new features for Kafka Connect, new features for Confluent Schema Registry, resolves a security advisory around Weak SSL/TLS Key Exchange, and fixes a number of minor bugs. See below for details of each and a full release changelog.

Product Accessibility at Factor House

As a part of our commitment to quality engineering, each future release of Kpow for Apache Kafka by Factor House will have a corresponding VPAT report published.

Flex for Apache Flink will achieve WCAG 2.1 AA compliance in April 2024, after which all Factor House product releases will contain a published VPAT report.

Kpow WCAG 2.1 AA Accessibility Compliance

Release 92.4 concludes a 12-month program of work in which the Factor House team resolved over 100 accessibility tickets.

We learned an enormous amount through the audits, workshops, issues, and expert guidance provided by the team at AccessibilityOz.

AccessibilityOz follow an exacting approach to accessibility audits. Their work includes testing with automated accessibility testing tool OzART, manual testing, testing with screen readers (JAWS, NVDA, VoiceOver, TalkBack), and color contrast analysis testing using TPG Colour Contrast Analyser.

We understand accessibility is not a tick-box, there remain areas where we can improve. We're committed to maintaing and working through accessibility tickets to further improve our products. As always, we welcome bug reports from our users.

See Kpow's Accessibility Documentation and VPAT for more details, including guides to using Factor House products with screen readers and keyboard shortcuts.

Kafka Connect Features

Release 92.4 introduces the ability to secure Kafka Connect connections with mTLS, see our Kafka Connect documentation for details.

This release also introduces the ability to 'STOP' Kafka Connectors, a new feature in Kafka 3.6.0 that is now available in Kpow.

Confluent Schema Registry Features

Confluent introduced changes to the streams governance pricing on March 04, 2024:

Effective March 4, the free schema limit in Stream Governance Essentials will be 100 schemas per environment. Schemas over the free schema limit will be billed at a rate of $0.002/schema/hour.

Because you have active environments over the 100 schema limit, we will credit your Confluent Cloud account to cover 90 days of new schema charges based on your current schema count.

We have 55 schemas in our demo environment Confluent Schema Registry but were notified that we would be billed for excess schema.

Then we realised Confluent must be intent on charging us for soft-deleted schema.

When you delete a schema in Confluent Schema Registry it is not actually deleted, just marked for deletion and considered 'soft-deleted'. We had 269 soft-deleted schema just hanging around waiting to cost us money.

The new 'permanent delete' schema function allowed us to bulk delete 269 schema in seconds, saving us ~US$170/mo in excess charges in this one environment alone.

Weak SSL/TLS Key Exchange Security Advisory

A recent security advisory resulting from a Qualsys scan raised an issue regarding potential weak ciphers being available in Kpow's SSL handshake.

Note: these ciphers only apply to you if you are using Kpow's in-built HTTPS UI serving capabilities.

Finding: Weak SSL/TLS Key Exchange

Result:

PROTOCOL  CIPHER                      GROUP   KEY-SIZE   FORWARD-SECRET   CLASSICAL-STRENGTH   QUANTUM-STRENGTH
TLSv1.2   DHE-RSA-AES256-GCM-SHA384   DHE     1024       yes              80                   low
TLSv1.2   DHE-RSA-AES128-GCM-SHA256   DHE     1024       yes              80                   low
TLSv1.2   DHE-RSA-AES256-SHA256       DHE     1024       yes              80                   low
TLSv1.2   DHE-RSA-AES128-SHA256       DHE     1024       yes              80                   low

These ciphers are now removed by default in v92.4 of Kpow, if you rely on these ciphers and are comfortable retaining them you can revert back to previous Kpow behaviour by setting the following environment variable:

HTTPS_CIPHER_SET=v1

Factor House release v93.1 brings a new major version to our suite of products for Apache Kafka and Apache Flink.

This is the 116th release of Factor House products, and marks more than 2.3M downloads of Kpow and Flex from Dockerhub in the past five years.

The major new feature in v93.1 is the secure, vendor-agnostic, OpenAPI 3.1 REST API for Kafka, Connect, and Schema resources that is now available in Kpow.

This release also includes:

• New capabilities for restricting visibility of menu items in our product UI
• Better control of user access to resources with new fine-grained RBAC permissions
• Support for cross-account (STSAssumeRole) authentication for MSK Managed Connect
• Fix for a Confluent Cloud cluster performance related bug

OpenAPI 3.1 Rest API for Apache Kafka

Kpow already provides a secure, acceessible, enterprise-grade Web UI covering the entire surface area of Apache Kafka, Kafka Connect, Schema Registry, and ksqlDB.

With release v93.1, you can now integrate the full power of Kpow's capabilities with your own internal products and/or CI/CD GitOps pipelines by using Kpow's secure REST API.

Getting started is easy, just add the following configuration to your Kpow deployment:

API_ENABLED="true"API_PORT="4000"

After a restart, you can begin accessing the API on the configured port:

curl -v kpow-staging.zcorp.com:4000/kafka/v1/clusters*   Trying 127.0.0.1:4000...* Connected to kpow-staging.zcorp.com (127.0.0.1) port 4000 (#0)> GET /kafka/v1/clusters HTTP/1.1> Host: kpow-staging.zcorp.com:4000> User-Agent: curl/7.79.1> Accept: */*>< HTTP/1.1 200 OK< Content-Type: application/json;charset=utf-8< Vary: Accept-Encoding, User-Agent< Content-Length: 139<* Connection #0 to host localhost left intact{"clusters":[{"id":"0TEeq2akSkGlrow1awdj_w","label":"Trade Book (Staging)","is_confluent":false}],"metadata":{"tenant_id":"__kpow_global"}

Kpow's new API is secured via the RBAC and Tenancy rules that govern access to the web UI.

View the full OpenAPI 3.1 Kpow REST API specification to learn more about securing the API and the provided modules and capabilities.

Future releases will include full control of ksqlDB resources, access to Kpow's world-class topic search and message production functions, a full OpenAPI 3.1 API for Apache Flink, and introduce support for mTLS and OpenID authentication of API users. Watch this space!

Fine-Grained RBAC Permissions

Kpow now provides greater control of user permissions with derived, fine-grained user actions:

SCHEMA_EDIT

Permission to edit and delete schema is governed by the SCHEMA_EDIT action.

You can now choose to assign either SCHEMA_EDIT_VERSION or SCHEMA_DELETE individually.

CONNECT_EDIT

Permission to edit, delete, and alter connectors is governed by the CONNECT_EDIT action.

You can now choose to assign either of CONNECT_EDIT_CONFIG, CONNECT_DELETE, or CONNECT_ALTER_STATE individually.

TOPIC_INSPECT

Permission to search for data on topics and download any applicable results is governed by the TOPIC_INSPECT action.

You can now choose to assign either TOPIC_DATA_QUERY or TOPIC_DATA_DOWNLOAD individually.

For more information see Kpow's authorization documentation.

Product Presentation Mode

Factor House products now offer the ability to restrict UI menu options where a user does not have visibility of a resource.

For example, the following configuration will hide the connect, schema, and ksqldb main navigation options when a user does not have access to that type of resource:

PRESENTATION_MODE=HIDE_RESOURCES

Presentation mode can be set at a global level, with the configuration described above, or at a user-tenant level.

Learn more about specifying the presentation mode for a tenant in Kpow's tenancy documentation.

MSK Cross-Account Authentication

Kpow now provides the ability to provide either static credentials or specify cross-account access (STSAssumeRole) when authenticating MSK Connect.

See Kpow's MSK Connect configuration documentation for more.

Confluent Cloud Observation Performance

We recently discovered a bug in Confluent Cloud that caused Kpow's observation of new clusters (presumably ones that use the Kraft protocol) to take longer than desired.

This bug has been resolved in Kpow v93.1.

Release v92.3 introduces extensive UI accessibility improvements to Kpow and Flex along with new features, improvements, and bug fixes.

Towards an Accessibile UI

Engineering teams require tools that are accessible to all staff throughout their organisation.

Meeting the needs of our customers means providing the best quality tooling and ensuring that tooling is accessible to all engineers.

In 2023 we engaged the team at AccessibilityOz to assist us in meeting WCAG 2.1 level AA guidelines for accessibility throughout our suite of products.

AccessibilityOz provided a full audit of our product UI, leading to several months of work remediating issues raised in the audit report and following workshops.

Release 92.3 is being re-audited by AccessibilityOz. When that re-audit is complete we will publish a full product accessibility guide and VPAT (Voluntary Product Accessibity Template). See our Accessibility Documentation for more details, including guides to using Factor House products with screen readers and keyboard shortcuts.

We understand that accessibility is a process, not a tick-box. Maintaining a high level of accessibility is one part of the commitment to engineering quality at Factor House.

New Features and Improvements

Release 92.3 includes a number of new features, including a new Topic Partition Reassignments UI that provides the ability to view and cancel ongoing partition reassignments.

See the full changelog for details of other improvements and bug fixes in this release.