The Importance of Accessibility at Factor House: A CEO's Perspective

When I was eleven there was an activity at school to draw a picture for your grandparents, the picture would then be laminated for you to give them as a gift. I was in my cool skateboarding phase, so I drew myself skateboarding on a half-pipe - and then I colored the sun in green. My teacher came over asked about my choice in colors. Some of the other kids had a curious peek, because they could see that I had done something wrong — even if I couldn't. That was the moment I learned that I am colorblind.

Color blindness hasn't significantly impacted my career beyond curtailing early thoughts of being a pilot. These days I use color blind modes in tools like GitHub to help with my work, and the biggest practical problem that protanopia gives me is that I can't tell when a banana is ripe until it starts getting spots. I eat a lot of green bananas.

Color blindness has, however, given me a better awareness of how easily products can become inaccessible. When a customer asked us if we could undertake to provide a Voluntary Product Accessibility Template (VPAT) for Kpow for Apache Kafka, we jumped at the chance to understand where our products were lacking and how to improve them.

Achieving WCAG 2.1 AA Compliance

A quick note before we congratulate ourselves too much - while the request for a VPAT was new to me, Web Content Accessibility Guidelines (WCAG) were not. I first heard of WCAG while working on a user portal for an insurance company in England way back in 2002. Web UI have changed a lot since those days, but the expectation that quality work results in accessible UI has remained the same.

Providing a VPAT requires a commitment to achieve and document established accessibility standards like WCAG and Section 508 of the Rehabilitation Act. Working towards this commitment helps organisations assess and communicate the accessibility features and compliance of their products, particularly for users who rely on assistive technologies or have accessibility needs. We understood that accessibility isn't just a product feature - it's a necessity for creating inclusive products that everyone can use effectively.

We partnered with an independent consultancy, AccessibilityOZ, to ensure we were doing this right. AccessibilityOZ involves people who use various assistive technologies and have different accessibility needs in the testing of your website and product. Their comprehensive approach included multiple audits as we completed our work, and was exactly what we needed to not just meet the accessibility standards but to exceed them.

I'm happy to say that Kpow for Apache Kafka is WCAG 2.1 AA compliant since March 2024, with a VPAT published for each release since. Flex for Apache Flink will have VPAT published from release 94.1 onwards - due shortly!

Web Accessibility is not a Tick-Box Exercise

The process of obtaining our VPAT was a significant undertaking, it required us to assess every aspect of Kpow and make necessary changes to ensure compliance with accessibility guidelines. We had to dig deep into our codebase, working from the ground up to close over 100 tickets raised by AccessibilityOz. It was a challenging process taking over 12 months, but it was also incredibly rewarding. We are better developers for it.

Working on accessibility not only improved our products for users with accessibility needs but has also enhanced the overall quality of Kpow for everyone. The improvements we made, such as better navigation and more readable layouts, have made Kpow a faster, more user-friendly tool.

There is no trade-off between accessibility and functionality; in fact, the two go hand in hand. The process of undertaking VPAT has made our products better.

Accessibility is now embedded in our development process at Factor House. We've implemented new tools like Storybook to detect accessibility issues, ensuring that every new feature we release is accessible from the start. We've also upskilled our team, making sure that all of our developers understand the importance of accessibility and are capable of delivering products that meet these standards.

As we continue to grow and evolve, accessibility will remain a core focus for us. We believe that providing accessible tools is not only the right thing to do but also essential for attracting and retaining the best customers. In today's market, organisations have a duty of care to their employees, and providing accessible tools is a fundamental part of that responsibility.

My advice to other tech leaders and developers is simple: prioritise accessibility. It's not just about meeting ticking a box; it's about improving your product for every user. At Factor House, we've seen firsthand how focusing on accessibility has made our products better for everyone, and I'm proud of the work we've done.

For us at Factor House, accessibility is not a one-time task but an ongoing commitment to excellence in product development.

At Factor House, we're dedicated to providing the best possible developer tooling for data-in-motion. Our products continue to evolve to meet the ever-growing demands of our customers, who rely on Kpow as the central repository for their organization's Kafka clusters and associated resources. For many teams, Kpow has become an integral part of their day-to-day Kafka workflow, offering a comprehensive suite of features to simplify and streamline Kafka management.

That's why we're thrilled to announce the release of Kpow's new API, available in version 93.1 onwards! This new API opens up a world of possibilities, allowing you to seamlessly integrate Kpow's capabilities into your own organization's tools, platforms, and operations.

In this blog post, we'll take a closer look at what our new API has to offer, how it can benefit your organization, and how you can get started with it. Let's dive in!

API overview

We're excited to introduce the latest addition to the Kpow family: our new API, now available in version 93.1 onwards. This API represents a significant milestone in our commitment to providing powerful, flexible, and easy-to-use tools for Kafka.

With our new API, you can now leverage Kpow's capabilities directly from your own tools and platforms, opening up a whole new range of possibilities for integrating Kpow into your existing workflows. Whether you're managing topics, consumer groups, or monitoring Kafka clusters, our API provides a seamless experience that mirrors the functionality of our user interface.

Our API is backed by all of Kpow's enterprise features, including role-based access control, multi-tenancy, and the audit log for data governance. This means you can securely manage access to your Kafka resources and isolate workloads as needed, all through a centralized interface.

In the sections below, we'll explore the different modules of Kpow's API, highlighting key features and benefits of each. Let's dive in and see how Kpow's API can empower your Kafka management!

Kafka API

Kpow's Kafka API allows you to perform a wide range of operations directly from your own tools and platforms. Here are some common Kafka operations you can perform with the API:

• Managing Topics: Create, delete, and modify Kafka topics, including configuring topic properties such as replication factor and partition count.
• Managing Consumer Groups: Create, delete, and manage consumer groups, including resetting consumer group offsets.
• Monitoring Kafka Clusters: Retrieve metrics and monitoring data for Kafka clusters, brokers, topics, and partitions.
• Managing ACLs (Access Control Lists): Configure access control for Kafka resources, including topics and consumer groups.
• Managing Quotas: Set and manage quotas for producer and consumer traffic, controlling the rate at which clients can produce or consume messages.
• Managing Transactional Producers: Implement transactional operations such as fencing and aborting transactions, ensuring data integrity in transactional systems.

Kpow's Kafka API supports the full surface of the AdminClient API, allowing you to leverage all the capabilities of Kafka within your own workflows.

Kpow's Kafka API is vendor-specific, meaning it supports any technology that speaks the Kafka protocol. Whether you're using RedPanda, MSK serverless, Confluent Cloud, or any other Kafka-compatible technology, you can integrate it seamlessly with Kpow.

For more details on the Kafka API, refer to our API documentation.

Schema Registry API

Kpow's Schema Registry API supports both AWS Glue and Confluent's schema registry. With this API you can:

• Create, edit and delete schemas
• Update schema compatibility
• Permanently delete soft-deleted schemas (saving money on your Confluent bill!)

For more details on the Schema Registry API, refer to our API documentation.

Kafka Connect API

Right now the Kafka Connect API endpoints only support Apache Kafka Connect and Confluent Cloud/Platform. We will be looking to support MSK Connect very shortly! With this API you can:

• Create, edit and delete connectors
• Restart/pause/stop connectors + tasks
• View connector task details like stacktraces

For more details on the Kafka Connect API, refer to our API documentation.

Kpow user management API

Kpow's user management API allows customers to manage user actions through an API:

• View user details (such as assigned roles)
• View and manage any temporary policies assigned to your user
• View and manage any scheduled mutations invoked by your user
• List all mutations performed by your user

Benefits for users

Kpow's API offers a centralized interface for managing all your Kafka clusters and related resources, regardless of the vendor or technology you're using. Whether you're operating in a multi-cloud environment, using multiple Kafka technologies, or managing on-premise clusters, Kpow provides a unified platform for managing all your Kafka infrastructure.

One of the key benefits of Kpow's API is its ability to simplify management and reduce complexity. By providing a single point of control for all your Kafka infrastructure, Kpow streamlines your Kafka workflows and makes it easier to manage your resources efficiently.

Additionally, Kpow's API is backed by robust role-based access control (RBAC) and multi-tenancy features. This ensures that you can securely manage access to your Kafka resources and isolate workloads as needed, providing the flexibility and security required for modern data management.

Already, many of our customers are leveraging Kpow's API and integrating it into their GitOps pipelines. This streamlines their operations and enhances their Kafka workflows, demonstrating the real-world benefits of using Kpow's API.

Getting started

Getting started with Kpow's API is quick and easy. Follow these simple steps to enable and start using the API in your environment:

API_ENABLED=true
API_PORT=3001

1. Enable the API: Add the following configuration to your Kpow deployment:

The API server will now be served at port 3001.

1. Verify the API: You can verify that the API is running using cURL:

curl -X GET http://kpow:3001/kafka/v1/clusters

1. Authentication setup: Configure authentication through API tokens. Instructions on how to set up authentication can be found in Kpow's API reference documentation.
2. Authorization setup: Configure RBAC for authorization to manage access to your Kafka resources. This is also covered in the API reference documentation.

Once you have completed these steps, you will be ready to start using Kpow's API to enhance your Kafka workflows. For more details and advanced usage, refer to our API reference documentation.

What's next

At Factor House, we're committed to continually enhancing our API to provide you with the best possible Kafka management experience. In the coming months, we will be investing heavily in expanding the capabilities of Kpow's API, with a focus on adding more modules and expanding the surface area.

Some of the major API enhancements we have planned include:

1. Kpow Data API: We are working on a data API that will allow querying and producing to Kafka topics directly through the API.
2. ksqlDB API: We plan to add support for the ksqlDB API, allowing you to manage your ksqlDB resources directly from the Kpow API.

Stay tuned for more updates on our API development, including guides on integrating the API with GitHub Actions for GitOps and walkthroughs on setting up clients for Kpow's API in Java and other languages. If you have any specific use cases or features you would like to see covered in future updates, please reach out and let us know!

You can learn more about the Kpow API by reading our official documentation.

Give us your feedback!

Thank you for joining us on this journey, and we look forward to seeing how you leverage Kpow's API to unlock new possibilities in your Kafka infrastructure.

Ready to experience the power of Kpow's API for yourself? Visit our API reference documentation to learn more about the capabilities of our API and how you can start integrating it into your workflows.

Have feedback or suggestions for future updates? We'd love to hear from you! Reach out to us and let us know how you're using Kpow's API to revolutionize your Kafka management.

Don't miss out on the latest updates and features! Subscribe to our newsletter to stay up-to-date with all the latest news and developments from Kpow.

Unlock the power of Kpow's API and take your Kafka management to the next level. Get started today!

Today we introduce Flex for Apache Flink and announce the Factor Platform, the future of distributed systems engineering.

Starting now, individuals can use Flex CE for free, even at work. Organisations can install Flex CE in up to three non-production environments.

Each installation of Flex CE can manage one Flink Cluster. See our feature matrix for more information.

Visiting Current '23? Stop past the Factor House booth for a full demo and a chat through our product roadmap!

Flex for Apache Flink is now GA 🚀

With the release of 92.1, we are excited to introduce the general availability of Flex for Apache Flink. Commercial and community editions available today for developers to enjoy.

Flex brings the power of Factor House’s core technology to the Flink tooling space, with the web application sharing Kpow’s sophisticated, intuitive UX, considered data management features, and advanced security capabilities including User Authentication, RBAC, Multi-Tenancy, Enterprise Integrations, and Audit Log alongside all the Flink job and task management and tracking capabilities you’d expect.

This enterprise-grade solution unlocks deep insights and provides unparalleled management capabilities across all environments, from local development to production, with users benefiting from cluster-level insights and individual job details, including comprehensive metrics, aggregated consumption, and production data.

A single Flex Standard or Enterprise Edition instance can manage up to 12 Flink clusters, with organizations able to configure Flex with RBAC and multi-tenancy to provide fine-grained control and access for their users.

The free community edition of Flex offers a suite of engineer-focused features designed to accelerate the Flink development process. Individual developers can use Flex to manage one Flink cluster, and Organizations can use Flex Community in up to three non-production environments.

Our mission is to delight and empower distributed systems engineers, and in talking with our community of expert users, it became apparent that the need for enterprise-grade Flink tooling was the next step.

Flex Community Edition only takes 5 minutes to sign up for, download, and install, so get on board and tell us what you think! Toot toot! 🚂

The Factor Platform: the Future of Distributed Systems Engineering

We are excited to announce the development of the Factor Platform, an integrated, multi-tech platform combining Kpow and Flex with full, secure API access providing complete control of distributed data systems. With Factor Platform organizations can manage, monitor, secure, and explore Apache Kafka and Apache Flink from a single Web-UI.

Unlock automation with a secure REST API for GitOPS control of the Factor Platform and connected Kafka and Flink resources - driving efficiency gains in enterprise engineering by providing unrivaled integration with distributed systems.

The plaform combines the market-leading features of Kpow and Flex, including the ability to deploy air-gapped and secure with production-tested integrations for Okta, OAuth, LDAP, SAML, RBAC, Multi-Tenancy, and audit capabilities for data governance.

The platform has a unique composable architecture. Our ultimate goal for the platform is to cover the entire surface area of distributed systems at both a UI and API level in a single consolidated UI for organisations who have several hundred instances of each resource to manage.

Factor House 2.0

Now that we’re a three-trick pony, a few things are changing, and a few aren’t.

Kpow for Apache Kafka is still our flagship solution - and we’ll ensure it remains the market-leading Kafka UI (and soon-to-be API) available.

We’ll also approach the ongoing improvement and development of Flex and Factor Platform with the same speed and zest as we have with Kpow. We’ve already started on our SQL gateway integration for Flex and can’t wait for the user feedback and feature requests to begin flowing in.

Kpow.io is being brought into the mothership and will live at factorhouse.io/kpow alongside Flex and Factor Platform, with the Factor House website getting a well-deserved makeover!

Automation Automation Automation

Factor House products share a mature, production-tested core IP and as such some features of the Factor Platform platform will naturally flow into our tooling products.

Building The Factor Platform allows us to venture into the exciting space between distributed systems, adding value where the engineer lives. The secure API provided by Factor Platform is an important progression product capability, allowing engineers to automate not only the control of Apache Flink and Apache Kafka resources but also to control the platform itself, modifing RBAC rules, adding and removing resources, and more.

We are pleased to confirm that Kpow and Flex will receive a capability boost in Quarter 1 / 2024 when the resource management API is integrated in those products respectively.

Factor House Product Roadmap and Changelogs

As part of the evolution and expansion of the Factor House toolset, we’ve published a complete product roadmap outlining the path ahead.

Similarly each product now has a details changelog, so you can reference previous iterations at a glance and measure our cadence:

• Kpow for Apache Kafka Changelog
• Flex for Apache Flink Changelog

About Factor House

Factor House specializes in developing market-leading tools for distributed systems. We believe that Kafka and Flink are transformative technologies, and every engineer should have access to tooling that makes working with distributed systems a joy.

Our solutions are built on the foundation of our extensive experience delivering data platforms and leverage our learnings from a decade of building systems with Apache Kafka, Apache Flink, Apache Cassandra, and Apache Storm.

As distributed systems engineers, we developed the tools we needed, which we consider the most powerful and user-friendly solutions in the market.

Read more about our team and experience.

We hope you find Flex Community Edition useful. If you encounter any problems or techincal questions just raise an issue on the Flex Github repository.

Interested in a commercial license? Reach out to sales@factorhouse.io any time to discuss requirements and start a POC.

Kylie Troy-West is a Co-Founder and COO of Factor House.

Factor House build essential tools for modern engineers.

Kpow Community Edition is a free, developer focused toolkit for Apache Kafka clusters, schema registries, and connect installations.

This may be the worst-kept secret in Kafka-tooling land because thousands of engineers are already using Kpow Community Edition (CE).

After a soft launch at Current '22, then a medium launch at Kafka Summit '23, we're making Kpow CE generally available to everyone, everywhere.

Starting today, individuals can use Kpow CE for free, even at work. Organisations can install Kpow CE in up to three non-production environments.

Each installation of Kpow CE can manage one Kafka Cluster, one Schema Registry, and one Connect cluster. See our feature matrix for more information.

Kpow for Apache Kafka

Apache Kafka is a transformative technology, every engineer should have access to tooling that makes Kafka a joy.

Built on our learnings from a decade of shipping systems with Kafka, we believe Kpow is the most powerful and intuitive Kafka Web UI available to engineers today.

Designed for enterprise integration from our first commit in 2018, Kpow runs securely in your network with zero data egress and a full suite of user authentication and authorization features including SAML, LDAP, OpenID, OAuth2, Okta, Keycloak, RBAC, Multi-Tenancy, Data-Masking, Audit Logs, and more.

Five years, 10k commits, and more than 800k Docker pulls later, Kpow now has tens of thousands of users in 100+ countries. Our focus on quality tooling sees Kpow stacked with features and our commitment to performance means that one instance of Kpow can manage up to a dozen Kafka clusters and associated resources.

Explore the full feature-set of Kpow in our live, multi-cluster demo environment.

Kpow Community Edition

Kpow CE is the free, light-weight version of Kpow that is packed with features to accelerate your Kafka experience (with none of the enterprise-y stuff, you know the drill).

If we have to give you one reason to install Kpow? Here it is..

Blazing Fast Multi-Topic Search!

Searching for data in Kafka can feel like trying to find the needle in a haystack. Kpow CE makes topic search easy with built-in support for JSON Query (JQ) predicates, e.g:

.key.id | endswith("4d4a") and
.value.trade.status == "final" and
.value.trade.price | to-double > 25.45 and
.value.partner.network == "AMEX" and
.headers.flag[0] == "audit"

Kpow's multi-topic search easily scans tens of thousands of messages a second, finding the ones that match your JQ predicate in a flash.

Read more about Kpow's implementation of JQ in our kJQ Documentation.

Monitor, Control, and Explore

Kpow CE covers the full surface area of Apache Kafka, from creating topics and resetting consumer group offsets to editing schema and restarting connectors.

Watch the Devoxx UK conference talk 'Tune-Up your Kafka Tooling with Kpow Community' and discover how to:

• Set up Kpow CE with a Kafka Cluster, Schema Registry, and Kafka Connect. [3:00]
• Skip the setup wizard and run Kpow CE with environment variables. [4:19]
• Learn how Kpow generates unique metrics and insights while running airgapped and secure. [6:45]
• Diagnose the root cause of production lag issues with multi-dimensional consumer metrics. [11:30]
• Skip poison-pill messages and recompute topics with consumer offset management. [13:30]
• Manage and monitor Kafka Connect clusters and schema registries. [15:22]
• Find needle-in-a-haystack messages with blazing-fast multi-topic search and built in JQ predicates. [16:46]
• Blend Clojure and JQ to make quick assertions about topic data with the kREPL. [22:18]

Tune-Up your Kafka Tooling!

Kpow CE works with any Kafka cluster v1.0.0+ including Confluent Cloud & Platform, AWS MSK, MSK Serverless, Redpanda, and clusters from providers like Aiven and Instaclustr.

Supported Kafka resource integrations include Confluent Schema Registry, AWS Glue, Apache Kafka Connect, Confluent Managed Connect, and MSK Connect.

Get started in minutes with the Kpow CE Docker container, our multi-arch build provides support for both ARM and x86.

docker run --pull=always -p 3000:3000 -m 2G factorhouse/kpow-ce:latest

Configure and launch Kpow CE:

• Access Kpow CE setup wizard on http://localhost:3000
• Follow the steps to get an individual or organization license.
• Configure your Kafka Cluster, Schema Registry, Connect Cluster.
• Launch Kpow CE!

See the configuration guide to learn how to configure Kpow CE with environment variables instead of using the setup wizard.

Running in Kubernetes? Use our Helm charts to deploy Kpow CE to your cluster.

We hope you find Kpow Community Edition useful. If you encounter any problems or techincal questions just raise an issue on the Kpow Github repository.

Interested in a commercial license? Reach out to sales@factorhouse.io any time to discuss requirements and start a POC.

Derek Troy-West is a Co-Founder and CEO of Factor House.

Factor House build essential tools for modern engineers.

This news has been many months in the making.

Today Derek, Tom, and I are proud to share our new look and name with the world. Factor House has arrived, and Operatr.IO is no more.

We are proud to be an independent, engineering-led software house. Our new name, Factor House, represents us, our software development style, and our business approach.

Software houses specialize in building bespoke, high-quality software products. In software design, code refactoring is the process of restructuring existing code or changing the factoring to improve its design, implementation, and structure without impacting its functionality.

Factor House reflects our commitment to our craft, passion for product development, and pride in our independence.

Building cutting-edge, high-quality enterprise software is at the heart of what we do.

Hey you! Don't you know who I am?

Nothing about bootstrapping a business is easy, including naming it. Truth be told, Operatr.IO was never meant to be our name. It was changed up in one of those 'your first idea won't work, and you need to make a decision right now' moments that often occur in the early days of launching a startup.

Because of this, our name lacked meaning and personal connection right from the start. It never complimented who we are or what we do. We've all worked in big organizations where this matters less, but Kpow and Factor House are deeply personal to us. Our founding team consists of Derek and I, who are married, and Tom, who is a long-time friend and collaborator.

Kpow is a passion project for us, bringing together 20 years of enterprise development experience and over 10 years of specialization with Apache Kafka®. We've all put everything into Factor House and Kpow and have worked tirelessly to get to this point.

But our lack of pride in our company brand impacted our ability to thrive.

None of this initially mattered because we're engineers who just want to build great tech - everything else was secondary. Because of this, our product Kpow is the exact thing we set out to build. No pivots, no compromises. Kpow continues to evolve every day, and our vision has expanded - but we built it, and now it's time for us to shout about it.

Factor House is the vehicle for us to do that.

The need for our company brand to evolve has been compounded by having the quality of our product recognized by global industry leaders; we regularly close deals with Fortune 500 companies yet are virtually unknown in the wider industry. This juxtaposition has motivated us to rebrand properly, expand our product offering, and take Kpow to the broader Kafka market.

We've come a long way

In our next major release we're excited to ship the free, community edition of Kpow (perfect for individual dev use) along with a beautiful brand-new Tailwind UI.

Our team are currently hard at work on our next major feature which is ksqlDB integration. If that interests you, get in touch!

I could show you a screenshot of Kpow from three years ago - but wow - thanks to the input of our amazing users we have come a long, long way.

Hello, we've arrived!

2022 has given us a series of small victories from 10xing our revenue, onboarding Fortune 500 clients, paying proper salaries, growing the team, setting up our first office, and the ceremonial burning of the desk in the bedroom (co-founding a company with your husband is one thing, a desk in the bedroom is another entirely). I digress. However, the best moment for me was seeing our new logo and brand identity for Factor House for the first time and thinking:

'We're a proper company now. We've arrived.'

Next week we are taking the team on a US tour sponsoring the Strange Loop and Current 2022 conferences. We're excited about the opportunities this will bring.

And after that? Well, you'll just have to wait and see. But we've got big ideas.

Kylie Troy-West is the Co-Founder and COO of Factor House.

Factor House build Kpow, a simple, secure, enterprise tool for Apache Kafka®.

‍

This post explores a breaking change to Apache Kafka producer behaviour, introduced in Kafka 3.2.0.

Update: We raised a PR to the Apache Kafka site and this information is now included in the upgrade-notes for Kafka 3.2.0.

Apache Kafka 3.2.0 implements KIP-679 that changes the default behaviour of Producer configuration to enable idempotence by default.

This change can cause message production to fail after you update to the 3.2.0 kafka-client libraries, it briefly impacted Kpow v88.6 (fixed in v88.7).

Originally released in Kafka 3.0.0 via KAFKA-10619, a bug in config validation meant this change was not fully implemented until KAFKA-13598 in Kafka 3.2.0.

It appears that issues were identified with the idea of changing default behaviour as KAFKA-13673 disables default idempotency when certain configuration is set on the producer, and KAFKA-13759 disables this change entirely for Kafka Connect. The cause of the issue that briefly impacted Kpow can be found in that last ticket:

> for brokers older than version 2.8 the IDEMPOTENT_WRITE ACL is required to be granted to the principal

In this post we explore:

1. Identifying this issue in Kpow
2. Details of the breaking change
3. Potential scope of impact
4. Issue remediation
5. Further implications

What is Kpow?

Kpow provides enterprise-grade monitoring, management, and control of Apache Kafka Clusters, Schema Registries, and Connect installations.

Uniquely for a product in use since 2018, Kpow is built for and from Kafka. We use Kafka Streams and internal topics for system state and long-term metrics computation (e.g. topic last-write, group last-read telemetry, etc). We use Kpow to monitor and build Kpow, it's turtles all the way down.

The combination of being widely used and well integrated with Kafka means we often have user-reports of issues before they become commonly known, as can be seen from our recent blog post on memory issues with Amazon Corretto 11. See the Kpow Changelog for full release notes.

Kpow Data Inspect UI

Impact on Kpow

Within one day of releasing Kpow v88.6 we received a user report that Kpow was showing the following error in the application logs:

/

10:06:29.278 ERROR [OperatrScheduler_Worker-2] operatr.compute.v3.materialization -- Error creating simple metrics for a7ca88d8-e1a7-4dec-9ae0-7248f5624c1d :cluster lkc-xxxx
org.apache.kafka.common.KafkaException: Cannot execute transactional method because we are in an error state
        at org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:1125)
        at org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartition(TransactionManager.java:442)
        at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:998)
        <...snip...>
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

Unfortunately the error log contained this red herring:

Caused by: org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

Kpow runs with every flavour of Kafka from v1.0.0 onwards, yes there are teams in the wild still using Kafka v1.0.0.

Our product monitors and manages Kafka clusters in organizations from publishing to payment networks. Where a cluster is protected by ACLs the Kpow user must have the minimum set of ACL permissions required for our product to work. When you don't have the correct permissions configured, you see that error.

We asked the user to check their ACL permissions, normally this quickly resolves the issue. They insist their permissions are correct. We run up a local 3-node Kafka cluster with ACLs configured (see our docker-compose configuration for a local SASL authenticated 3-node Kafka cluster that allows ACL testing) and run Kpow without encountering any issue. This is a proper head scratcher. Then two things happen at roughly the same time.

Firstly we noticed this part of the error log:

org.apache.kafka.common.KafkaException: Cannot execute transactional method because we are in an error state

We don't use transactional producers. We're aware of what they are and the role they play, but we have no need for them and have not configured them.

Then our tenacious user came back to us with an update:

> I think there's a bug in the 88.6 version.

> I rolled back to the 88.5 with the same set of permissions listed in this email and I'm able to see the metrics

We closed a record 49 minor issue tickets in Kpow v88.6. The majority were either old and redundant, or tweaks to our UI/UX. There were zero changes related to message production other than this seemingly innocuous library version bump:

<< [org.apache.kafka/kafka-streams "3.1.0"]
>> [org.apache.kafka/kafka-streams "3.2.0"]

We favour keeping close to latest of major libraries like Apache Kafka since they offer great quality and reliability.

That said we are fastidious about our dependency management. We read release notes and upgrade guides where available, changelogs if they exist, and if required we'll look at the commit diff between different versions of a library to determine if it is safe to proceed. We don't read every KIP though.

In this case if there was a note on a breaking change, we missed it. Reverting the Kafka-Streams library back to 3.1.0 fixed our issue and we released Kpow v88.7.

Breaking Change Details

The switch to default idempotent Producers can cause production to fail where:

1. The Kafka Cluster has brokers running version < 2.8.0, and
2. The Kafka Cluster has ACLs configured, but not IDEMPOTENT_WRITE and
3. Producer configuration is default, or is capable of being defaulted to idempotent, and
4. The producing application is using Kafka-Clients version > 3.2.0

This is because in Kafka prior to v2.8.0 there was an ACL specifically for idempotent production named IDEMPOTENT_WRITE.

If you are not concerned with idempotency and have ACL set it is likely that IDEMPOTENT_WRITE is false, and TOPIC_WRITE is set instead.

If your producers have default configuration, or are not explicitly idempotent but fall within the bounds of KAFKA-13673, and you update the client libraries to > 3.2.0, you will now have idempotent producers and they will fail to write with the ACL error that we encountered.

Potential Scope of Impact

Perhaps Kpow is an uncommon application, being widely used against nearly every sort of Kafka cluster.

However if we take another look at the conditions for the breaking change to occur:

1. The Kafka Cluster has brokers running version < 2.8.0. This is fairly common.
2. The Kafka Cluster has ACLs configured but not IDEMPOTENT_WRITE. This is fairly common.
3. Producers configuration is default, or is capable of being defaulted to idempotent. This is fairly common.
4. The producing application is using Kafka-Clients version > 3.2.0. This is very easy to do.

The first three circumstances are more common than you might expect, and bumping a Kafka client library in your application dependencies is really easy to do. Much easier than upgrading the version of your Kafka brokers, for instance.

Issue Remediation

If you encounter this issue it is fairly easy to remediate, you have several options:

1. Rollback your kafka-client library to < 3.2.0, or
2. Configure producer idempotency to false, or
3. Configure IDEMPOTENT_WRITE ACLs, or
4. Upgrade your broker version to > 2.8.0

Further Considerations

This change to Kafka leaves us in a slightly strange state. Is your producer default idempotent or not? The answer is maybe. If you haven't configured anything contrary to idempotency then yes, it should be idempotent. That's a weird default position to hold that requires further knowledge from a user than I would expect.

If enterprise-grade Apache Kafka tooling with a focus on performance and reliability interests you, sign up for a free 30-day trial today.