Kpow Custom Serdes and Protobuf v4.31.1

Note: The potential compatibility issues described in this post only impacts users who have implemented Custom Serdes that contain generated protobuf classes.

Resolution: If you encounter these compatibility issues, resolve them by re-generating any generated protobuf classes with protoc v31.1.

In the upcoming v94.6 release of Kpow, we're updating all Confluent Serdes dependencies to the latest major version 8.0.1.

In io.confluent/kafka-protobuf-serializer:8.0.1 the protobuf version is advanced from 3.25.5 to 4.31.1, and so the version of protobuf used by Kpow changes.

• Confluent protobuf upgrade PR: https://github.com/confluentinc/schema-registry/pull/3569
• Related Github issue: https://github.com/confluentinc/schema-registry/issues/3047

This is a major upgrade of the underlying protobuf libraries, and there are some breaking changes related to generated code.

Protobuf 3.26.6 introduces a breaking change that fails at runtime (deliberately) if the makeExtensionsImmutable method is called as part of generated protobuf code.

The decision to break at runtime was taken because earlier versions of protobuf were found to be vulnerable to the footmitten CVE.

• Protobuf footmitten CVE and breaking change announcement: https://protobuf.dev/news/2025-01-23/
• Apache protobuf discussion thread: https://lists.apache.org/thread/87osjw051xnx5l5v50dt3t81yfjxygwr
• Comment on a Schema Registry ticket: https://github.com/confluentinc/schema-registry/issues/3360

We found that when we advanced to the 8.0.1 version of the libraries; we encountered issues with some test classes generated by 3.x protobuf libraries.

Compilation issues:

Compiling 14 source files to /home/runner/work/core/core/target/kpow-enterprise/classes
/home/runner/work/core/core/modules/kpow/src-java-dev/factorhouse/serdes/MyRecordOuterClass.java:129: error: cannot find symbol
        makeExtensionsImmutable();
        ^
  symbol:   method makeExtensionsImmutable()
  location: class MyRecord

Runtime issues:

Bad type on operand stack
Exception Details:
Location:
io/confluent/kafka/schemaregistry/protobuf/ProtobufSchema.toMessage(Lcom/google/protobuf/DescriptorProtos$FileDescriptorProto;Lcom/google/protobuf/DescriptorProtos$DescriptorProto;)Lcom/squareup/wire/schema/internal/parser/MessageElement; : invokestatic
Reason:
Type 'com/google/protobuf/DescriptorProtos$MessageOptions' (current frame, stack[1]) is not assignable to 'com/google/protobuf/GeneratedMessage$ExtendableMessage'
Current Frame:
bci:
flags: { }
locals: { 'com/google/protobuf/DescriptorProtos$FileDescriptorProto', 'com/google/protobuf/DescriptorProtos$DescriptorProto', 'java/lang/String', 'com/google/common/collect/ImmutableList$Builder', 'com/google/common/collect/ImmutableList$Builder', 'com/google/common/collect/ImmutableList$Builder', 'com/google/common/collect/ImmutableList$Builder', 'java/util/LinkedHashMap', 'java/util/LinkedHashMap', 'java/util/List', 'com/google/common/collect/ImmutableList$Builder' }
stack: { 'com/google/common/collect/ImmutableList$Builder', 'com/google/protobuf/DescriptorProtos$MessageOptions' }
Bytecode:
0000000: 2bb6 0334 4db2 0072 1303 352c b903 3703
0000010: 00b8 0159 4eb8 0159 3a04 b801 593a 05b8
0000020: 0159 3a06 bb02 8959 b702 8b3a 07bb 0289

If you encounter these compatibility issues, resolve them by re-generating any generated protobuf classes with protoc v31.1.

Overview

In a distributed system like Apache Kafka, data is partitioned and replicated across multiple brokers to ensure high availability and fault tolerance. A partition is considered an under-replicated partition (URP) when the number of in-sync replicas (ISRs) falls below the configured replication factor. This scenario can arise from various issues, including broker failures, network partitions, or high load on specific brokers.

The presence of URPs is a significant concern as it indicates a degradation in your topics' fault tolerance. If another broker fails before the cluster recovers, you risk permanent data loss. A key challenge in Kafka management is accurately detecting these URPs in real-time, especially during common operational events like a broker failure. Standard monitoring methods can sometimes lag, creating a temporary but dangerous blind spot where a cluster appears healthy even though its resilience has been compromised.

This is precisely the challenge that Kpow's enhanced URP detection is designed to solve. By providing a more accurate and immediate assessment of your cluster's true fault tolerance, this feature delivers significant benefits. It gives you the confidence to act quickly on reliable data, the ability to proactively mitigate risks before they escalate, and ultimately, the power to ensure the resilience and durability of your critical data pipelines.

💡 Enhancement of URP detection is implemented in Release 94.5. For an overview of all the changes, check out the release note: Release 94.5: New Factor House docs, enhanced data inspection, and URP & KRaft improvements.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Enhanced calculation for more accurate health monitoring

Ensuring the fault tolerance of your Kafka clusters requires a precise and accurate count of under-replicated partitions. A challenging but common operational scenario can arise where a broker becomes unavailable, yet the overall cluster health status does not immediately reflect this change. This can mask a critical degradation in data durability, leading operations teams to believe their cluster is healthier than it actually is. Making decisions based on this incomplete information can delay necessary interventions.

To provide a more reliable and trustworthy view, Kpow has enhanced its calculation for under-replicated partitions. Instead of calculating replication status by iterating through each broker—a method that can be incomplete if a broker is offline and unreachable—our new calculation iterates directly through every topic-partition defined in the cluster.

This partition-centric approach provides a more comprehensive and authoritative view of the cluster's state. It is precisely this change that allows the system to correctly detect partitions with fewer in-sync replicas than the configured replication factor, even when brokers are offline and not reported by the AdminClient.

This enhancement ensures that Kpow's health monitoring is a precise reflection of your cluster's real-time condition. It gives you the confidence to trust the metrics you see and act decisively to investigate and resolve replication issues, thereby maintaining a resilient and robust system.

Surfacing URP details in Kpow

This vital health information, now powered by our more accurate calculation, continues to be clearly presented on both the Brokers and Topics pages of the user interface. We've retained this dual perspective as it remains essential for diagnosing problems from different angles—whether you're investigating a single problematic broker or assessing the health of a critical application's topic.

On both pages, summary statistics displays the total number of under-replicated partitions. If this count is greater than zero, it serves as an immediate visual alert. A detailed table automatically appears, listing all affected topics along with their specific URPs. This allows you to quickly identify which topics are at risk and gather the necessary context to restore full replication.

On the Brokers Page:

On the Topics Page:

To further strengthen monitoring and alerting capabilities, new Prometheus metrics have been introduced to track under-replicated partitions. These metrics integrate seamlessly with your existing observability stack and provide more granular insights for automated alerting and historical trend analysis:

• broker_urp: The total number of under replicated topic partitions belonging to this broker.
• topic_urp: The total number of under replicated partitions belonging to this topic.
• topic_urp_total: The total number of under replicated partitions of all topics in the Kafka cluster.

Conclusion

Accurate and timely detection of under-replicated partitions (URPs) is fundamental to maintaining a resilient and reliable Apache Kafka cluster. With its enhanced calculation, Kpow provides a more precise and immediate understanding of your cluster's health by correctly identifying replication issues, particularly in scenarios involving broker failures. This enhanced detection, combined with detailed visibility in the Kpow UI and new Prometheus metrics for automated alerting, empowers you to proactively address replication issues, mitigate the risk of data loss, and ensure the continuous high performance of your real-time data pipelines. This feature update reaffirms Kpow's commitment to providing comprehensive and intuitive tooling for Kafka management and monitoring.

Overview

We are excited to announce the launch of our new, unified documentation site: Factor House Docs. This new hub brings all our product documentation together into a single, streamlined resource, making it easier than ever to find the information you need.

Previously, our documentation was split between community and enterprise editions. Now, with Factor House Docs, all of that content is in one place with a simplified structure. This move is designed to provide a more cohesive and user-friendly experience for everyone.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Key hightlights

Unified product content

No more switching between different sites for community and enterprise documentation. All documentation for Kpow, Flex, and the upcoming Factor Platform is now in one centralized location. This consolidation simplifies the user experience by providing a single source of truth for all product-related information, regardless of your edition.

Clear feature availability

To help you quickly identify which features are available in your edition, we've introduced clear COMMUNITY, TEAM, and ENTERPRISE badges throughout the documentation. This makes it easy to understand the capabilities of your current plan and see what additional features are available if you choose to upgrade.

Improved organization

One of the most impactful changes in the new Factor House Docs is the move from a dense, reference-style list to a clear, task-oriented structure. This isn't just a redesign; it's a fundamental improvement in how you find and use our documentation.

A good example of this is how we've consolidated administrative features.

In the previous documentation, getting a complete view of your administrative capabilities required you to hunt through completely different sections. Core admin controls like Staged mutations and Temporary policies were buried under the User authorisation section. Meanwhile, other powerful administrative tools like Bulk actions and Data governance were located in the general Features list, mixed in with dozens of other unrelated items. To perform your duties, you had to already know which features we considered part of "authorization" and which were just "features."

The new documentation eliminates this guesswork by introducing a single, dedicated Administrative workflows section.

Content is now organized to reflect a natural user journey:

• A clear onboarding path: Instead of a jumble of topics, you are now greeted with a clear, sequential path: Getting started, followed by dedicated Installation and Configuration sections. This guides you logically from initial setup to a fully running instance.
• Topic-based grouping: The monolithic "Features" section is gone. Related capabilities are now grouped into intuitive, high-level categories. For instance, powerful data tools are now consolidated under the Query languages & Data management section. This creates a focused hub for anyone responsible for the data itself, bringing together everything from querying with kJQ to inspecting topics.
• Enhanced discoverability: This new structure makes it much easier to discover the full range of Kpow's capabilities. By browsing top-level sections like Integration & Kafka management, you can quickly get a sense of the available tools without having to read through a long, unstructured list.

This streamlined layout is designed to get you to your answers faster, reducing search time and making the entire platform more accessible and easier to master.

Powerful search, instant answers

Finding the exact information you need is now faster and easier than ever. Using Algolia DocSearch, our documentation site delivers instant, relevant results across all sections. No more navigating multiple pages to locate a specific configuration property or function.

Whether you’re looking for an environment variable, a kJQ example, or details on Role-Based Access Control, the global search guides you directly to the content you need. This improvement ensures you spend less time searching and more time building.

Interactive kJQ examples

Our new documentation includes a dedicated section for fully interactive kJQ examples. You can now modify both the kJQ filter and the JSON data in real-time to see how different filters work and experiment with the syntax. This hands-on approach allows you to test everything from basic filters and function calls to complex, nested logic right in your browser.

Hands-on playground

To further enhance your learning experience, we've added a new "Playground" section. This area features a collection of labs and projects designed to help you explore the full range of product capabilities in a hands-on environment.

Ready for the future

The new Factor House Docs site is built to be a long-lasting resource. The documentation for the new Factor Platform will be added and expanded upon its release, ensuring that this hub will remain the most up-to-date and comprehensive resource for all product information now and in the future.

Conclusion

We are confident that the new Factor House Docs will be a valuable resource for our users. We encourage you to explore the new site and update your bookmarks. The previous documentation site will be retired.

‍

Overview

Following our recent improvements in Kpow release 94.3, which introduced AI-powered filtering and automatic deserialization, release 94.5 continues to enhance the data inspect experience. This latest version introduces several key additions, including comma-separated kJQ Projection expressions, in-browser search, multiple deserialization options, and attribute sorting. It also features a significant optimization with high-performance streaming and an enhancement to our query language with expanded kJQ capabilities. You can try out the new transforms and functions on the new interactive examples page on our Factor House docs. These updates provide more granular control and deeper insights into your Kafka topics, empowering developers to navigate and analyze their data with greater efficiency and precision.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

Targeted data views with comma-separated kJQ Projection expressions

Kpow 94.5 introduces support for comma-separated projection expressions in kJQ, such as .value.base, .value.rates. This powerful feature allows you to extract multiple fields from Kafka records in a single query. Now you can create targeted data views without unnecessary information, streamlining your workflow and keeping your output clean. This functionality is available for both key and value sub-paths, offering greater flexibility in how you inspect your data.

Faster navigation with in-browser search

You can now use your browser's native search functionality (Ctrl + F) to quickly find records by JSON path or value when using kJQ filters. This eliminates the need to re-run queries, saving you time and effort. The results component is fully keyboard-friendly, adhering to the Listbox pattern for improved accessibility. This ensures a smoother and more predictable navigation experience for all users, including those who rely on screen readers.

Deeper insights into schemas and deserialization

Data Inspect now offers detailed schema metadata for each message, including schema IDs and deserializer types. This makes it easier to identify misaligned schemas and poison messages. To handle these problematic records, Kpow 94.5 provides several deserialization options:

• Drop record (default): This option ignores erroneous records, displaying only the well-formatted ones.
• Retain record: This includes both well-formatted and erroneous records. Problematic records are flagged with a 'Deserialization exception' message instead of displaying the raw, poisonous value.
• Poison only: This option displays only the erroneous records, with the value recorded as 'Deserialization exception'.

Improved readability with attribute sorting

When you select the 'Pretty printed (sorted)' display option, the attributes of the key or value are now sorted alphabetically by name. This improves the readability and consistency of your data during inspection, making it easier to compare records and identify specific fields.

High-performance streaming for large datasets

Kpow 94.5 is optimized for performance, allowing you to stream over 500,000 records without any UI lag. This enables the efficient analysis of large datasets, so you can work with high-volume topics without compromising on speed or responsiveness.

Expanded kJQ capabilities with new transforms and functions

The kJQ language has been significantly expanded with a host of new transforms, including parse-json, floor, ceil, upper-case, lower-case, trim, ltrim, rtrim, reverse, sort, unique, first, last, keys, values, is-empty, and flatten. Additionally, new functions such as within, split, and join have been added to enable more complex data manipulation directly within your kJQ queries.

For more details on these new features, please refer to the updated kJQ manual. Also, be sure to visit the new interactive examples page on our new Factor House docs site—it's a great way to quickly verify your kJQ queries.

Conclusion

Kpow 94.5 builds upon the foundation of previous releases to deliver a more powerful and user-friendly data inspection experience. The latest additions—including kJQ Projection expressions, in-browser search, and flexible deserialization options—provide more granular control over your data. Together with high-performance streaming and expanded kJQ capabilities, these enhancements solidify Kpow as an essential tool for any developer working with Apache Kafka. Be sure to test out the new kJQ features on the interactive examples page on our Factor House docs. Ultimately, this release is designed to streamline your workflows, reduce debugging time, and empower you to unlock the full potential of your data.

‍

At Factor House, we’re always tracking what’s new in the Apache Kafka ecosystem, both for our own products and to support the growing community of stream-first developers we work with every day. Here’s a quick rundown of what’s exciting in the 4.1.0 release.

Queues for Kafka move to preview (KIP-932)

After a long wait, Kafka’s new native queueing model is moving from Early Access to Preview. That means KIP-932, which introduces Share Consumer Groups, is stabilising. And while it’s not yet production-ready, it’s getting much closer.

So what does this mean? In short:

• It brings queue-like semantics to Kafka, enabling multiple consumers to read from the same partition in parallel.
• It supports out-of-order processing, with individual message acknowledgements and retries.
• It introduces a more flexible model for building event-driven architectures that straddle the line between pub/sub and traditional messaging queues.

This is a big shift for teams building scalable consumer architectures, and one we’re watching very closely.

Kafka Streams gets smarter with Stream Groups (KIP-1071)

Kafka Streams applications just got a coordination upgrade.

KIP-1071 introduces a new rebalance protocol for Streams apps, based on KIP-848’s consumer group protocol. This update:

• Streamlines how stream tasks are assigned and rebalanced
• Makes scaling Kafka Streams applications smoother
• Adds transparency and predictability to the rebalance process

If you’ve ever wrangled a cluster of Kafka Streams apps and found yourself wondering “why did that rebalance happen?”, this one’s for you.

Other noteworthy improvements

A few other highlights from the release that caught our eye:

• KIP-877: A standardised API for plugin metrics — more visibility into Kafka internals, especially custom components.
• KIP-891: Kafka Connect now supports multiple plugin versions, making upgrades and rollbacks less painful.
• KIP-1050: Improved error handling for Transactional Producers, with clear exception categories that should simplify recovery strategies.
• KIP-1139: Adds support for JWT Bearer OAuth 2.0, making it easier to manage secure access without static secrets.

All told, Kafka 4.1 includes contributions from 167 engineers across the globe. That's a testament to the strength and growth of the open source streaming community.

Our take at Factor House

“At Factor House, we’re already preparing to update our clients to Kafka 4.x in an upcoming product release. Features like improved plugin management and transactional error clarity are going to make life easier for developers, and we’re excited about what the queueing model means for the future of real-time stream consumption.” — Derek Troy-West, CEO, Factor House

Whether you’re running Kafka locally or at scale in production, Kafka 4.1 is a milestone release that makes the platform more powerful, more flexible, and more secure.

We’ll be diving deeper into some of these changes in future blog posts, particularly around how they affect real-world streaming workloads using tools like Kafka Connect, Kafka Streams, and our own product stack at Factor House.

Resources

• Official Apache Kafka 4.1 Release Announcement
• Apache Kafka 4.1 Release Notes
• Apache Kafka 4.1 Documentation

On July 31st, we welcomed the Melbourne data streaming community to our Factor House HQ in Northcote for an evening of real-time insights, local dev tools, and excellent company, cohosted with our friends at Confluent.

The event, From Real-Time Data to Insights & Local Development with Kafka and Flink, was packed with engineers, data practitioners, and curious minds looking to deepen their knowledge of stream processing and hands-on development workflows.

Pizza, drinks, and chatter kicked things off, and the vibes didn't disappoint. The crowd was buzzing with conversation, from real-time architectures to favorite CLI tools.

Talks That Delivered

Olena Kutsenko, Staff Developer Advocate at Confluent

See Olena's presentation

Olena Kutsenko, Staff Developer Advocate at Confluent, took the stage first with The Art of Structuring Real-time Data Streams into Actionable Insights. Her talk offered a compelling walkthrough of the Kafka–Flink–Iceberg stack, showing how to tame messy real-time data and prepare it for analytics and AI. Attendees praised her clarity and technical depth:

“Olena managed to communicate her ideas in a way that was comprehensible to non-technical people, whilst still articulating the value for more experienced attendees.”

Jaehyeon Kim, Developer Experience Engineer at Factor House

Read Jae's blog post about Factor House Local & Labs

Next up, Jaehyeon Kim, Developer Experience Engineer at Factor House, gave a tour of Hands-On Local Development: Kafka, Flink, Docker, and more. Jae introduced Factor House Labs — a collection of open-source Docker environments designed to get engineers experimenting with modern data platforms fast.

“Jae’s passion for the product was infectious. His demo showed just how easy it is to spin up a full end-to-end data pipeline locally.”

Community, Connection, and What’s Next

Beyond the talks, a big highlight was the crowd itself. We saw first-timers, veterans, and everyone in between sharing ideas and chatting about use cases.

This meetup also marked the launch of the Factor House Community on Slack, a space to keep the conversation going, swap tips, and collaborate on all things real-time.

Next Stop: Sydney!

Couldn’t make it to Melbourne? This meetup will happen again in Sydney on September 16. Sydney Apache Kafka x Flink Meetup

Stay Connected

Want to hear about future events, tools, and hands-on learning experiences?

Join the Factor House Community on Slack

Follow Factor House on LinkedIn