Come and #say-hi

We've opened the Factor House Community Slack: a public space for anyone working in the streaming data world. Whether you're a seasoned data engineer, exploring streaming technologies for the first time, or just curious about real-time systems, this is your place to connect, learn, and share.

Join the Factor House Community Slack →

Our community isn’t just a chatroom, it’s part of building the collective intelligence that will shape the future of data in motion. We’re connecting the engineers, operators, and curious learners who will define what streaming infrastructure looks like in the AI era.

What changed (and why)

For years, our team, customers, and broader community were all in one busy workspace. It was a good start, but as our products and community grew, so did the noise. Support conversations were mixed with team chatter and community discussions, leaving newcomers unsure of where to begin.

We've split into two focused spaces:

• Factor House Community (public) - Where the magic happens
• Private workspace - For our team and client engagements

To our existing customers: thank you for your patience during this transition. Your feedback helps us build something better, and we're grateful for engineers who push us to improve.

Why join?

• Current users: Get faster answers from real humans who've solved similar problems. Our team is active daily on weekdays (Australian timezone), and we're building a community that helps each other.
• Newcomers: This is your friendly on-ramp to the data streaming world. Ask the "obvious" questions - we love helping engineers grow.
• Ecosystem: We're vendor-neutral and open-source friendly. Discuss any tools, share knowledge, and make announcements. The more diverse perspectives, the better.

Here’s where you’ll find us (and each other):

• #say-hi: introduce yourself to the community
• #getting-started: new to Kpow or Flex? Begin your journey here
• #ask-anything: all questions welcome, big or small
• #product-kpow & #product-flex: features, releases, and best practices for FH tooling
• #house-party: off-topic bants, memes, and pet pics

Our team is in the mix too. You’ll spot us by the Factor House logo in our status.

Community Guidelines

We want this community to reflect the best parts of engineering culture: openness, generosity, and curiosity. It’s not just about solving problems faster, it’s about building a place where people can do their best work together.

This is a friendly, moderated space. We ask everyone to be respectful and inclusive (read our code of conduct). Keep conversations in public channels wherever possible so everyone benefits.

For our customers: use your dedicated support channels or support@factorhouse.io for SLA-bound requests and bug reports. The community Slack is best-effort support.

Ready to Join?

This Slack is the seed of a wider ecosystem. It's a place where engineers share knowledge, swap stories, and push the boundaries of what’s possible with streaming data. It’s the beginning of a developer community that will grow alongside our platform.

This community will become what we make of it. We're hoping for technical discussions, mutual help, and the kind of engineering conversations that make your day better.

Join the Factor House Community Slack →

Come #say-hi and tell us what you're working on. We're genuinely curious about what keeps data engineers busy.

Factor House Local is a collection of pre-configured Docker Compose environments that demonstrate modern data platform architectures. Each setup is purpose-built around a specific use case and incorporates widely adopted technologies such as Kafka, Flink, Spark, Iceberg, and Pinot. These environments are further enhanced by enterprise-grade tools from Factor House: Kpow, for Kafka management and control, and Flex, for seamless integration with Flink.

Data Stack

Kafka Development & Monitoring with Kpow

This stack provides a comprehensive, locally deployable Apache Kafka environment designed for robust development, testing, and operations. It utilizes Confluent Platform components, featuring a high-availability 3-node Kafka cluster, Zookeeper, Schema Registry for data governance, and Kafka Connect for data integration.

The centerpiece of the stack is Kpow (by Factorhouse), an enterprise-grade management and observability toolkit. Kpow offers a powerful web UI that provides deep visibility into brokers, topics, and consumer groups. Key features include real-time monitoring, advanced data inspection using kJQ (allowing complex queries across various data formats like Avro and Protobuf), and management of Schema Registry and Kafka Connect. Kpow also adds critical enterprise features such as Role-Based Access Control (RBAC), data masking/redaction for sensitive information, and audit logging.

Ideal For: Building and testing microservices, managing data integration pipelines, troubleshooting Kafka issues, and enforcing data governance in event-driven architectures.

Unified Analytics Platform (Flex, Flink, Spark, Iceberg & Hive Metastore)

This architecture establishes a modern Data Lakehouse that seamlessly integrates real-time stream processing and large-scale batch analytics. It eliminates data silos by allowing both Apache Flink (for streaming) and Apache Spark (for batch) to operate on the same data.

The foundation is built on Apache Iceberg tables stored in MinIO (S3-compatible storage), providing ACID transactions and schema evolution. A Hive Metastore, backed by PostgreSQL, acts as the unified catalog for both Flink and Spark. The PostgreSQL instance is also configured for Change Data Capture (CDC), enabling real-time synchronization from transactional databases into the lakehouse.

The stack includes Flex (by Factorhouse), an enterprise toolkit for managing and monitoring Apache Flink, offering enhanced security, multi-tenancy, and deep insights into Flink jobs. A Flink SQL Gateway is also included for interactive queries on live data streams.

Ideal For: Unified batch and stream analytics, real-time ETL, CDC pipelines from operational databases, fraud detection, and interactive self-service analytics on a single source of truth.

Apache Pinot Real-Time OLAP Cluster

This stack deploys the core components of Apache Pinot, a distributed OLAP (Online Analytical Processing) datastore specifically engineered for ultra-low-latency analytics at high throughput. Pinot is designed to ingest data from both batch sources (like S3) and streaming sources (like Kafka) and serve analytical queries with millisecond response times.

Ideal For: Powering real-time, interactive dashboards; user-facing analytics embedded within applications (where immediate feedback is crucial); anomaly detection; and rapid A/B testing analysis.

Centralized Observability & Data Lineage

This stack provides a complete solution for understanding both system health and data provenance. It combines Marquez, the reference implementation of the OpenLineage standard, with the industry-standard monitoring suite of Prometheus, Grafana, and Alertmanager.

At its core, OpenLineage enables automated data lineage tracking for Kafka, Flink, and Spark workloads by providing a standardized API for emitting metadata about jobs and datasets. Marquez consumes these events to build a living, interactive map of your data ecosystem. This allows you to trace how datasets are created and consumed, making it invaluable for impact analysis and debugging. The Prometheus stack complements this by collecting time-series metrics from all applications, visualizing them in Grafana dashboards, and using Alertmanager to send proactive notifications about potential system issues.

Ideal For: Tracking data provenance, performing root cause analysis for data quality issues, monitoring the performance of the entire data platform, and providing a unified view of both data lineage and system health.

Factor House Local Labs

The Factor House Local labs are a series of 12 hands-on tutorials designed to guide developers through building real-time data pipelines and analytics systems. The labs use a common dataset of orders from a Kafka topic to demonstrate a complete, end-to-end workflow, from data ingestion to real-time analytics.

The labs are organized around a few key themes:

💧 Lab 1 - Streaming with Confidence:

• Learn to produce and consume Avro data using Schema Registry. This lab helps you ensure data integrity and build robust, schema-aware Kafka streams.

🔗 Lab 2 - Building Data Pipelines with Kafka Connect:

• Discover the power of Kafka Connect! This lab shows you how to stream data from sources to sinks (e.g., databases, files) efficiently, often without writing a single line of code.

🧠 Labs 3, 4, 5 - From Events to Insights:

• Unlock the potential of your event streams! Dive into building real-time analytics applications using powerful stream processing techniques. You'll work on transforming raw data into actionable intelligence.

🏞️ Labs 6, 7, 8, 9, 10 - Streaming to the Data Lake:

• Build modern data lake foundations. These labs guide you through ingesting Kafka data into highly efficient and queryable formats like Parquet and Apache Iceberg, setting the stage for powerful batch and ad-hoc analytics.

💡 Labs 11, 12 - Bringing Real-Time Analytics to Life:

• See your data in motion! You'll construct reactive client applications and dashboards that respond to live data streams, providing immediate insights and visualizations.

Overall, the labs provide a practical, production-inspired journey, showing how to leverage Kafka, Flink, Spark, Iceberg, and Pinot together to build sophisticated, real-time data platforms.

Conclusion

Factor House Local is more than just a collection of Docker containers; it represents a holistic learning and development ecosystem for modern data engineering.

The pre-configured stacks serve as the ready-to-use "what," providing the foundational architecture for today's data platforms. The hands-on labs provide the practical "how," guiding users step-by-step through building real-world data pipelines that solve concrete problems.

By bridging the gap between event streaming (Kafka), large-scale processing (Flink, Spark), modern data storage (Iceberg), and low-latency analytics (Pinot), Factor House Local demystifies the complexity of building integrated data systems. Furthermore, the inclusion of enterprise-grade tools like Kpow and Flex demonstrates how to operate these systems with the observability, control, and security required for production environments.

Whether you are a developer looking to learn new technologies, an architect prototyping a new design, or a team building the foundation for your next data product, Factor House Local provides the ideal starting point to accelerate your journey.

Overview

Kpow's Data Inspect feature has always been a cornerstone for developers working with Apache Kafka, offering a powerful way to query and understand topic data, as introduced in our earlier guide on how to query a Kafka topic.

The 94.3 release dramatically enhances this experience by introducing a suite of intelligent and user-friendly upgrades. This release focuses on making data inspection more accessible for all users while adding even more power for advanced use cases. The key highlights include AI-powered message filtering, which allows you to query Kafka using plain English; automatic deserialization, which removes the guesswork when dealing with unknown data formats; and significant enhancements to the kJQ language itself, providing more flexible and powerful filtering capabilities.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

AI-Powered Message Filtering

Kpow now supports the integration of external AI models to enhance its capabilities, most notably through its "bring your own" (BYO) AI model functionality. This allows you to connect Kpow with various AI providers to power features within the platform.

AI Model Configuration

You have the flexibility to configure one or more AI model providers. Within your Kpow user preferences, you can then set a default model for all AI-assisted tasks. Configuration is managed through environment variables and is supported for the following providers:

If you need support for a different AI provider, you can contact the Factor House support team.

Enhanced AI Features

The primary AI-driven feature is the kJQ filter generation. This powerful tool enables you to query Kafka topics using natural language. Instead of writing complex kJQ expressions, you can simply describe the data you're looking for in plain English.

Here's how it works:

• Natural Language Processing: The system converts your conversational prompts (e.g., "show me all orders over $100 from the last hour") into precise kJQ filter expressions.
• Schema-Awareness: To improve accuracy, the AI can optionally use the schemas of your Kafka topics to understand field names, data types, and the overall structure of your data.
• Built-in Validation: Every filter generated by the AI is automatically checked against Kpow's kJQ engine to ensure it is syntactically correct before you run it.

This feature is accessible from the Data Inspect view for any topic. After the AI generates a filter, you have the option to execute it immediately, modify it for more specific needs, or save it for later use. For best results, it is recommended to provide specific and actionable descriptions in your natural language queries.

It is important to be mindful that AI-generated filters are probabilistic and may not always be perfect. Additionally, when using cloud-based AI providers, your data will be processed by them, so for sensitive information, using local models via Ollama or enterprise-grade AI services with strong privacy guarantees is recommended.

For more details, see the AI Models documentation.

Automatic Deserialization

Kpow simplifies data inspection with its "Auto SerDes" feature. In the Data Inspect view, you can select "Auto" as the deserializer, and Kpow will analyze the raw data to determine its format (like JSON, Avro, etc.) and decode it for you. This is especially useful in several scenarios, including:

• When you are exploring unfamiliar topics for the first time.
• While working with topics that may contain mixed or inconsistent data formats.
• When debugging serialization problems across different environments.
• For onboarding new team members who need to get up to speed on topic data quickly.

To make these findings permanent, you can enable the Topic SerDes Observation job by setting INFER_TOPIC_SERDES=true. When active, this job saves the automatically detected deserializer settings and any associated schema IDs, making them visible and persistent in the Kpow UI for future reference.

kJQ Language Enhancements

In response to our customers' evolving filtering needs, we've significantly improved the kJQ language to make Kafka record filtering more powerful and flexible. Check out the updated kJQ filters documentation for full details.

Below are some highlights of the improvements:

Chained alternatives

Selects the first non-null email address and checks if it ends with ".com":

.value.primary_email // .value.secondary_email // .value.contact_email | endswith(".com")

String/Array slices

Matches where the first 3 characters of transaction_id equal TXN:

.value.transaction_id[0:3] == "TXN"

For example, { "transaction_id": "TXN12345" } matches, while { "transaction_id": "ORD12345" } does not

UUID type support

kJQ supports UUID types out of the box, including the UUID deserializer, AVRO + logical types, or Transit / JSON and EDN deserializers that have richer data types.

To compare against literal UUID strings, prefix them with #uuid to coerce into a UUID:

.key == #uuid "fc1ba6a8-6d77-46a0-b9cf-277b6d355fa6"

Conclusion

The 94.3 release marks a significant leap forward for data exploration in Kpow. By integrating AI for natural language queries, automating the complexities of deserialization, and enriching the kJQ language with advanced functions, Kpow now caters to an even broader range of users. These updates streamline workflows for everyone, from new team members who can now inspect topics without prior knowledge of data formats, to seasoned engineers who can craft more sophisticated and precise queries than ever before. This release reaffirms our commitment to simplifying the complexities of Apache Kafka and empowering teams to unlock the full potential of their data with ease and efficiency.

Introduction: What’s Changing?

The EU Data Act, effective as of September 12, 2025, introduces new requirements for data access, sharing, and interoperability across the European Union's single market. For engineers managing data streaming platforms like Apache Kafka, this means re-evaluating how your stack handles user data, security, provider switching, and compliance with both the Data Act and GDPR.

What Does the EU Data Act Mean for Data Streaming?

At its core, the Data Act requires that users of connected products and services have access to their raw usage data and metadata. This means that streaming platforms processing such data need to enable seamless and secure access and sharing capabilities. Additionally, the Act outlines rules for business-to-business and business-to-government data sharing, while safeguarding trade secrets and ensuring the security of data.

Key highlights of the Act

Why These Changes Matter for Kafka Engineers

Kafka is at the heart of many modern data streaming architectures, making it critical to align Kafka operations with these new legal requirements. Engineers must ensure:

• Data streams can be accessed and shared securely and transparently.
• Systems support interoperability to facilitate provider switching without data loss or downtime.
• Access controls and auditing are in place to protect sensitive data and demonstrate compliance.
• Data subject rights under GDPR, such as access, correction, and deletion, can be fulfilled efficiently.

How Kpow Supports Compliance and Operational Excellence

At Factor House, we designed Kpow with these challenges in mind. As an enterprise-native company, our flagship solution Kpow is a comprehensive Kafka management and monitoring platform that empowers engineers to meet the demands of the new EU data landscape. Right out of the box, Kpow enables engineers to meet stringent EU Data Act requirements with ease.

Practical Steps for Engineers

• Review your current Kafka stack: Ensure configurations support data access, portability, and interoperability.
• Implement robust authentication and RBAC: Protect sensitive streams and support GDPR compliance.
• Enable detailed audit logging: Prepare for regulatory audits and internal monitoring.
• Test provider switching: Validate that you can migrate workloads without disruption or extra costs.
• Stay updated: Monitor regulatory updates and best practices for ongoing compliance.

Access "Turnkey" Compliance with Kpow

Kpow’s secure, transparent, and flexible Kafka management capabilities align with the EU Data Act’s requirements, enabling controlled data access, robust security, local data storage, auditability, and interoperability. This makes it an effective tool for data streaming engineers and organizations aiming to comply with the EU’s new data sharing and protection rules starting September 2025.

Future-Proof Your Kafka Streaming

The EU Data Act is reshaping how data streaming services operate in Europe. Ensuring your Kafka infrastructure is compliant and resilient is no longer optional—it’s essential.

To help you navigate this transition, Factor House offers a free 30-day fully-featured trial license of Kpow.

Experience firsthand how Kpow’s secure, transparent, and flexible Kafka management capabilities can simplify compliance and enhance your streaming operations. Start your free trial of Kpow today.

Sources (and further information)

1. Data Act explained, European Commission, accessed 12th June 2025
2. The EU Data Act: What does it mean for you?, Deloitte, accessed 12th June 2025
3. EU Data Act Imposes New Data Sharing Obligations, The WSGR Data Advisor, accessed 12th June 2025
4. Kafka Compliance, Scalar Topics, accessed 12th June 2025

The transition from Kafka Summit to Current is now complete, with this year's London conference now rebranded to match Confluent’s US and India events and providing a strong indicator that the real-time ecosystem now extends far beyond Apache Kafka. With more than 2,200 attendees, in-depth technical presentations, and countless exhibitor hall discussions, it is evident that real-time streaming is here to stay and the ecosystem is evolving quickly, branching out into AI-native systems, multi-technology stacks, and production-grade stream processing.

What We Saw

For the third consecutive year, Factor House was on the ground in London as a Silver Sponsor, running product demos, meeting clients, and, more importantly, learning about the needs of engineers managing complex deployments, platform teams integrating Flink and Kafka, and architects exploring AI built on live data.

While the Kafka Summit name has been replaced, Kafka remains a foundational technology. However, attention is shifting to system-level control, end-to-end observability, and tools that reduce operational friction without sacrificing power. We’re focused on that space with Kpow for Kafka, Flex for Flink, and, soon, the Factor Platform.

Key Signals

1. AI Is Going Event-Driven - But Engineers Remain Cautious
2. Streaming-first AI was a recurring theme at Current. Sessions like "Flink Jobs as Agents" (Airy Inc.) explored how AI agents can interact with a live state, reacting in real time rather than relying on stale snapshots.
3. But several engineers we spoke to flagged concerns.
4. While Kafka and Flink provide the backbone, durable, deterministic, and observable, the idea of introducing autonomous agents into critical pipelines raised eyebrows. There’s excitement, yes, but also scepticism around operational safety, debuggability, and unintended consequences. As one engineer put it:
5. “If an LLM is making decisions in my pipeline, I want to know what it saw, why it acted, and how to stop it fast.”
6. Visibility and control are not optional; they’re the line between innovation and outage. AI might be event-driven, but it’s still infrastructure. And infrastructure needs guardrails.
7. Production Resilience > Architectural Purity
8. Sessions from OpenAI, AWS, and Daimler all emphasized pragmatism. OpenAI’s Changing Engines Mid-Flight offered real lessons on handling Kafka migrations under load. Elegant designs are great, but shipping reliable systems matters most.
9. Flink Is Now a First-Class Citizen
10. Flink moved from curiosity to cornerstone. Teams from ShareChat, Wix, and Pinterest shared how they reduced latency and costs while simplifying their pipelines. However, Flink remains operationally raw; hence, Flex, our UI and API, is designed to make Flink observable and manageable in real production environments.

Noteworthy Tools

We saw an uptick in focused tools solving specific friction points, some standouts for us:

• ShadowTraffic – Safe, controlled Kafka traffic simulation.
• RisingWave – Real-time SQL queries over Kafka streams.
• Gravitee – Fine-grained Kafka API access control.
• Imply – Sub-second dashboards on live data.
• Snowplow – Clean, structured enrichment pipelines for streaming events.

Where Factor House Fits

As complexity grows and streaming intersects with AI, teams need visibility, safety, and efficiency, not more abstraction. Our upcoming Factor Platform unifies Kpow and Flex into a single control plane for data in motion, enabling teams to manage Kafka and Flink with confidence across clusters, clouds, and regions, and providing a layer of clarity across an organizations complete streaming ecosystem.

If you’d like to learn more about Factor House products book a demo today.

Introduction

Reagent is a popular library that enables ClojureScript developers to write clean and concise React components using simple Clojure data structures and functions. Reagent is commonly used with re-frame, a simlarly popular ClojureScript UI library.

Both libraries have been fundamental to how we build modern, sophisticated, accessible UI/UX at Factor House over the past seven years.

Now, after 121 product releases, we have replaced Reagent and re-frame in our products with two new libraries:

• HSX: a Hiccup-to-React compiler that lets us write components the way we always have, but produces pure React function components under the hood.
• RFX: a re-frame-inspired subscription and event system built entirely on React hooks and context.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.

The Problem

Our front-end tech stack was starting to accumulate technical debt, not from the usual entropy of growing software, but from the slow diversion of Reagent from the underlying Javascript library that it leverages - React.

In many ways, Reagent was ahead of its time. Simple state primitives (the Ratom), function components, and even batched updates for state changes were innovations Reagent offered well before React itself. It provided a remarkably elegant abstraction that made building UIs in ClojureScript a joy.

But it’s now 2025. React has caught up and in many areas surpassed those early innovations. Hooks offer state management. Concurrent rendering and built-in batched updates are first-class features. While it took React a decade to reach this point, the landscape has undeniably shifted.

Unfortunately, Reagent hasn’t kept pace. Its internals are built around class-based components and are increasingly at odds with React's architecture. Most critically for us, Reagent is fundamentally incompatible with React 19’s new internals.

This incompatibility created serious technical debt for us at Factor House. More and more of our vital front-end dependencies, from libraries for virtualized lists to accessible UI components, are starting to require React 18 or 19. Without a way forward, we risked stagnation.

However, we deeply value what Reagent and re-frame gave us - a simple, expressive syntax based on Hiccup, and a clean event-driven model. We didn’t want to abandon these strengths. Instead, we chose to move forward by building new libraries, ones that preserve the spirit of Reagent and re-frame and modernize their foundations to align with today's React.

In this post, we'll walk you through why we had to move beyond Reagent and re-frame, how we built new libraries to modernize our stack, and the real-world outcomes of embracing React 19's capabilities.

The Migration Challenge

Our goal wasn’t to rewrite our entire front-end stack, but to modernize it. That meant preserving two things that serve us well:

• The ability to write React components using Hiccup-style markup, which we now call HSX.
• Continued use of re-frame’s event and subscription model.

At the same time, we wanted to align ourselves much more closely with React’s internals. We were ready to fully embrace idiomatic React. That meant we were happy to let go of:

• Ratoms — in favor of React’s native useState, useEffect, and useReducer primitives.
• Class-based components — which are no longer relevant in a hooks-first React world.

Choosing to move away from Reagent’s internals, especially Ratoms, was not a loss. To us Ratoms were always an implementation detail. Since we already manage app state through re-frame subscriptions, local component state was minimal.

So the real migration challenge became this:

Could we capture the spirit of Reagent and re-frame — using nothing but React itself?

And if we could, would the resulting behavior and performance match (or exceed) what we had before?

With these in hand, we were ready to test them where it matters most: against our real-world products. Both Kpow for Apache Kafka and Flex for Apache Flink are complex, enterprise-grade applications. Could HSX and RFX support them without regressions? Could we maintain backward compatibility, migrate incrementally, and still unlock the benefits of React 19?

These were the questions we set out to answer, and as we’ll see, they led to some surprising and exciting results.

Migrating Kpow and Flex

We began by sketching out minimal viable implementations of HSX and RFX — enough to prove the migration path could work.

HSX: Building a Modern Hiccup-to-React Layer

For HSX, the first goal was essentially to reimplement the behavior of reagent.core/as-element. We required:

• The same props and tag SerDes logic as Reagent.
• Special operators like Fragments (:<>) and direct React component invocation (:>).
• Memoization semantics — essentially replicating Reagent’s implicit shouldComponentUpdate optimization.

This would allow us to preserve the developer experience of writing Hiccup-style components while outputting React function components under the hood.

RFX: Reimagining re-frame on Pure React Foundations

Migrating re-frame was more challenging because of its much larger API surface area. We needed to implement:

• Subscriptions, events, coeffects, effects — the full re-frame public API.
• A global state store compatible with React Hooks.
• A queuing system for efficiently processing dispatched events

Implementing functions like reg-event-db and subscribe was straightforward. The bigger challenge was syncing global state changes into the React UI without relying on Ratoms and 'reactions'.

To solve this, we initially deferred a custom solution and instead leaned on a battle-tested JavaScript library: Zustand.

For event queuing, we adapted re-frame’s own FIFO router, which was pleasantly decoupled from Reagent internals and easily portable.

First Steps in Production: Tweaking Kpow and Flex

With early versions of HSX and RFX in hand, we moved quickly to integrate them into our products. The migration required surprisingly few code changes at the application level:

• Replacing reagent.core/atom with react/useState where needed (thankfully very few places).
• Replacing reagent.core/as-element calls with io.factorhouse.hsx.core/create-element.
• Replacing react/createRef calls with react/useRef.
• Updating the entry points to use the react-dom/client API (createRoot) instead of the legacy render method.
• Introducing a re-frame.core shim namespace over RFX, mapping 1:1 to the re-frame public API and requiring no migration for event handlers or subscriptions!

With these adjustments in place, and some rapid iteration on HSX and RFX, we were able to compile and run Kpow (our larger application at ~60,000 lines of ClojureScript) entirely on top of React 19!

The first results were rough: performance was poor and some pages failed to render correctly.

But critically, the foundation worked and these early failures became the catalyst for aggressively refining and productionizing our libraries.

Optimizing HSX: Learnings Along the Way

As we moved toward a pure React model, we found ourselves learning a lot more about React’s internals. Sometimes the hard way.

The biggest issue we faced stemmed from React’s reliance on referential equality. In React, referential equality (whether two variables point to the same object in memory) underpins how React identifies components across renders and how it optimizes updates, handles memoization, etc.

This presented a fundamental problem for HSX:

Just like Reagent, HSX creates React elements dynamically at runtime (when io.factorhouse.hsx.core/create-element is called).

Unlike other ClojureScript React templating libraries, we do not rely on Clojure macros to precompile Hiccup into React elements.

We quickly encountered several major symptoms:

• Component memoization failed: React could not track components properly across renders.
• Hook rule violations: Clicking around the app often triggered Hook violations, a sign that React's internal assumptions were being broken.
• Internal React errors: Most concerning was the obscure: Internal React error: Expected static flag was missing.

To understand why, consider a simple example:

(defn my-div-component [props text] 
[:div props text])

HSX compiles this by creating a proxy function component that:

• Maps React’s props object to a Reagent-style function signature.
• Compiles the returned Hiccup ([:div props text]) into a React element via react/createElement.

The problem is that a new intermediate proxy function is created between renders, even if the logic is identical.

React, relying on referential equality, treated each instance as a brand-new component, thus resulting in the above bugs.

The Solution: WeakMap-Based Caching

Our solution was a simple but powerful idea: cache the translated component functions using a JavaScript WeakMap.

• Keys: the user-defined HSX components (e.g., my-div-component), which have stable memory references.
• Values: the compiled React function components.

Using a WeakMap was essential, without it the cache could grow unbounded if components created new anonymous functions every render.

WeakMaps automatically clean up entries when keys (functions) are garbage collected.

However, this approach revealed a secondary problem: Higher-Order Components (HOCs).

The Hidden Trap: Anonymous Functions and HOCs

When users define anonymous functions inside render methods, React treats them as Higher-Order Components.

Example:

(defn my-complex-component [props text]
	(let [inner-component (fn [] [:div props text])]
		[inner-component]))

In this case, inner-component is redefined every render, breaking referential equality, exactly the problem we had just solved. This exact issue is even highlighted in the legacy React docs.

To address this, we added explicit logging and warnings whenever HSX detected HOC-like patterns during compilation.

This forced us to clean up the codebase by refactoring anonymous components into top-level named components.

Unexpectedly, this not only improved correctness but significantly improved performance.

Even when using Reagent previously, anonymous functions inside components had led to unnecessary re-renders, an invisible cost that we were now able to eliminate.

Optimizing RFX: Learnings Along the Way

The challenge with RFX was twofold:

• Without Ratoms, how would we sync application state to the UI efficiently and correctly?
• How could we faithfully reimplement re-frame’s subscription graph, ensuring minimal recomputation when parts of the database change?

Signal Graph: Re-frame’s Core Innovation

In re-frame, subscriptions form a DAG called the signal graph.

Subscriptions can depend on other subscriptions (materialised views), and on each state change, re-frame walks this graph and only recomputes nodes where upstream values have changed.

For example:

;; :foo will update on every app-db change
(reg-sub :foo [db _] (:foo db))
;; :consumes-foo will update only when the value of :foo changes
	(reg-sub :consumes-foo  :<- [:foo]  (fn [foo _]
		(str "Consumed foo")))

In this setup:

• :foo listens directly to the app-db and updates on every change.
• :consumes-foo listens to :foo and only recomputes if :foo’s output changes, not just because the db changed.

This graph-based optimization is a key reason re-frame scales so well even in complex applications.

useSyncExternalStore: The Missing Piece

Fortunately, React 18+ provides a new primitive that fits our needs perfectly: useSyncExternalStore.

This hook allows external data sources to integrate cleanly with React. We used this to wrap a regular ClojureScript atom, turning it into a fully React-compatible external store.

On top of this, we layered the store's signal graph logic: fine-grained subscription invalidation and recomputation based on upstream changes.

Accounting for Differences

With HSX and RFX at a production-grade checkpoint, it was time to audit Kpow’s functionality and identify any performance regressions between the old (Reagent-based) and new (React 19-based) implementations.

As we touched on earlier, the key architectural difference was relying on React’s batched updates instead of Reagent’s custom batching system.

Up to this point we had Kpow running on HSX and RFX without any structural changes to our view or data layers. We had effectively the same application, just running on a new foundation.

Our Only Regression: Data Inspect

We noticed only one major area where performance regressed after the migration: our Data Inspect feature.

Data Inspect is one of Kpow’s most sophisticated pieces of functionality. It allows users to query Kafka topics and stream results into the browser in real-time. Given that Kafka topics can contain tens of millions of records, this feature has always demanded a high level of performance.

We observed that when result sets grew beyond 10,000 records, front-end performance degraded when a user clicked the "Continue Consuming" button to load more data.

Root Cause: Subscriptions vs Component Responsibility

Upon investigation, the root cause was clear: we were performing sorting operations inside a re-frame subscription.

Because React’s batched update model differs subtly from Reagent’s, this subscription was being recomputed more frequently as individual records streamed in from the backend.

Each recomputation triggered an expensive sort over an increasingly large dataset. Under the old model (Reagent), our batched updates masked some of this cost. Under React’s model, these inefficiencies became more visible.

Solution: Move Presentation Logic to Components

The fix was simple and logical:

• Sorting and other presentation-specific logic were moved out of the re-frame database layer and into the UI components themselves using local state.
• Components could now locally manage view-specific transforms on the shared data stream, without polluting the central app-db or affecting unrelated views.
• This also better modeled reality: multiple queries might view the same underlying result set with different sort preferences or filters.

This change not only solved the performance regression, but improved architectural clarity, separating global application state from local view presentation.

Features like data inspect could be better served by React APIs like Suspense.

Figuring out how newer React API features like suspense and transitions fit into HSX+RFX is part of ongoing research at Factor House!

The Outcome: Better Performance, Better Developer Experience

Performance Improvements

We saw performance gains across several dimensions:

• Embracing concurrent rendering in React 19 allowed React to interrupt, schedule, and batch rendering more intelligently — especially critical in data-heavy UIs.
• Eliminating class-based components, which Reagent relied on under the hood, removed unnecessary rendering layers (via :f>) and improved interop with React libraries.
• Fixing long-standing Reagent interop quirks such as the well-documented controlled input hacks gave us more predictable form behavior with fewer workarounds.
• Removing all use of Higher-Order Components (HOCs), which had previously introduced subtle performance traps and referential equality issues.

Profiling Kpow

We benchmarked two versions of Kpow using React's Profiler:

• Kpow 94.1: Reagent + re-frame + React 17
• Kpow 94.2: HSX + RFX + React 19

The result: HSX+React19 led to overall fewer commits.

With both versions of the product observing the same Kafka cluster (thus identical data for each version), we ran a simple headed script in Chrome navigating through Kpow's user interface.

We found that HSX resulted in a total of 63 commits vs Reagent's 228 commits:

Reagent profiling at 228 commits

HSX profiling at 63 commits!

Some notes:

• The larger yellow spikes of render duration in both bar charts were roughly identical (around ~160ms)
• These spikes relate to the performance of our product, not Reagent or HSX. This is something we need to improve on!
• This isn't the most scientific test, but seems to confirm that migrating to React19 has resulted in overall less commits without blowing out the render duration.
• See this gist on how you can create a production React profiling build with shadow-cljs.

Developer Experience

We also took this opportunity to address long-standing developer pain points in Reagent and re-frame, especially around testability and component isolation.

Goodbye Global Singletons

Re-frame's global singleton model, while convenient, made it hard to:

• Isolate component state in tests
• Run multiple independent environments (e.g., previewing components in StorybookJS)
• Compose components dynamically with context-specific state

With RFX, we took a idiomatic React approach by using Context Providers to inject isolated app environments where needed.

(defmethod storybook/story "Kpow/Sampler/KJQFilter" [_]
	(let [{:keys [dispatch] :as ctx} (rfx/init {})]
    	{:component [:> rfx/ReactContextProvider #js {"value" ctx} [kjq/editor "kjq-filter-label"]]
        	:stories   {:Filter     {}
            	:ValidInput {:play (fn [x]
                						(dispatch [:input/context :kjq "foo"])                                      										(dispatch [:input/context :kjq "bar"]))}}}))

Here, rfx/init spins up a completely fresh RFX instance, including its own app-db and event queue, scoped just to this story.

Accessing Subscriptions Outside a React Context

One of the limitations we frequently ran into with re-frame was the inability to easily access a subscription outside of a React component. Doing so often required hacks or leaking internal implementation details.

But in real-world applications, this use case comes up more than you might expect.

For example, we integrate with CodeMirror, a JavaScript-based code editor that lives outside of React’s render cycle. Within CodeMirror, we implement rich intellisense for several domain-specific languages we support including kSQL, kJQ, and Clojure.

These autocomplete features often rely on data stored in app-db. But much of that data is already computed via subscriptions in other parts of the application (materialized views). Re-computing those values manually would introduce duplication and potential inconsistency.

Another example: when writing complex event handlers in reg-event-fx, it's often useful to pull in a computed subscription value (using inject-cofx) to use as part of a side-effect or payload.

With RFX, this problem is solved cleanly via the snapshot-sub function:

(defn codemirror-autocomplete-suggestions
	[rfx]
   	(let [database-completions (rfx/snapshot-sub rfx [:ksql/database-completions])]
    	;; Logic to wire up codemirror6 completions based on re-frame data goes here
        ...))

This gives us access to the latest materialized value of a subscription without needing to be inside a React component. No hacks, no coupling, just a clean, synchronous read from the store.

It's a small feature, but one that has made a big impact on the architecture of our side-effecting code.

Developer Tooling

As a developer tooling company, it should come as no surprise that we're also building powerful tools around these new libraries!

Following from our earlier point about isolated RFX contexts, this architectural shift unlocked an entirely new class of debugging and introspection capabilities, all of which we're packaging into a developer-focused suite we're calling rfx-dev.

Here’s what it can do, all plug and play:

• Subscription Inspector
  See which components in the React DOM tree are using which subscriptions, including:
  • How often they render
  • When they last rendered
  • Which signals triggered them
• Registry Explorer
  A dynamic view of:
  • Subscriptions
  • Registered events
  • Their current inputs and handlers
• Profiling Metrics
  Measure performance across the entire data loop:
  • Event dispatch durations
  • Subscription realization and recomputation timings
• Event Log A chronological record of dispatched events - useful for debugging complex flows or reproducing state issues.
• Interactive REPL
  Dispatch events, subscribe to signals, and inspect current app-db state in real-time - all from the browser.
• Time Travel Debugging
  Snapshot, restore, and export app-db state - perfect for debugging regressions or sharing minimal reproduction cases.
• Live Signal Graph
  A visual, interactive graph of your subscriptions dependency tree.
  See how subscriptions depend on one another and trace data flows across your app in real-time.

rfx-dev is still a work-in-progress but we’re excited about where it’s heading. We hope to open-source it soon. Stay tuned! 🚀

Summary

What began as a necessary migration became an opportunity to radically improve our front-end stack.

We didn’t just swap out dependencies, we:

• Preserved what we loved from Reagent and re-frame: Hiccup and a data-oriented event and subscription model.
• Dropped what was holding us back: class-based internals, ratoms, global singletons.
• Aligned ourselves with idiomatic React: hooks, context, and newer API features.

HSX and RFX are more than just drop-in replacements, they’re the result of over a decades experience working in ClojureScript UIs - rethought for React’s present and future.

After adopting these libraries we find our UI snappier and our code easier to test and reason about. Our team is better equipped to work with the broader React ecosystem, no compromises or awkward interop. Our intent is to continue to hold close to React as the underlying library evolves further in the future.

For years, the Reagent + re-frame stack was the gold standard for building reactive UIs in ClojureScript and many companies (like ours) adopted it with great success. We know we're not alone in experiencing the issue of migrating to React 19 and beyond, if you find yourself in the same boat let us know if these libraries help you.

HSX and RFX are open-source and Apache 2.0 licensed, we're hopeful they contribute some value back to the Clojure community.