Overview
Factor House Local is a collection of pre-configured Docker Compose environments that demonstrate modern data platform architectures. Each setup is purpose-built around a specific use case and incorporates widely adopted technologies such as Kafka, Flink, Spark, Iceberg, and Pinot. These environments are further enhanced by enterprise-grade tools from Factor House: Kpow, for Kafka management and control, and Flex, for seamless integration with Flink.
Data Stack
Kafka Development & Monitoring with Kpow
This stack provides a comprehensive, locally deployable Apache Kafka environment designed for robust development, testing, and operations. It utilizes Confluent Platform components, featuring a high-availability 3-node Kafka cluster, Zookeeper, Schema Registry for data governance, and Kafka Connect for data integration.
The centerpiece of the stack is Kpow (by Factorhouse), an enterprise-grade management and observability toolkit. Kpow offers a powerful web UI that provides deep visibility into brokers, topics, and consumer groups. Key features include real-time monitoring, advanced data inspection using kJQ (allowing complex queries across various data formats like Avro and Protobuf), and management of Schema Registry and Kafka Connect. Kpow also adds critical enterprise features such as Role-Based Access Control (RBAC), data masking/redaction for sensitive information, and audit logging.
Ideal For: Building and testing microservices, managing data integration pipelines, troubleshooting Kafka issues, and enforcing data governance in event-driven architectures.
Unified Analytics Platform (Flex, Flink, Spark, Iceberg & Hive Metastore)
This architecture establishes a modern Data Lakehouse that seamlessly integrates real-time stream processing and large-scale batch analytics. It eliminates data silos by allowing both Apache Flink (for streaming) and Apache Spark (for batch) to operate on the same data.
The foundation is built on Apache Iceberg tables stored in MinIO (S3-compatible storage), providing ACID transactions and schema evolution. A Hive Metastore, backed by PostgreSQL, acts as the unified catalog for both Flink and Spark. The PostgreSQL instance is also configured for Change Data Capture (CDC), enabling real-time synchronization from transactional databases into the lakehouse.
The stack includes Flex (by Factorhouse), an enterprise toolkit for managing and monitoring Apache Flink, offering enhanced security, multi-tenancy, and deep insights into Flink jobs. A Flink SQL Gateway is also included for interactive queries on live data streams.
Ideal For: Unified batch and stream analytics, real-time ETL, CDC pipelines from operational databases, fraud detection, and interactive self-service analytics on a single source of truth.
Apache Pinot Real-Time OLAP Cluster
This stack deploys the core components of Apache Pinot, a distributed OLAP (Online Analytical Processing) datastore specifically engineered for ultra-low-latency analytics at high throughput. Pinot is designed to ingest data from both batch sources (like S3) and streaming sources (like Kafka) and serve analytical queries with millisecond response times.
Ideal For: Powering real-time, interactive dashboards; user-facing analytics embedded within applications (where immediate feedback is crucial); anomaly detection; and rapid A/B testing analysis.
Factor House Local Labs
The Factor House Local labs are a series of 12 hands-on tutorials designed to guide developers through building real-time data pipelines and analytics systems. The labs use a common dataset of orders from a Kafka topic to demonstrate a complete, end-to-end workflow, from data ingestion to real-time analytics.
The labs are organized around a few key themes:
š§ Lab 1 - Streaming with Confidence:
- Learn to produce and consume Avro data using Schema Registry. This lab helps you ensure data integrity and build robust, schema-aware Kafka streams.
š Lab 2 - Building Data Pipelines with Kafka Connect:
- Discover the power of Kafka Connect! This lab shows you how to stream data from sources to sinks (e.g., databases, files) efficiently, often without writing a single line of code.
š§ Labs 3, 4, 5 - From Events to Insights:
- Unlock the potential of your event streams! Dive into building real-time analytics applications using powerful stream processing techniques. You'll work on transforming raw data into actionable intelligence.
šļø Labs 6, 7, 8, 9, 10 - Streaming to the Data Lake:
- Build modern data lake foundations. These labs guide you through ingesting Kafka data into highly efficient and queryable formats like Parquet and Apache Iceberg, setting the stage for powerful batch and ad-hoc analytics.
š” Labs 11, 12 - Bringing Real-Time Analytics to Life:
- See your data in motion! You'll construct reactive client applications and dashboards that respond to live data streams, providing immediate insights and visualizations.
Overall, the labs provide a practical, production-inspired journey, showing how to leverage Kafka, Flink, Spark, Iceberg, and Pinot together to build sophisticated, real-time data platforms.
Conclusion
Factor House Local is more than just a collection of Docker containers; it represents a holistic learning and development ecosystem for modern data engineering.
The pre-configured stacks serve as the ready-to-use "what," providing the foundational architecture for today's data platforms. The hands-on labs provide the practical "how," guiding users step-by-step through building real-world data pipelines that solve concrete problems.
By bridging the gap between event streaming (Kafka), large-scale processing (Flink, Spark), modern data storage (Iceberg), and low-latency analytics (Pinot), Factor House Local demystifies the complexity of building integrated data systems. Furthermore, the inclusion of enterprise-grade tools like Kpow and Flex demonstrates how to operate these systems with the observability, control, and security required for production environments.
Whether you are a developer looking to learn new technologies, an architect prototyping a new design, or a team building the foundation for your next data product, Factor House Local provides the ideal starting point to accelerate your journey.