How Pickles replaced hours of Kafka checks with instant visibility

From stock and station agent to real-time auction marketplace

Pickles started in 1964 as a stock and station agent in Moree, in rural northern New South Wales. It has since grown into Australia’s leading auction marketplace, specializing in asset remarketing across industrial equipment, cars, trucks, and salvage. In its last full financial year, Pickles sold 228,000 vehicles and more than $460 million in transport and construction equipment.

Solutions Architect Krishnakumar Arjunan is one of 20 engineers on the Pickles development team, spanning Australia and India. That team builds and runs the Kafka-backed infrastructure behind Pickles’ real-time services, including PicklesLIVE, which lets customers bid remotely at live auctions, and PicklesONLINE, an automated bidding system for auctions that run over several days. Both depend on Kafka to move bidding and inventory data in real time, which means any gap in visibility into that data has a direct impact on live auctions in progress.

The challenge: validating Kafka messages without purpose-built tooling

Easy access to data on topics is key to validating that a Kafka-backed system is behaving as expected. Krishnakumar’s team found that simple tasks, like confirming a message had actually been delivered, could take hours without the right tooling. The problem was common enough that the team started building custom tooling of its own. “We used a custom microservice to read data from a topic and validate message delivery, but we can’t build microservices to check each use case,” Krishnakumar said.

After months of searching for a better way to work with Kafka, one of Krishnakumar’s Australian team members came across Factor House’s Kpow. Pickles started a free trial in January 2020, and by March had signed on for a paid subscription. What convinced the team was Data Inspect, Kpow’s feature for browsing topic data directly. It let engineers view messages as soon as they were persisted, check schema names and versions, and confirm consumer group names were unique, all without writing a single-purpose microservice first. “Kpow is intuitive and user-friendly. It saves us time, is efficient, and our developers like using it,” Krishnakumar said.

The solution: Data Inspect as the team’s day-to-day Kafka interface

Day to day, Pickles’ engineers use Kpow’s Data Inspect to browse topics, filter through message payloads with JQ-like queries, and confirm schema and consumer group correctness across the Kafka estate behind PicklesLIVE and PicklesONLINE. Configurable serdes with full Avro support mean engineers can read messages in their native format rather than working around raw bytes, and role-based access control with data masking keeps sensitive auction and customer data protected as more of the team relies on the tool.

That day-to-day reliance is backed by a fast pace of product development and support. Kpow shipped 59 updates in its first year on the market, and Krishnakumar’s team has come to count on quick turnaround when they hit something new. “It’s awesome. We connect with the Factor House team, and someone helps me within 24 to 48 hours if it’s a small one, and if it’s bigger, within a week. They’re very friendly and make sure the problems are solved,” Krishnakumar said. For a team supporting live, real-time auctions, that responsiveness matters as much as the tooling itself, since a slow-moving vendor would undercut the same visibility Kpow was brought in to provide.

The results: from hours of manual checks to instant visibility

What used to take hours of manual validation, or a custom-built microservice, is now handled directly in Kpow’s Data Inspect view. Engineers can confirm message delivery, schema correctness, and consumer group health as soon as data lands in a topic, without writing new tooling for each use case. That shift was decisive enough that Pickles moved from a free trial to a paid subscription within two months, and Kpow has since become the team’s default way of working with Kafka. “If there’s a problem with our Kafka system, there’s no way to track it unless you have Kpow,” Krishnakumar said. “If you know Kafka, you should be able to use Kpow without an issue.”

Consumer lag, presented directly in the Kpow UI, has become another point of daily reference for the team, giving engineers an early warning on the kind of processing delays that would otherwise surface only once they’d affected a live auction. Taken together, the time saved on message validation, the fast move from trial to production use, and the confidence of reliable vendor support have made Kpow the tool Pickles’ engineering team reaches for whenever something in their Kafka pipeline needs an answer.