Implementing Kafka at Belong

At Belong, when we decided to bring in a streaming platform, we were evaluating between MSK and Confluent. But at the time, the approval process within Telstra was so complex it was going to take quite a lot of time to bring in Confluent — that's the only reason we didn't choose it; if Confluent had already been a pre-approved tool, we would have gone with it. Given that AWS MSK was already approved as a managed streaming platform for Kafka, we went with it. We set it up as a highly available platform across multiple availability zones, integrated with the AWS ecosystem — IAM, CloudWatch for alarms, and so on. The benefits of a managed service were reduced operational overhead, built-in security and compliance, and seamless integration with existing AWS services. When we started there were fewer services that could integrate seamlessly, but there's more integration with MSK now.

The next few things are foundations we wanted to address before teams started adopting the platform. We wanted the cluster secure, and we wanted standards for how people contributed. First, we wanted to ensure everyone communicating with the cluster is authenticated and authorized. MSK gave us a bunch of options at the time, and SASL/SCRAM was the recommended way to go — a salted hash where you use a username and password through the SASL framework. The username is the principal you use when you perform RBAC on resources within the cluster, and the password is for authentication. When you create a username and password, you first create them in Secrets Manager, associate that secret with the cluster, and then move on to authorization, where you associate that principal with an ACL on each topic — a producer principal can only write to a topic, while a consumer principal can read from the topic, or create or describe a consumer group. We wanted every producer and consumer to authenticate with the cluster, because the cluster would eventually store PII or other sensitive information — we didn't want anyone to have a free pass. And the important decision was that all of this had to be done through code: over time, when 40 engineers are contributing, you don't want people running random commands against your cluster. So everything is managed through code — we chose the Kafka Terraform plugin to manage configuration declaratively, from authentication and authorization through to the pipelines that configure the cluster. That way we could scale easily across the organization, and if we ever lost the structure to a crash, we could restore all of our configuration. It was a conscious choice.

Once the cluster was secured, the first thing we do is create topics. You can name topics however you want, but we decided we wanted a pattern the teams would actually use, for two reasons: discoverability — clear ownership and purpose at a glance — and governance: consistency, enforcing boundaries between events and operations, easy filtering, monitoring, and access control. Our pattern starts with environment, followed by event type, data set, and data name. Environment could be dev, staging, or prod. Event type could be an application event, a CDC event, a domain event, or an integration event. The data set could be CRM, billing, or network, and the data name — for CRM it might be about a customer, for billing a subscription, for network an outage. This way we can easily locate our data, and it gives teams a convention to follow, especially when the team emitting the data and the team consuming it are completely different — it provides a clear structure for what the topic is about. For example, a CDC topic: change data capture — say our CRM system is Salesforce, and anytime there's a change in customer attributes, that gets copied to a topic, and the data team pulls it and pushes it to the data lake.

Next was how we manage different types of events across our systems. What mattered was having a category of events, and knowing which events we have control over versus which we don't, and the data formats we use for each. We settled on three event types. The first is application events — consumed primarily by internal applications, in JSON, and these are events we don't have control over: for instance, something emitted by an IDP system or by Telstra or an external vendor. We store these as-is, in whatever format they arrive in — most vendors today support JSON, so we just store it as given. Then there are domain events, consumed within a domain — say you have order activation and order fulfillment in the order domain, and orders emitted by order activation are consumed by order fulfillment: that's a domain event, and the topic name gets a "dom" prefix. For anything domain or inter-domain, we chose Avro as the format, because it emitted by Belong and we wanted schema evolution — teams within Belong can discover the schema and consume the messages — and it's binary, so it takes less space than JSON. Then integration events: cross-domain events, for instance an order event that needs to be consumed by billing to do something in the billing system. These are also Avro, again for schema evolution and discoverability. The key principle was decreasing coupling and increasing contract stability as you move up that hierarchy, so we stay backward compatible with schema changes. The schema registry we use also provides a catalog of schemas for people to discover and use — given we're on AWS MSK, we can't use Confluent's schema registry, so we use the AWS Glue schema registry.

So we've covered authentication, authorization, topic naming conventions, and event and data format strategy. Next: how do you actually look inside a topic? AWS doesn't provide a tool for that out of the box — you either buy a tool or use the CLI, and with the CLI I sometimes feel lost in the ocean of options each command has. So we made a conscious choice to buy a tool, and we use Kpow, for four reasons: real-time querying using kJQ, the ability to visualize messages, the ability to replay messages in case of production issues, and — given our topics can hold sensitive data — the PII masking feature in Kpow, which lets us mask fields like mobile number, first name, last name, or address. That's helped a lot with our cyber security and risk teams — they're glad we can mask PII, and it helps prevent people copying data out as well. Even a couple of days ago we were using Kpow to troubleshoot a production incident. It makes working with topics at the Kafka broker level really easy.

Let's talk about a use case. There are many use cases at Belong that use SMS — this one's interesting. As a telco, we wanted to send SMS for a variety of reasons: OTPs, reminders for customers to pay their bills, outage notifications, or marketing messages. OTPs are critical messages, so we didn't use Kafka for those. But there are two categories we did handle this way: operational messages, like bill reminders, and marketing messages. Traffic was bursty, and we wanted a solution that was resilient, fault-tolerant, and could respect downstream systems' rate limits. We wanted producers and consumers to evolve independently; for resilience, retry topics with a built-in retry mechanism give durable storage and guaranteed delivery; and we wanted rate limiting, prioritization, and message ordering — when I send multiple messages to the same customer, they need to be sequenced, and topic partitioning helps with that.

Our setup: messages could come from a marketing cloud, a CRM system, or our IDP, all flowing into a notification service. Based on message type, it would push into one of two topics — for example, app.notifications.sms-operational — you can see the topic naming convention in action. We stored it as JSON because the notification service was an API already enforcing a contract, so consuming systems had to comply with that message structure. The notification service picked up messages with two different consumers, one for operational and one for marketing, since rate limits differed by message type. The notification service would call a token service to ask if a token was available; if so, it called the SMS service and the message went out. If a token wasn't available, it would do an exponential backoff with jitter and request again on the next cycle.

This was a very simple setup, and we never thought it would go wrong — but it did. Our initial consumer configuration in production had three workloads for high availability, using the Streams API. We configured two stream threads per workload, so six consumer threads total, and by default each thread pulled 500 messages — default configuration, we didn't change it. The max poll interval — the time before which the consumer has to acknowledge it's processed all 500 messages — was five minutes, also unchanged. If a token wasn't available, we'd retry after an exponential backoff with jitter. And because the SMS service could only take six tokens per second across message types, we configured the token bucket to fill with six tokens every second.

Can you guess what went wrong? We exceeded the max poll interval — the worst thing that can happen, especially with SMS messages. My token service was only giving six tokens a second, so why would I even fetch 500 messages in the first place? We realized Kafka maybe wasn't even the right solution for this, but it had been chosen because there was a much bigger architecture we wanted to build that never eventuated, and it hit us when we went to production. Six threads pulling 500 messages each is 3,000 messages, and we know we can only do six tokens a second — so straight away the seventh message calls for a token within that second, fails because there's no token, and goes into exponential backoff. Imagine that happening across the remaining thousands of messages. When you do an exponential backoff you're blocking the thread, which meant we weren't sending heartbeats back to the broker — the broker decided the consumer thread was dead, and replayed the messages. What goes wrong with replaying messages? Customers receiving multiple copies of the same SMS, which doesn't play well at all.

So whenever you're designing solutions, always check the defaults against the problem you're solving — sometimes you copy-paste from a previous solution that was right there but not correct for what you're building now. We immediately reduced stream threads from two per workload to one, and messages pulled per thread from 500 to about 10 — the rate we could actually pump messages through, since this was a live production incident. After the change, everything flowed through smoothly. Reflecting on other ways we could have solved it: if we'd used SQS instead, you'd hit a similar problem with the visibility timeout expiring and the message becoming visible for another consumer again. Within Kafka, Kafka Streams offers another option called punctuators, which come online at specified intervals, read a set number of messages, and call the SMS service — but that comes with its own complexity, because the punctuator needs to manage state, which means using RocksDB, which means your workloads need to be stateful. So instead of stateful workloads, we stuck with the simpler configuration, since we knew it worked. The lesson from this incident: always revisit the defaults, make sure they fit your solution, and if you need back pressure, tune your configuration to the limits of your downstream system. We haven't had a single incident since making that change.

We felt that whether you're starting a streaming journey or already on one, it's very important that the platform's foundations and governance are strong, because that's what lets engineers contribute efficiently and confidently — I don't have to worry about people configuring the cluster directly, because we've locked down the cluster, and if there's an incident tomorrow, we know how to reproduce it. We've secured the cluster, and we have a catalog of all the schemas we use, which lets teams talk to each other through the schema and understand data formats. You spend time with your teams understanding what they're trying to achieve from the platform and how they want to use it. It's important to get this right early, because getting it right later is very expensive.

During Q&A, an attendee asked whether it had been hard to retain that operational knowledge inside the team as people moved in and out of roles — since avoiding the SMS incident again means remembering "magic numbers" like the default poll size and token rate. Nagaraj said post-incident reviews and chapter-connect sessions are one way the team shares learnings, but that knowledge is still lost as people move roles; AI has made it a bit easier, letting the team write up knowledge articles about past issues and query why a configuration was set a certain way, but there's no foolproof way to retain that knowledge — it's mostly reinforced through discussions like this one about not overlooking default configuration.

Asked whether AI was on the team's radar for operational work with Kafka data beyond that knowledge base, Nagaraj said it wasn't yet — the team has thought about using AI once data reaches the data lake, where all the data is joined and insights can be created, but not within Kafka itself, since not every data journey passes through it. What would help developers most is AI that, given a use case, could suggest what configuration to use and what to consider — but the team isn't yet at a point where they'd hand data to stakeholders and derive insights from it directly; they're constantly looking for opportunities to embed AI and realize that value.