Running Kafka at bank scale

My name is Sandy. I'm the staff engineer and technical asset owner of TD's event streaming platform. We also call it ESP, so if I refer to ESP, it's us. My philosophy is we should build something useful, help the team grow together, and keep work fun: at least 20% of your working time, not all the time — it's not possible in a big bank, I would say, but most of the time we have fun.

Back in 2020 I worked on TD's enterprise data platform team, in the big data area. We had ETL, analytics, all sorts of stuff. But a handful of us figured out that batch workload isn't enough for a bank — we needed something faster. So we started to lobby our leadership and say, "Hey, we want to build a Kafka to make event streaming happen." We got some money, so we went on a roadshow: we went to all sorts of lines of business, asked for their requirements and use cases, and came back with everybody wanting it but nobody having the money for it. We pretty much got our budget cut every quarter, until somebody came in — we got one person from our IT team who went to the payment team and said, "We need to do a payment modernization project, let's come build a Kafka cluster for us." That's how we ended up in the payment team, and we built our first on-premises cluster, launched in 2021.

Immediately after we launched it, a lot of business came to us on the payment cluster — for example, business banking and fraud teams all said, "We need event streaming." So we onboarded them, and in 2023 the team became an enterprise-level event streaming, Kafka team. My leader figured out that staying inside payments didn't make sense to us when there were so many other clients who wanted to onboard to this cluster. In 2023 we also launched our Confluent Cloud cluster — that was a multi-year battle to get to that stage, because Confluent Cloud at the time was on the internet, and the TD network couldn't just direct-connect and expose our credit card data. So we adopted the strategy of encrypting everything at the message level — that's the library we built for everybody, an important part of it.

We grew and we grew. Fast forward to today, we can very confidently say we are the short-term memory of the whole bank: data comes in from different sources and action is taken immediately, whether it's fraud detection, a notification, or AI enrichment — and if the data needs to be saved, it's saved to the data lake. Lesson learned number one: leadership matters. If leaders don't take the risk or don't have the vision, we won't get here — we'd probably have multiple little Kafka clusters that can't be maintained for two or three years, and the project that launched the Kafka cluster is somebody's side task. Lesson number two: an enterprise event streaming platform is the right call.

Right now we have more than 20 clusters across different areas, in four different flavors: on-premises virtual machines, on-premises physical servers for very low-latency clusters, Confluent Cloud general clusters, and Confluent Cloud business clusters. We have many topics, partitions, and schemas, and — most important — more than 300 projects onboarded to our platform, with lots of different patterns; it's not one pattern fits all. Our team has three parts. The dev part handles the build: we build the cluster, the encryption library, the self-service tooling, and the different patterns. Our operations team is the biggest of all our parts, because we need to handle critical services 24/7, deal with all the DR, patching, and governance requirements, and support clients in production once they're launched. It's a very busy team. Customer success is last but not least: it handles day-to-day, end-to-end client onboarding — client here means producer, consumer, line of business for us — all the way to billing and communication about library upgrades and so on.

So back to whether two people could support this ecosystem: there's so many different aspects. DR alone — we have more than 10 applications under ESP, and DR needs to be done every year, so scheduling and DR take forever. For billing, we need to make sure everybody is charged fairly and we don't mischarge anyone. It's pretty complicated. Lesson learned here: execution matters. Having a vision alone is not enough — we need a team, and we have a very good team that works together to make it happen. Collaboration is a must: some big organizations have silos of teams, but we don't — we have to work together, otherwise there's no way we can survive. And talking about survival, we have to address the burnout issue.

In the early days, we hand-held our clients — we were the friends of our clients. We manually created all the artifacts for them, using Confluent Control Center or the command line, but we didn't have a tool to self-service. Then we definitely got burned out, because we had Teams messages, Outlook messages, everybody had so many different ways of contacting us. We felt overwhelmed, and the decision was made that we needed self-service. The self-service journey isn't one day — it took years, gradually getting everything in, and luckily we got to this point: clients just use YAML files, put them in GitHub, and all we do is review their file, merge it, and approve it. ServiceNow serves as our book of record. The funny thing about ServiceNow is I hated it five years ago — I associated it with change orders that took forever to get approved — but now we leverage it every day, using ServiceNow forms to enable our automation. It's a really good tool for us.

Kpow, on the other side, is about monitoring and observability. Before we used Kpow, Confluent Control Center could only accept admin access and wasn't scalable. When we started using Kpow, all the clients and us are on it — when we need to do an admin test, sometimes we still use Confluent Control Center, but more importantly, clients are on Kpow. On the right side you can see what I call the matcha cake — these terms are so evocative that as an engineer it took me a while to understand what exactly this cake is about. In a nutshell, from the bottom up: platform economics means you need to build a platform at the lowest cost — you cost it very carefully, and luckily we have a lean, highly skilled team, so we enabled that. The second layer, enablement efficiency, means having as many clients, and high-profile, high-throughput clients, on it. As a result, we get to the top layer, business ROI — return on investment — and that's how we aim ourselves. Kpow gives us the best ROI: having more people using it, and us using it, gives us a very good ROI. So thank you, Kpow team.

Let me introduce what Kpow can do with some screenshots. Here's the overview of Kpow: cluster, brokers, topics, consumer groups, and so on, plus stats that both clients and us can look at. The next tab is Data — it lets us inspect and produce data. In the screenshot I have here, there's encrypted data — Kpow doesn't natively enable encryption, since that's part of TD's own system, but Kpow allows custom SerDes, so we can put our own JAR onto Kpow, and clients can see encrypted data in an unencrypted format. So although it looks like gibberish, it's actually unencrypted data as our clients see it — this gives clients confidence about what's going through Kafka. If you don't have an interface to show them what a message looks like, they think of Kafka as a very abstract thing; with this visualization of what's inside a topic, business people can go in and say, "My data looks good." That kind of confidence is very important. You can also produce to a Kafka topic using Kpow — this is another bonus in dev environments, when a consumer is ready but a producer isn't: consumers can just go into Kpow and produce sample data to consume, without needing to create their own Kafka command-line tool or producer.

This slide also shows monitoring — you can see consumer-related information, and here's the Connectors view. This one connector just failed, and if you click the hamburger menu you can either restart the connector or see why it failed, including the stack trace. Very convenient. For schemas, you can see the normal schema or compare different schema versions — here we have version 95 and version 94, and you can see clearly what was added. Kpow has many settings features too. Here we saw performance: one time we had a performance issue with Kpow, so we had a team of people load-test it through the Kpow interface, watched Kpow's own performance in real time, saw when we broke it, and reported that back to the Factor House team to look into. We also use settings to set up RBAC, as well as tenant policy and other admin-related items.

Talking about self-service: one big piece is Kpow access. We gave people access to Kpow based on their AD group, and we have a ServiceNow knowledge base for them — one version already has 1,300 views from a month ago, and probably a lot more now. This shows how many people are using Kpow self-service. Dev, staging, and production use different access patterns — in production, for example, you have to be on the operations team to get access, otherwise you won't get it. For getting access, the process is: the client goes through self-service, opens the firewall, creates an Active Directory group, creates a change order in ServiceNow, and adds the user to their AD group. Once the change order gets to us, we set up a tenant and access policy for them, and they verify access. That's it.

Production is a little special — there are two use cases. One is inspect access, where clients just want to see what's in the data: we have a ServiceNow integration using the Kpow API, so they fill in a ServiceNow form and get access within a minute, scoped for an hour or two — you don't have to give full access forever. This gives TD an audit trail. The other aspect: anyone who wants to publish to production has very rigid rules — you shouldn't publish to any topic in production; if you do, you have to ask for permission and an exception, and we grill them on why and when they'll change it. This is how we guarantee people don't just produce data to production.

On the API side: we already talked about RBAC auto-approval using the Kpow API — the syntax is very simple, just a curl command from ServiceNow to get access quickly. We also use Kpow metrics to collect a whole lot of different metrics, initially for clients who needed consumer lag information. Before, we'd run Kafka CLI commands to describe topics and offsets and do a lot of manipulation — it took 20 minutes or more per cluster, which isn't good enough if a line of business has a lot of messages. With Kpow it's way better: we can run a Kpow job and other processes within two minutes, so we run it every five minutes and collect data. Clients get their lag information, create a ticket, and go resolve it themselves — self-service on the client side too.

Another big aspect of using the Kpow API: with this metrics data we can generate executive dashboards or scorecards, and surface that data to leadership so they can act on it. For example, if a line of business has no data in the real-time engine, an executive can ask, "Why aren't you doing real time? Why are you still going with batch all the time?" That's how we use the data to show executives that Kafka is important and real time needs their support.

That's pretty much everything I have — thank you so much. We didn't have all the answers: some clients still ask if they can get 200 or 2,000 partitions because of consumer lag, or want their schema compatibility set to none, and we have to go over, again and again, why that's not a good idea and what coordination it would need. So that's me — thank you for joining, and thank you, Factor House, for hosting.