Agreed vs. validated: fixing schema quality across multi-producer Kafka topics at Siemens

My talk today is about quality, and the challenges we currently have in my area of responsibility. I'm a Senior Key Expert in Data Integration at Siemens, and my subfields are data streaming and event-centric architecture. I became responsible for this by being part of, and then more or less responsible for, the data streaming side of a new integration that produces product master data and shares it with the outside world. I sit directly in IT, so I don't have customer contact except at venues like this one — otherwise I'm in the machine room. We send the data to an internal system, and from there the customer gets it.

The title is "agreed versus validated," and that's the important distinction. We have agreed structures for our messages, but we don't validate them. If something is agreed but not checked, it becomes a challenge. Our current problem is that we have multi-producer setups for commercial product master data — for example, marketing text describing a product's highlights, and the product's lifecycle status: whether it's still sold regularly or already in the spare-parts stage. That information goes out to our customers. When one producer among several slips, everybody needs to resupply, and that's exactly what we faced around the start of this year.

We weren't taking care of schemas or quality, so when one producer did something wrong, every consumer was affected, and every producer was affected too, because we share one data store topic. Kafka gives you no native way to redo something here: to correct it, you either resupply the whole chain or delete it and recreate it from scratch. That's a big problem, because it hits every source system and every consumer along the chain.

What we did was create two topics for each source system, and introduce a "merger" component in front of the shared data store topic. Now, when one source system sends wrong messages, the other source system is unaffected and doesn't need a resupply — we do the resupply through the merger instead. Consumers are still affected when we fix something, but the issue no longer spreads back to every producer.

I then did an analysis of what was actually flowing through these topics, and it got interesting. Looking at it from a distance, we had three different systems, from two suppliers, sending messages into the shared topic: one source system, its on-demand API, and a second source system. I checked the messages themselves rather than just relying on the existing schema, which turned out to be incomplete, and found that supplier one was sending a field called "textM1" while supplier two was sending "textM" — no consumer had ever complained about the inconsistency. We also had a stray "@segment" field: source system one sends IDocs into our gateway, which translates them to JSON, and that translation introduces "@segment" field names — which is an illegal Avro field name. Oddly, the on-demand interface call for the same source didn't produce "@segment" fields at all, so we ended up with three different interpretations of what was structurally meant to be the same data, none of it visible just by looking at the nominal schema.

So we measured. The full schema runs to over 800 fields, and the schema I'd inherited didn't hold up. What I did was use Claude Code to connect to the topic, iteratively build up the schema field by field, and check the outcome against real messages each time: consume, compare, classify, patch, re-verify. It was a lot of iteration. I started with Avro rather than JSON Schema, because Avro is the schema format Siemens has standardized on, even though we weren't yet using schemas in practice — I wanted to understand what committing to Avro would actually mean for us. I made it strict at first, and that produced a lot of failures: the "@segment" field alone would have invalidated every affected message, so I had to work around it, and a lot of genuinely optional fields I didn't know about were also being filtered out. So I nullified everything — made every field optional so real messages would validate — but that meant the schema now accepted almost anything, and the harder work of figuring out what's actually mandatory is where we are today.

That's the core lesson about validators: you always have to keep in mind that validation can be too strict or not strict enough, and it depends entirely on the data you have. Too strict and you get zero valid messages; too loose — all fields optional — and you get effectively everything validating. If you're doing a brownfield integration like ours, you'll lean toward nullable-by-default at first. Since we already had to touch the message at the merger, we introduced a validator there too — we call it the "gatekeeper" — to move validation further left without breaking the consumer side.

Some numbers from this: our last test run, over 16 hours, produced 400,000 error messages, and 97.7% of them traced back to a single defect from one producer. Fix that one thing and nearly everything is fixed. We also discovered that in one cluster, every value comes through as a string regardless of whether it's really an integer or a boolean — an internal IDoc-to-JSON translation issue we've since corrected. We'll fix the "@segment" issue too. The lesson here is that you really do need to look at your dead letter queue when you validate, and sum up the errors to see which fix gets you the most impact.

Based on everything I learned from this iterative schema work, our direction is what I'd call a C-B-A approach. "C" is validating at the end: we keep messages as they are, but once we introduce Avro-serialized messages, validation happens on the consumer side, on the data store topic itself, with a dead letter queue for anything invalid — so at least the consumer is no longer hit by malformed messages. That's the step we started with, to see what issues we actually had. "B" is the gatekeeper approach at the merger, which needs a valid schema and more implementation work, and only lets valid messages through to the data store topic, again backed by a dead letter queue. The best option, "A", is producer-side validation, using the producer's own native validation, so errors surface at the producer and the whole downstream chain stays clean. Moving from C to A, the blast radius shrinks and the cleanup cost drops.

One thing that matters less for us, because we run centralized schema management with a dedicated team that owns schema uploads, is auto-register. On platforms like Confluent Cloud or Apache Kafka, auto-register lets a producer register its own schema — self-certification, essentially: the producer decides what's valid and sends whatever fits. We don't allow that for producers, because we don't yet have a schema for this kind of data — I pushed to create one so we could make progress. And a gatekeeper is only ever as good as its schema, which is true at every stage. If you use a dead letter queue but never look at it, it's just a data grave — you need a manual process to check it regularly and push producers to fix what's landing there.

Where this leaves us at Siemens today: validation at the end is live for commercial product master data, which we now publish as data products, backed by the schema I created and the processes and workflows I set up around it — and we have a dedicated dead letter queue for it. The gatekeeper at the merger is currently being built for the same schema. Over the next two years, the plan is to push toward valid schemas and producer-side validation everywhere, and eventually ramp down the gatekeeper and the end-of-chain validation again. That will take time, because it means touching legacy systems owned by other teams, each of which needs its own time and budget.

To close, ten things I'd tell myself in hindsight. Agreement versus schema versus validation: only validation is the real assurance. The unit of consistency is the interface, not the source system. Producer topics without a gatekeeper give you forensics, not isolation. Don't grow the schema against the messages you already have — that's only necessary when you're retrofitting a schema onto a system that was never validated from the start; if you start with a good schema, you don't need to grow it. Never parse error text: comparing only the error string, rather than the full structure, meant I only ever saw the first failure and missed everything behind it — a validator stops at the first failure, so expect a second wave once you fix it. Too strict blocks correct data — nullability is a trade-off, not a goal in itself. Put a dead letter queue at every stage from day one, even before you have a process for it, because without one — or the log files that stand in for one — you can't see your errors. Manage the schema centrally, so producers can't change it unilaterally. And the later you validate, the more defects are already sitting immutably in the log.

One more thing I'd add: we started with plain JSON values in our messages, and now that we're looking at schema validation, our preferred direction is JSON Schema combined with the schema ID in the message header, rather than in the value itself. That way we validate the existing JSON value against a JSON Schema, but only write the schema ID into the header — any consumer that understands validated messages and schemas can read them, and any consumer that doesn't is completely unaffected, because the value itself is untouched. In the past, embedding the schema ID as a magic byte in the value always broke consumers who weren't expecting JSON Schema. With the schema ID in the header instead, we can leave existing consumers exactly as they are and add quality on top.