Data governance policy examples for streaming platforms
KafkaData governance policies for a streaming platform are enforceable rules that state how schemas may evolve, how topics are created and retired, how sensitive fields are classified and masked, who can access which resources, and how every record is traced back to its producer. On Kafka each policy maps to a concrete control: a registry compatibility rule, a topic configuration, an ACL, or a required record header. This page holds worked policy blueprints. The definition and the four control areas live on what is a data governance policy, and the practice area around them on stream governance.
I see a lot of organisations reach for cluster boundaries as their governance mechanism, where every domain gets its own cluster. I think that is an inefficient way to do it, because governance domains change all the time. Two domains you kept separate can turn out to be the same domain next year, or the reverse. Policy belongs in the controls this page describes, not in your cluster topology.
Schema evolution and compatibility policies
A schema evolution policy states which changes to a message format are allowed and which are rejected before they reach a topic, building on the topic and partition fundamentals covered in the complete Kafka guide. The enforcement point is a schema registry, which stores Avro, Protobuf and JSON Schema definitions and checks every proposed change against a compatibility level.
The standard compatibility levels, as defined in the Apicurio Registry rule reference, are:
- Backward. Consumers using the new schema can read data written with the previous schema.
- Forward. Consumers using the previous schema can read data written with the new schema.
- Full. The new schema is both backward and forward compatible with the previous one.
- Transitive variants. The same guarantees held against all earlier versions, not only the most recent.
An example policy in practice: payment topics run FULL_TRANSITIVE so any consumer replaying history can read every version ever written, while low-risk internal telemetry runs BACKWARD to keep iteration cheap. The policy also names who may register a new subject and whether producers must serialize against a registered schema before publishing at all.
The pattern we keep seeing in published engineering accounts is that schema governance pays for itself only if it arrives before scale. Cloudflare’s Kafka platform started on JSON, and what should have been a decoupling layer produced tightly coupled producers and consumers. The fix that held was Protobuf with a strict one-type-per-topic rule and client-side validation. The lesson we take from it: pick the compatibility policy while changing it is still cheap.
Automated topic creation and lifecycle management
A topic lifecycle policy states how topics are provisioned, named, configured and retired, so a shared cluster does not accumulate thousands of unowned topics. An unused topic still costs the cluster something: it holds partition metadata the controller must carry and replicate, and it hides the topics that matter.
An audit log showing who changed what, and a break-glass access grant.
A workable policy covers four things:
- A naming convention that encodes ownership and purpose, for example
environment.domain.dataset.version, so a topic’s owner is readable from its name and access rules can match on prefixes. - Mandatory baseline configuration. Production topics commonly require a replication factor of 3 with
min.insync.replicasof 2. Per the Apache Kafka topic configuration reference,min.insync.replicassets the minimum number of in-sync replicas that must acknowledge a write when a producer uses acks=all, which is what makes the durability guarantee real. - Provisioning through code, not ad-hoc CLI access: topics are declared in reviewed configuration and applied by automation, so every topic has an owner of record.
- A retirement path. Topics with no producers and no consumer groups for a defined period are flagged and deleted. LinkedIn built automation for exactly this after unused topics inflated controller response payloads to 0.75 MB, close to the 1 MB limit.
GitOps and cloud vendors have made it very easy to create new Kafka clusters and topics, and I have seen organisations scale prematurely because of it, with every team or service getting its own cluster until there is a proliferation nobody owns. The lifecycle policy is what stops the same thing happening one level down, at the topic. The scale this matters at is real: DoorDash provisions roughly 100 new Kafka topics every week. And even with CI and automated GitOps workflows in place, keep a break-glass option for emergency config changes, then periodically audit the running config against your documented policy, because break-glass changes are exactly the ones that never make it back into GitOps.
Data classification and PII guardrails
A classification policy assigns every topic and, where schemas allow, every field a sensitivity tier, and states what each tier requires. A common matrix runs Public, Internal, Restricted and Highly Confidential, with the tier recorded as topic metadata so tooling and reviews can act on it.
The guardrails a classification policy typically mandates:
- Field-level protection at the producer. Fields classified as PII or cardholder data are encrypted or tokenised before the record is published, so plaintext never lands on the broker disk. Kafka itself does not encrypt data at rest, which is why this responsibility sits with the producer or with disk-level encryption. The pattern that makes per-field encryption workable at Kafka throughput is envelope encryption.
- Masking on read. Operational tooling that lets engineers inspect topic data applies redaction rules to classified fields, so a debugging session does not become a data exposure.
- Sink restrictions. Connectors moving data to long-term storage may only attach to topics whose classification permits it, which keeps Restricted data out of unmanaged buckets.
Open metadata platforms implement the tagging half of this pattern. OpenMetadata, for example, uses tags and classifications to categorise data, identify sensitive information and enforce access policies.
One thing we flag from production experience: teams that turn on DEBUG-level authorizer logging to satisfy a compliance requirement usually find it unmanageable in practice, with logs that capture denials but not successful access. A classification policy that names its guardrails up front avoids retrofitting audit evidence out of logs that were never designed for it.
Role-based access control and ACL templates
An access policy states which principals may perform which operations on which resources, and it is written as templates so granting access is a fill-in exercise rather than a judgement call.
Kafka’s native mechanism is the ACL: an ACL grants or denies operations on resources such as topics, consumer groups and the cluster itself to principals such as users and service accounts. Access governance in practice runs in layers: topic access, schema access and cluster-level control, and RBAC in management tooling sits alongside the native ACLs that govern the data plane. Both layers are typically needed, because the ACLs decide what an application may do and the RBAC decides what a human operator may do.
Template examples a policy document carries:
- Producer isolation. A service principal may write only to its own domain prefix, for example
billing.*, and read nothing. - Consumer isolation. A consuming application may read named topics and commit offsets only for its own consumer group.
- Principal mapping. How TLS certificate identities or SASL identities map to principals, so a certificate rotation does not silently change who holds access.
- Break-glass access. Temporary elevated rights with an expiry and an audit entry, granted through a recorded workflow rather than a standing admin account.
The practical failure mode in Kafka authorization is permission creep: ACLs accumulate over time through broad grants and principals nobody revokes. Templates exist to make the narrow grant the easy grant. At the far end of the scale, Uber replaced thousands of individual ACL entries with a single attribute-based policy in a custom authorizer, which is the same idea taken to its conclusion. On temporary access, Derek Troy-West, our co-founder, put it this way: “temporary policies are a bit more useful than we expected - teams build them into their workflows to manage access to stuff.” Our guide to managing temporary access to Kafka resources covers the break-glass template in practice.
Data lineage and metadata requirements
A lineage policy states what tracking metadata every record must carry, so that when a downstream system reports bad data the trail leads back to the exact producing service. Without it, tracing a corrupt record through several topics and processing jobs is archaeology.
The common implementation is mandatory record headers. A policy names the headers every producer must populate, typically a trace identifier, the producing service, a producer timestamp and the schema version, and CI or a platform library rejects producers that omit them. Because the headers travel with the record, any consumer or audit job can reconstruct the path without a side channel.
At the platform level, most organisations with a data governance program pair the header contract with a metadata catalog that accumulates lineage described in the OpenLineage standard, an open framework for lineage collection that models datasets, jobs and runs. That gives compliance reviewers a queryable record of dataset versions, schema changes and job executions rather than a folder of diagrams.
The requirement that shaped our own lineage work came from an auditor. A German commercial bank we work with was asked, in an audit, “who owns this data flow, and what depends on it?” They had a business catalog already. What they did not want was a second, parallel catalog inside their operational tooling. The policy lesson generalises: lineage metadata should live where the operators work, and read from the catalog the organisation already maintains.
FAQ
What are some examples of data governance tools?
On a streaming platform the tooling splits by policy area: a schema registry such as Apicurio Registry enforces evolution rules, metadata platforms such as OpenMetadata carry classification tags, and the OpenLineage standard describes lineage that catalogs accumulate. Kafka’s own ACLs plus RBAC in management tooling cover the access policies.
What are the five pillars of data governance?
Frameworks vary, but on a streaming platform governance policy consistently breaks into the five classes this page works through: schema evolution and compatibility, topic creation and lifecycle, data classification and PII guardrails, role-based access control with ACL templates, and lineage with metadata requirements.
What are the best practices for data governance?
Make every policy enforceable as configuration or code rather than a document: registry compatibility levels per topic class, topics provisioned through reviewed automation with a retirement path, sensitivity tiers recorded as metadata that tooling acts on, access granted from templates, and mandatory record headers for lineage.