A data governance policy is a documented, enforceable rule that states how data must be structured, who may access it, how long it is retained, and how its movement is traced. On a streaming platform the policy is implemented as configuration and code, in schema compatibility rules, access control lists, retention settings and audit logging, rather than as a document that describes intent. This page defines the policy artefact and its four control areas. Worked blueprints live on data governance policy examples, and the surrounding practice area on stream governance.
One PII field masked for one role and visible to another, in the same data view.
Schema enforcement and evolution
The first thing a governance policy fixes is the shape of the data, starting from the topic and partition fundamentals covered in the complete Kafka guide. On Kafka that means requiring producers to serialize against a schema registered in a schema registry, so a record that does not match its declared format is rejected at the point of production instead of crashing consumers downstream. The strictest form is schema-at-producer enforcement, where events must use a registered schema before they reach a topic at all.
The evolution half of the policy states which schema changes are legal, expressed as a registry compatibility level as defined in the Apicurio Registry rule reference. Backward compatibility lets consumers on the new schema read old data, forward compatibility lets consumers on the old schema read new data, and full compatibility guarantees both. A governance policy picks the level per topic class and names who may change it.
The policy also needs a stated behaviour for records that violate it anyway, commonly routing to a quarantine or dead letter topic so one malformed message does not halt a pipeline.
A detail from the field on why the enforcement point matters: a German banking IT provider we work with runs one schema registry per environment and told us plainly they would love to reduce the amount and version drift across them. Registry sprawl is itself a governance problem, because a compatibility rule only protects the topics that actually go through the registry that enforces it.
Access control and data security
The access half of a governance policy states which principals may perform which operations on which resources. Kafka expresses this natively as ACLs on topics, consumer groups and the cluster, and platform tooling adds role-based access control for what human operators may see and do. A complete policy covers both layers, plus multi-tenancy rules where several teams share one cluster.
For sensitive data the policy goes below topic level. Fields carrying PII are masked or encrypted at the producer before they are published, because Kafka does not encrypt data at rest itself. The scalable pattern for that per-field protection is envelope encryption. These controls matter most on change data capture topics, which mirror entire database tables into Kafka and therefore carry whatever sensitive data the source tables hold.
PayPal’s published account is the clean example of a policy closing a real gap: an earlier deployment carried a plaintext port, and removing it in favour of ACLs both improved the security posture and, less obviously, simplified capacity planning, because access rules made topic usage visible. Good access policy tends to pay a second dividend in operational clarity.
Lifecycle and retention management
The retention part of a governance policy states how long data lives and what happens when the limit is reached, and on Kafka it maps directly to topic configuration.
Kafka’s delete retention removes whole log segments once records age past a time limit or a partition grows past a size cap. Per the Apache Kafka topic configuration reference, the controls are: retention.ms for the time limit, retention.bytes for the per-partition size cap, and brokers default to a 168 hour retention window through log.retention.hours. A policy sets these per topic class, because a high-throughput topic with no size cap can exhaust broker disk inside its time window.
The alternative is log compaction. With cleanup.policy set to compact, Kafka retains the latest value for each record key instead of deleting by age, which suits topics that hold current state rather than an event history. A governance policy states which topic classes use which mode, and both can apply together.
For data that must outlive the retention window, the policy names the archival path, commonly a connector into object storage or a table format, so compliance retention does not mean holding everything on broker disk indefinitely.
The retention default is where I see this policy bite most often. Log retention on most installs defaults to 7 days, and if you are producing gigabytes per hour, gigabytes per hour for 7 days adds up to a lot of disk. On high-throughput topics I set retention.bytes as well as the time-based retention, so a topic’s on-disk size has a hard cap the policy can point to.
Data lineage and auditing
The final part of a governance policy is proof. Lineage records where data came from and where it went, and audit logs record who touched the platform and what they did.
Lineage on a streaming platform traces a record from its producing service, through topics and stream processing jobs, to its final sinks. The practical mechanism is metadata that travels with the data, such as mandatory record headers naming the producing service and schema version, paired with a catalog that accumulates the resulting lineage.
The audit side has a concrete checklist, laid out in our Kafka security architecture guide: a complete audit trail for Kafka captures authentication events, authorization decisions including denials, ACL changes, topic access per principal, and administrative operations. Denials matter as much as approvals, because a governance review asks not only who accessed the data but who tried to.
Data inspection is the unglamorous half of auditability, and it is the half that settles arguments. If I had a dollar for every lost-message complaint that ended with the message found sitting on the topic all along, I would have a lot more money than I do. In the incident that taught me this, the alerts were firing on a business logic error, workflows were stuck, and the developers’ complaint was that Kafka had lost messages. We inspected the topic and the messages were there. Kafka does not lose messages you have actually written to it. A governance policy that mandates inspection tooling turns that argument into a five-minute lookup.
FAQ
What are the four pillars of data governance?
Framework names vary, but on a streaming platform an enforceable policy consistently covers the four control areas on this page: schema enforcement and evolution, access control and data security, lifecycle and retention management, and data lineage with auditing.
What are the best practices for data governance?
Write every rule as something the platform can enforce: a registry compatibility level rather than a style guide, ACLs and RBAC rather than an access request wiki, retention.ms and retention.bytes per topic class rather than a cleanup rota, and mandatory record headers plus audit logging so the trail exists before anyone asks for it.
How do data platforms help with data governance?
The platform is where policy becomes enforcement: the schema registry rejects illegal changes at produce time, broker configuration applies retention automatically, ACLs decide access on every request, and management tooling adds the RBAC, masking and audit layers that native Kafka does not carry.