An audit screenshot shows full card numbers and customer emails in a Kafka topic browser, readable by every engineer with access. Kafka data masking is the control that replaces or redacts those sensitive field values before a person or downstream system sees them, and fixing it takes three steps: find, mask, prove.
Masking only the topic in the screenshot tends to miss the copies, because the same payload often also flows through retry topics, dead letter queues and change data capture streams. This page walks the full sequence, with the command or screen that settles each step. Masking is one layer of the four covered in Kafka security architecture for production, and it sits beside encryption rather than replacing it. The rest of the governance layer is covered in Kafka stream governance and the complete Kafka guide.
- Find the PII: detection methods
Finding PII means building a list of every topic and field that carries it, before choosing any masking tool. There are two ways to look, and a thorough pass uses both.
Start with the schemas. If your topics are schema-governed, the field names are the cheapest signal. List every subject in the registry and search the schemas for names like pan, card, email, phone, dob, ssn and address. This finds structured PII across hundreds of topics in minutes and costs nothing at runtime. It misses anything hidden inside a free-text field, a JSON string embedded in a field, or a topic with no schema at all.
Change data capture topics deserve a check of their own, because a CDC topic carries the columns of its source table, sensitive or not. As the complete guide to Kafka change data capture notes, CDC topics often carry customer PII and payment details.
Then sample the payloads. For topics without schemas, or fields like notes and description, read real records and test them against patterns. For a JSON topic, a bounded sample from the command line is enough to confirm or rule out a suspicion:
kafka-console-consumer.sh --bootstrap-server broker:9092 \
--topic payments.events --from-beginning --max-messages 10000 \
| grep -E -c '[0-9]{13,19}'A non-zero count of 13 to 19 digit runs is a lead, not proof, since order IDs match too. Binary formats such as Avro and Protobuf need a schema-aware consumer before any pattern match means anything. Tools that search message contents across topics are compared in Kafka message search tools. In Kpow, the same test runs as a kJQ filter in data inspect, which deserializes the record first and can search a nested field directly:
.value.payment.card | test("^[0-9]{13,19}$")For free text, use a PII detector. Regular expressions find card numbers and emails. They do not find a customer’s name typed into a support note. Presidio, an open-source project moving from Microsoft to the community-governed Data Privacy Stack organisation, is a context-aware detection and anonymization SDK for text that recognises entities such as credit card numbers, names, locations and social security numbers. Run it over sampled records offline to classify topics, rather than inline on every message, and treat its findings as input to your topic list.
Ruling a topic out is as useful as ruling it in. Record which topics you sampled and found clean, with the date, so the next audit starts from a known list rather than from zero.
- Choose where to mask: enforcement points
Where the mask runs decides what it protects against. There are five places it can go, and they are not interchangeable. Once you know which placement fits, the Kafka data masking tools comparison scores the products that implement each one.
In the producer, before the record is written. A serializer or the producing service itself removes, hashes or tokenises the field, so the value never reaches a broker. Every consumer sees the masked value and none can opt back in. This is the strongest option and the least flexible. When a downstream process genuinely needs the real value, the answer is encryption rather than masking, which what is envelope encryption? covers in depth.
In Kafka Connect, at ingestion or egress. Kafka Connect ships single message transformations that run per record inside the connector. The Apache Kafka documentation describes MaskField as replacing a field “with valid null value for the type (0, empty string, etc) or custom replacement”, and ReplaceField as able to filter fields out entirely:
transforms=maskPan
transforms.maskPan.type=org.apache.kafka.connect.transforms.MaskField$Value
transforms.maskPan.fields=card_number,cvvThis needs no application code, and it is the natural fix for a CDC topic that pulls in a column nobody downstream should see. Source: Apache Kafka Connect user guide.
In a stream processor, as a PII firewall. A Flink or Kafka Streams job reads a raw topic, masks or drops the fields, and writes a clean topic that general consumers use. ACLs then restrict the raw topic to the few services that need it. The cost is a second copy of the data and a job to operate, and the raw topic is still there for anyone who holds the right credential.
In a proxy between clients and brokers. A Kafka-protocol proxy can mask or encrypt fields as records pass through, without changing client code. Conduktor’s documentation, data masking page, points to Conduktor Gateway as its option for masking or encrypting the underlying data. The proxy sits in the data path for every client that uses it.
In the tool people use to read the data. The record on the topic is unchanged, and the UI applies the mask when an engineer inspects it. This protects against the exact finding in the audit screenshot, people browsing production data, and does nothing for applications that consume the topic. Kpow’s data policies, AKHQ’s and Kafbat UI’s masking settings, and Conduktor Console’s data masking all work at this layer.
The access model this makes possible has three states for any sensitive field: full access, masked access and no access. Most teams need all three, for different people on the same topic, and the enforcement point you pick decides whether you can have them.
Check the operational constraints
Before committing to an enforcement point, check it against the three constraints platform engineers reject solutions over.
Latency. Producer-side masking and Connect transforms add work per record in the write path. A proxy adds a network hop and processing for every request that passes through it. A stream-processing firewall adds a full processing stage before clean data is available. Read-time masking adds nothing to the produce or consume path, because it only runs when someone inspects data. Measure the option you choose on your own payloads before trusting any vendor figure.
Serialization. A mask that cannot parse the record cannot mask a field inside it. Check your formats against each tool’s documentation. Kpow’s data policies support Protobuf, Avro, JSON, Transit and EDN, and custom SerDes that produce JSON. Kafbat UI’s masking policies act on JSON object fields and fall back to masking or nulling the whole string for non-JSON values. AKHQ offers regex masking across all record values and keys, or JSON field masking with one filter per topic.
Access-level control. Can the mask depend on who is looking? Tools differ here, and it is worth reading the documentation closely. Conduktor’s documentation, data masking page, lets you exclude users or groups from a policy, so one group sees the real value on the same topic. AKHQ and Kafbat UI configure masking per topic pattern, not per viewer. Kpow’s data policies are scoped by cluster, topic and key, value or headers rather than by role, and it handles “who may see what” with RBAC instead: the TOPIC_INSPECT permission decides whether a role can read a topic at all, temporary policies grant that access for a fixed window, and a topic named in the policy file’s exclusions can be inspected unmasked by anyone allowed to inspect it.
Masking approaches compared
| Approach | Where it runs | Data on the topic | Code changes | Protects against | Main caveat | Source |
|---|---|---|---|---|---|---|
| Producer-side masking or tokenisation | Producing service or serializer | Masked before write | Yes, in every producer | Everyone downstream, including broker and disk access | Irreversible for consumers unless you tokenise | Your application code |
| Kafka Connect SMTs | Connector worker | Masked at ingestion, or at egress for sinks | Config only | Consumers of that pipeline | Only data passing through that connector | Apache Kafka docs |
| Stream-processing firewall | Flink or Kafka Streams job | Raw topic stays, clean copy written | A job to build and run | Consumers of the clean topic | Raw topic still needs tight ACLs | Your stream job |
| Kafka-protocol proxy | Between clients and brokers | Unchanged, or encrypted by the proxy | None for clients using the proxy | Clients routed through the proxy | Clients that bypass it see raw data | Vendor documentation |
| Read-time masking in a UI | The inspection tool | Unchanged | None | People browsing data in that tool | Applications and other tools see raw data | Kpow data policies, Kafbat UI, AKHQ |
A common production pattern combines two rows: producer-side masking or envelope encryption for the fields no consumer should ever see, and read-time masking so engineers can debug production without reading the fields that remain. The audit finding that started this page is almost always a read-time problem, and it is the fastest one to close.
- Prove the masking holds
An auditor will not accept “we configured a policy”. Prove it three ways.
Test the rule against real record shapes. Every masking tool behaves differently when a field’s shape changes. Kpow’s documentation is explicit that it is conservative: when a redaction such as show-last-four meets a structured value it cannot apply to, it falls back to full redaction of that field. Kpow’s data policy sandbox, under Admin, Data policies, lets you test a policy against sample data before rollout. Whatever tool you use, test the policy against a record where the sensitive field is nested, null, and a different type from the one you wrote the rule for.
Try to get around it. The usual bypass for read-time masking is choosing a different deserializer and reading the raw bytes. Kpow removes String SerDes from data inspect when data policies are configured, for exactly this reason, and its docs say so. Check your tool the same way. Then check who holds credentials that can read the topic directly, since read-time masking does not apply to them:
kafka-acls.sh --bootstrap-server broker:9092 --list \
--topic payments.events --resource-pattern-type matchThe match pattern type lists literal, wildcard and prefixed ACLs that affect the topic, which a plain --list misses.
Keep a record of who looked. Masking reduces what people see, and an audit log shows what they did see. The options for that record are compared in Kafka audit logging tools. Kpow’s webhook can send data inspect queries as well as changes to your SIEM when its verbosity is set to include queries, and TD’s platform team described in their talk with us how they grant inspect access for an hour or two through ServiceNow, noting that “This gives TD an audit trail.” For why the auditor asked in the first place, PCI SSC’s glossary defines masking as a “method of concealing a segment of PAN when displayed or printed”, used “when there is no business need to view the entire PAN”, and GDPR Article 32 names pseudonymisation and encryption among the measures for security of processing (PCI SSC glossary, GDPR on EUR-Lex).
How Kpow masks PII at inspection time
Kpow applies data policies to data inspect and ksqlDB query results on the server, so the masked value is what reaches the browser. Policies live in a YAML file set by DATA_POLICY_CONFIGURATION_FILE. This example, from the documentation, shows the last four digits of any field named credit_card, creditcard or pan on every topic:
policies:
- name: Credit Card
category: PII
resources:
- ["cluster", "*", "topic", "*", "value"]
redaction: ShowLast4
type: non-scalar
fields: [credit_card, creditcard, pan]Redactions include full masking, SHA-512 hashing, show-email-host, and show-first or show-last of one to six characters, and they apply inside nested maps and collections in keys, values and headers. Belong, part of Telstra, gave this as one of four reasons they chose Kpow: Nagaraj Ballapuram Gopal, their Head of Enablement, said in his talk with us that it lets them mask fields like mobile number, first name, last name or address, which “helped a lot with our cyber security and risk teams”.
Where Kpow does not fit: read-time masking protects people using Kpow, not applications consuming the topic, so pair it with producer-side controls for fields no consumer needs. Its policies are keyed to resources rather than roles, so per-viewer differences come from RBAC and topic exclusions as described above. For how masking fits the wider regulatory picture, see Kafka governance tools for financial services. For combining masking with tenancy and RBAC across teams on shared clusters, see Kpow multi-tenancy.
To try the find step before configuring anything, open the Kpow demo and run a kJQ filter like the one in step 1 against its live topics, then compare the results with how the redaction examples above would render them.
Product demo · 2 min
Apache Kafka data masking & PII protection: Kpow demo
Chad Harris walks through data masking in Kpow: last-four, full, and email-domain masking rules applied during data inspection, and the data policy playground for testing redaction rules like show-first and hashing before rolling them out.
Kpow live demo
Test a masking policy in the demo
Explore Kpow's data inspection in a live environment, the surface where its data policies redact sensitive fields at view time.
For platform and security teams who have to show who can do what.
Try the Kpow demoFAQ
Does Kafka have built-in data masking?
Not at the broker. The closest built-in mechanism is Kafka Connect’s MaskField transform, which masks named fields as records pass through a specific connector. Masking across every reader of a topic needs producer-side changes, a stream-processing job, a proxy, or a tool that masks at read time.
What is the difference between masking and encryption in Kafka?
Masking changes what a reader sees and is often irreversible. Encryption protects the bytes so only key holders can read them, and it is reversible for whoever holds the key. Regulated platforms typically use encryption for data at rest and in transit, and masking for people who need to inspect records.
How do I find PII in Kafka topics without slowing the cluster?
Scan schemas first, which costs nothing at runtime, then sample bounded sets of records with a consumer or a UI filter rather than scanning every message inline. Use an NER-based detector such as Presidio offline for free-text fields.
Can I mask Kafka data differently for different users?
Some tools can. Conduktor Console lets you exclude users or groups from a masking policy. In Kpow, masking is set per resource and RBAC decides who can inspect a topic at all, with temporary policies for time-boxed access and exclusions for topics that should be readable unmasked.