This minor release from Factor House introduces BYO AI model support, topic data inference, and major enhancements to data inspect—such as AI-powered filtering, new UI elements, and AVRO date formatting. It also adds integration with GCP Managed Kafka Schema Registry, improved webhook support, updated policy handling, and several UI and error-handling improvements.
BYO AI
Kpow now offers optional integrations with popular AI models, from Ollama to enterprise solutions like Azure OpenAI. These features are entirely opt-in: unless you configure an AI provider, Kpow will not expose any AI functionality.
As of release 94.3, Kpow supports integrations with:
- Azure OpenAI
- AWS Bedrock
- OpenAI
- Anthropic
- Ollama
You can configure one or more AI models, and set a default model in your user preferences for use with Kpow’s AI-powered features.
These AI models power all AI-powered features in Kpow. Read the documentation for more details.
kJQ filter generation
Transform natural language queries into powerful kJQ filters with AI-assisted query generation. This feature empowers users of all technical backgrounds to extract insights from Kafka topics without requiring deep JQ programming knowledge.
How it works
Simply describe what you're looking for in plain English, and the AI model generates a syntactically correct kJQ filter tailored to your data. The system leverages:
- Natural language processing: Convert conversational prompts like "show me all orders over $100 from the last hour" into precise kJQ expressions.
- Schema-aware generation: When topic schemas are available, the AI optionally incorporates field names, data types, and structure to create more accurate filters.
- Validation integration: Generated filters are automatically validated against Kpow's kJQ engine to ensure syntactic correctness before execution.
Usage
Navigate to any topic's Data Inspect view and select the AI Filter option. Enter your query in natural language, and Kpow will generate the corresponding kJQ filter. You can then execute, modify, or save the generated filter for future use.
The AI filter generator works best when provided with specific, actionable descriptions of the data you want to find. Include field names, value ranges, example data and logical operators in your natural language query for optimal results.
Auto deserializers
Within Kpow's Data Inspect UI, you can specify Auto as the key or value deserializer and Kpow will attempt to infer the data format and deserialize the records it consumes.
Auto SerDes provides immediate data inspection capabilities without requiring prior knowledge of topic serialization formats. This feature is particularly valuable when:
- Exploring unfamiliar topics for the first time
- Working with multiple topics that may contain mixed data formats
- Debugging serialization issues across different environments
- Onboarding new team members who need quick topic insights
The Auto SerDes option appears alongside manually configurable deserializers like JSON, Avro, String, and custom SerDes in the Data Inspect interface.
When selected, Kpow analyzes each topic and applies the most appropriate deserializer automatically.
Topic inference observation
To persist and query inferred topic information—such as key deserializer, value deserializer, and schema registry ID—in Kpow’s UI, enable the Topic SerDes Observation job by setting:
INFER_TOPIC_SERDES=true
kJQ language improvements
In response to our customers' evolving filtering needs, we've significantly improved the kJQ language to make Kafka record filtering more powerful and flexible. Check out the updated kJQ filters documentation for full details.
Below are some highlights of the improvements:
Chained alternatives
Selects the first non-null email address and checks if it ends with ".com":
.value.primary_email // .value.secondary_email // .value.contact_email | endswith(".com")
String/Array slices
Matches where the first 3 characters of transaction_id equal TXN:
.value.transaction_id[0:3] == "TXN"
For example, { "transaction_id": "TXN12345" } matches, while { "transaction_id": "ORD12345" } does not
UUID type support
kJQ supports UUID types out of the box, including the UUID deserializer, AVRO + logical types, or Transit / JSON and EDN deserializers that have richer data types.
To compare against literal UUID strings, prefix them with #uuid to coerce into a UUID:
.key == #uuid "fc1ba6a8-6d77-46a0-b9cf-277b6d355fa6"
Data inspect improvements
UI overhaul
We've enhanced the data inspect UI with smaller toolbar buttons and reorganized dropdowns to accommodate the new features detailed below.
Event log
All queries display the 'Event log' when anomalies occur during search, such as data policy applications, deserialization errors, or record processing exceptions. Each event includes a timestamp and a severity level (INFO, WARN, or ERROR).
Send to kREPL
Customers can now send data inspect queries directly to the kREPL, our programmatic Kafka interface designed for power users. The kREPL enables notebook-style data transformations—such as aggregations and groupings—on consumed data.
You can access this feature through the 'Send' dropdown in the result metadata toolbar. Learn more about the kREPL by visiting our documentation.
GCP managed Kafka schema registry integration
Kpow now supports the Google Schema Registry as a varient of Confluent-compatible schema registry. Here is an example configuration.
ENVIRONMENT_NAME=GCP Kafka Cluster BOOTSTRAP=bootstrap.<cluster-id>.<gcp-region>.managedkafka.<gcp-project-id>.cloud.goog:9092 SECURITY_PROTOCOL=SASL_SSL SASL_MECHANISM=OAUTHBEARER SASL_LOGIN_CALLBACK_HANDLER_CLASS=com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler SASL_JAAS_CONFIG=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required; SCHEMA_REGISTRY_NAME=GCP Schema Registry SCHEMA_REGISTRY_URL=https://managedkafka.googleapis.com/v1/projects/<gcp-project-id>/locations/<gcp-region>/schemaRegistries/<registry-id> SCHEMA_REGISTRY_BEARER_AUTH_CUSTOM_PROVIDER_CLASS=com.google.cloud.hosted.kafka.auth.GcpBearerAuthCredentialProvider SCHEMA_REGISTRY_BEARER_AUTH_CREDENTIALS_SOURCE=CUSTOM
Additional webhooks
We now support both Microsoft Teams and a generic HTTP webhook call. When configured, Kpow can send Data governance (Audit log) records to your Slack, all you need to do is configure a webhook.
Temporary policy improvements
Administrators can now set a custom expiration date and time when creating temporary policies. Selecting Custom in the duration dropdown activates a mandatory datetime picker that only allows future dates.
Configuration options include the TEMPORARY_POLICY_MAX_MS environment variable to control the maximum policy duration (default: 1 hour). Setting this to -1 removes the duration limit.
Discover more about these updates in our documentation.
Helm charts
Artifact Hub updates
- Added values schema support for improved chart validation
- Charts are now signed for enhanced security and trust
- Publisher identity is verified to ensure authenticity
- Charts are marked as official for trusted, curated content
Container security improvements
The container security context has been tightened by disabling privilege escalation (AllowPrivilegeEscalation=false), running the container as a non-root user (UID 1001), disallowing privileged mode, and dropping all Linux capabilities.