This article teaches you how to configure Kpow to restrict visibility of Kafka resources with Multi-Tenancy.

Kpow provides sophisticated Role Based Access Control to allow, deny, or stage user actions for any Kafka resource, to a group or topic level. However, for some of our users controlling the actions that a user can take wasn't quite enough.

"I have hundreds of topics and groups, showing users all of them is confusing. Can I restrict visibility of resources with RBAC?"

The best part of working on Kpow is understanding the needs of engineering teams who use Apache Kafka. On the face of it using RBAC to restrict user visibility as well control of resources is reasonable, but when we considered the broader idea we understood this is a bigger problem.

Introducing Multi-Tenancy

Kpow Multi-Tenancy allows you to assign user roles to one or more tenants.

Each tenant explicitly includes or excludes resources such as Kafka Clusters, Groups, Topics, Schema Registries and Connect Clusters.

A user role may be assigned multiple tenants, and a user with multiple tenants has the ability to easily switch between them.

When operating within a tenant a user can only see resources included by that tenant or create resources that would be valid within that tenant.

Importantly, users will see a fully consistent synthetic cluster-view of their aggregated resources. The overall user experience is simply of a restricted set of Kafka resources as if they were truly the only resources in the system.

Now our user with hundreds of groups and topics can configure views for different business units and provide a simplified Kafka experience to their users.

Tenants In Action

Let's start at the end, below you can see the Broker UI of two different tenants operating in the one Kpow instance:

• Global tenant is configured to contain all resources
• Transaction tenant is configured to contain only topics starting with tx_*

Global Tenant UI

We can see 233 topics in the global tenant.

Transaction Tenant UI

The transaction tenant only shows 200 topics, and they are much more uniform.

A user can switch between these two tenants if they have roles with each tenant assigned. Kpow continues to observe and control all attached Kafka resources, but provides a consistent view of synthetic clusters constructed of only the groups and topics included in each tenant. Aggregated metrics like write/s and total disk space can be seen in either view with different figures.

Uses of Multi-Tenancy

The primary intended use of Multi-Tenancy is for you to provide restricted views of Kafka resources to users from different teams in your organization.

However with the growth of Managed Kafka Services you may also want to configure basic tenants that exclude topics and groups of no regular interest.

Kpow stores all information regarding your Kafka resources in internal topics within your cluster, including an audit log of user actions. Kpow is also constructed of two Kafka Streams applications that run in unison to build the telemetry presented back to you.

A common user request has been to hide these internal topics and groups in the general UI as they're not of interest to our end users. Previously this had been a complicated task of in-place exclusion in the front-end, but aggregated metrics were hard to achieve.

If you have no tenants configured Kpow automatically provides two. A Global tenant that shows all attached Kafka resources and a Kpow Hidden tenant that hides Kpow resources and the consumer offsets topic.

You may want to provide tenants for specific business units, or you might just want to exclude internal topics from your cloud or managed service provider, or both!

Get Started

Multi-Tenancy is related to Role Based Access Control and both require User Authentication which is not available to trial users.

If you are on trial and would like to explore any of these features please request a license upgrade from sales@factorhouse.io.

Configuration

Within your RBAC yaml configuration file you can specify a top-level tenants key:

The following example configuration matches the default tenants that Kpow provides if you have none configured.

tenants:
  - name: "Global"
    description: "All configured Kafka resources."
    resources:
      - include:
          - [ "*" ]
    roles:
      - "*"
  - name: "Kpow Hidden"
    description: "All configured Kafka resources except internal Kpow resources and __consumer_offsets."
    resources:
      - include:
          - [ "*" ]    
      - exclude:
          - [ "cluster", "*", "topic", "oprtr*" ]
          - [ "cluster", "*", "topic", "__oprtr*" ]
          - [ "cluster", "*", "topic", "__consumer_offsets" ]
          - [ "cluster", "*", "group", "oprtr*" ]
    roles:
      - "*"

For more information about Multi-Tenancy see our online documentation, for help contact support@factorhouse.io.

This article introduces Kpow for Apache Kafka®'s new Temporary Policies feature.

Introducing Temporary Policies

Introduced to the Kpow Kafka Management and Monitoring toolkit in v79 is the ability to Stage Mutations , create Temporary Role Based Access Control Policies (temporary policies), and a suite of new admin features giving greater control over Kpow to Admin Users.

This blog post introduces temporary policies through the lense of a common real-world scenario.

Temporary policies allow Admins the ability to assign access control policies for a fixed duration. A common use-case would be providing a user TOPIC_INSPECT access to read data from a topic for an hour while resolving an issue in a Production environment.

Temporary Policies Use Case

You wake up one morning to a dreaded sight: a poison message has taken down one of your services.

Your team decides the simplest solution is to skip the message by incrementing your consumer group's offset for the topic.

Now here's the problem. Access to production is limited, and for such a simple action (incrementing the offset), a team member generally must jump through the hoops of configuring the VPN, connecting to the jumpbox, and making sure they execute the right combination of bash commands against the Kafka cluster.

Often these operations are unnecessarily time-consuming, brittle, and frustrating in a time-critical moment when you need to restore production access. Furthermore, the jumpbox generally has full access to the Kafka cluster, and there is no audit log recording the actions being committed.

In combination with Kpow's existing Role-Based Access Controls and powerful mutation actions, Temporary Policies improve this experience by giving teams the tools they need to easily effect change in a secured environment, like production, when things go wrong.

Configuring Role-Based Access Control

In this example, two roles are coming from our Identity provider: devs and owners.

We will assign anyone with the role owners admin access, and give them GROUP_EDIT access to the production cluster.

The devs role will be implicitly denied from undertaking any action against the cluster, but are authorized for read-only access to view the production cluster in Kpow.

Our example RBAC yaml file might look something like:

admin_roles:
  - "owners"

authorized_roles:
  - "owners"
  - "devs"

policies:
  -
    actions:
      - GROUP_EDIT
    effect: Allow
    resource:
      - "*"
    role: "owners"

This configuration prevents regular developers from making changes against the production cluster.

The Poison Pill

Today is the day when your team has to fix the consumer group on the production cluster.

Everyone has been briefed on the plan, and it has been decided that the team lead will temporarily grant the devs role Allow access for GROUP_EDIT. This will enable one of the developers on the team to make the required change to the production cluster.

This has been done through the Temporary Policies section of Kpow's settings UI:

Once a temporary policy has been created, team members can be notified via Slack with the Kpow Slack integration.

Incrementing the offset

A team member has been tasked with the job of incrementing the offset of the consumer group for the problematic topic.

The developer looks to the application logs and notices that it is partition 3 of topic tx_trade1 that contains the poison message.

The erroring consumer group is named trade_b2.

The developer then opens Kpow, navigates to the "Workflows" tab, and selects the consumer group.

From within the consumer group view, the dev clicks on the partition and selects "Skip Offset".

This action will schedule the mutation, and once someone on the team scales down the trade_b2 service, the offset will be incremented.

Post-Mortem

Kpow also provides valuable information and insights for teams to use after a production incident when you are completing your incident post-mortem.

Kpow has an Audit Log for Data Governance, and all the actions undertaken to resolve any production incident are persisted in Kpow's audit log topic. Meaning you can use the Audit Log to see the recorded history of all actions taken to restore the production service.

Inspecting the audit log message reveals the offset that was skipped.

You can use Kpow's data inspect functionality to view the poison message to help investigate why that message took down the consumer group.

You can find further information on setting up, viewing and managing temporary policies here.

Further reading/references

Explore our documentation to learn more about the Kpow's features mentioned in this article:

• Role-Based Access Control
• Temporary Policies
• Group Actions

You might also be interested in the following articles:

• Kafka Alerting with Kpow, Prometheus and Alertmanager

Manage, Monitor and Learn Apache Kafka with Kpow by Factor House.

We know how easy Apache Kafka® can be with the right tools. We built Kpow to make the developer experience with Kafka simple and enjoyable, and to save businesses time and money while growing their Kafka expertise. A single Docker container or JAR file that installs in minutes, Kpow's unique Kafka UI gives you instant visibility of your clusters and immediate access to your data.

Kpow is compatible with Apache Kafka+1.0, Red Hat AMQ Streams, Amazon MSK, Instaclustr, Aiven, Vectorized, Azure Event Hubs, Confluent Platform, and Confluent Cloud.

Start with a free 30-day trial and solve your Kafka issues within minutes.

This article dives into deploying Clojure projects to Maven.

Maven Central and Clojure

We have just released our first open-source consumable: kpow-streams-agent, a monitoring tool for Kafka Streams, to Maven Central.

The Kpow Streams Agent integrates your Kafka Streams topologies with Kpow, offering near-realtime monitoring and visualisation of your streaming compute:

We intend for this software to be used by the wider JVM ecosystem (eg, Java, Kotlin, Scala), so would like our library to be available on Maven Central.

There weren't many resources online documenting anyone's experience deploying Clojure-centric software to Maven Central and the few resources that I came across were out of date with the current requirements (as of June 2021).

This blog post documents the steps needed to make your Clojure code available to a wider audience.

Create a Sonatype account

The entire process for claiming your own namespace on Maven Central starts with creating a JIRA ticket. This seemed a bit archaic to us, coming from Clojars, NPM, and Crates.io backgrounds. Certainly, with these modern package managers, the integration between language/ecosystem/registry seems a lot more streamlined, thus publishing software much easier.

We weren't sure if this would be an automated process or if we would have to wait for a human to manually approve our request. We were relieved that it was indeed somewhat automated, with a bot automatically approving each step.

Sign up to JIRA

The first step is to create an account on the Sonatype JIRA.

The credentials you provide here will also be the same credentials you use to deploy, so keep that in mind as you proceed.

Create a new ticket

Once you have signed up for the JIRA, create a new ticket and choose the issue type "New Project".

This is where you claim your group.id on Maven Central. This must be related to a website domain you own. For us, we claimed io.operatr as our companies domain is operatr.io. You will need to prove ownership of your domain for the next section.

Add a TXT entry to your domain

Once you have submitted your JIRA ticket, a bot should automatically reply to your issue within a few minutes asking for verification of your domain. The simplest way to verify your domain is to create a TXT entry.

For example, if you use Cloudflare to manage your DNS records you could follow these steps to add a TXT entry to your domain.

You will need to create a TXT entry containing the Jira issue ID of your ticket (for example OSSRH-70400)

Once you have added the TXT entry to your domain, the bot should automatically reply and confirm that your group.id has been prepared

Deploy Requirements

You will now be granted the ability to deploy snapshot and release artifacts to s01.oss.sonatype.org for the group.id you have just registered.

All artifacts you deploy here are staged and can only be promoted to Maven Central if they meet the requirements.

This section will document how you can configure your project.clj to meet the Sonatype requirements

GPG Keys

Firstly, we want to create a GPG key to sign our release. Lein's GPG Guide is a good starting place on how you can do that.

Once you have created your GPG key you will need to upload your public key to a keyserver, such as https://keyserver.ubuntu.com/

In order to do this, you can export your GPG public key with the following command:

gpg --armor --export $MY_EMAIL

Credentials

Next, you will need to update your ~/.lein/credentials.clj file to include your Sonatype credentials:

{#"https://s01.oss.sonatype.org/.*" {:username "JIRA_USERNAME" :password "JIRA_PASSWORD"}}

Once you have created credentials.clj you will need to encrypt it:

gpg --default-recipient-self -e \
    ~/.lein/credentials.clj > ~/.lein/credentials.clj.gpg

Finally, you will need to setup the correct :deploy-repositories within your project's project.clj:

  :deploy-repositories [["releases" {:url   "https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/"
                                     :creds :gpg}
                         "snapshots" {:url   "https://s01.oss.sonatype.org/content/repositories/snapshots/"
                                      :creds :gpg}]]

Source and Javadoc Jars

It is a requirement to include both a -sources.jar and -javadoc.jar jar as part of your deployment.

These requirements are tailored more towards Java codebases than Clojure:

If, for some reason (for example, license issue or it's a Scala project), you can not provide -sources.jar or -javadoc.jar , please make fake -sources.jar or -javadoc.jar with simple README inside to pass the checking

You can create -sources.jar and -javadoc.jar jars by using a :classifiers key in lein:

  :classifiers [["sources" {:source-paths      ^:replace []
                            :java-source-paths ^:replace ["src/java"]
                            :resource-paths    ^:replace []}]
                ["javadoc" {:source-paths      ^:replace []
                            :java-source-paths ^:replace []
                            :resource-paths    ^:replace ["javadoc"]}]]

This specific example will bundle Java source code in the sources jar, and Java docs in the javadoc jar.

If you intend to create "fake" source jars, you could leave the source paths and resource paths empty.

Project Details

In order to meet the Project name, description and URL requirements, you will need to correctly populate the following keys in project.clj:

  :description "A Clojure project deployed to Maven"
  :url "https://github.com/org/repo"

License Information

In order to meet the License information requirement you will need to correctly populate the :license key in project.clj:

  :license {:name         "Apache-2.0 License"
            :url          "https://www.apache.org/licenses/LICENSE-2.0"
            :distribution :repo
            :comments     "same as Kafka"}

Developer Information

In order to meet the Developer Information requirement you will need to correctly structure your :pom-additions key in project.clj like so:

  :pom-addition ([:developers
                  [:developer
                   [:id "johnsmith"]
                   [:name "John Smith"]
                   [:url "https://mycorp.org"]
                   [:roles
                    [:role "developer"]
                    [:role "maintainer"]]]])

SCM Information

In order to meet the SCM Information requirement, you will need to correctly populate the :scm key in project.clj like so:

:scm {:name "git" :url "https://github.com/org/repo"}

Deploying to Central

If you have followed all of the steps from the previous section, you should have a lein project that meets all Sonatype requirements and is ready to be deployed!

You can do this via a regular:

lein deploy

Once you have deployed your artifacts, the next step is to log in to the Nexus Repository Manager at https://s01.oss.sonatype.org/. Again, your JIRA credentials from before are used to log in.

Once inside the, navigate to "Staging Repositories" - you should see an entry labeled XXX-1000.

Click on this item and verify that the contents you wish to deploy to Central are present.

If everything looks good, click the "Close" button. This will trigger the requirements check

If the requirements check passes, you will be able to press the "Release" button. Once you press this button, your release will shortly be synced with Maven Central!

Note : It can take up to 4 hours for the sync with https://search.maven.org/

Further Reading and References

The following resources might be useful for more information about deployments:

• Lein deployment docs
• Sample project.clj
• Sonatype requirements
• Releasing Sonatype artifacts

Manage, Monitor and Learn Apache Kafka with Kpow by Factor House.

We know how easy Apache Kafka® can be with the right tools. We built Kpow to make the developer experience with Kafka simple and enjoyable, and to save businesses time and money while growing their Kafka expertise. A single Docker container or JAR file that installs in minutes, Kpow's unique Kafka UI gives you instant visibility of your clusters and immediate access to your data.

Kpow is compatible with Apache Kafka+1.0, Red Hat AMQ Streams, Amazon MSK, Instaclustr, Aiven, Vectorized, Azure Event Hubs, Confluent Platform, and Confluent Cloud.

Start with a free 30-day trial and solve your Kafka issues within minutes.