Overview
Apache Kafka is a cornerstone for many real-time data pipelines, but managing its infrastructure can be complex.
Google Cloud Managed Service for Apache Kafka offers a fully managed solution, simplifying deployment and operations, however effective monitoring and management remain crucial for ensuring the health and performance of Kafka clusters.
This article provides a practical, step-by-step guide on setting up a Google Cloud Managed Service for Apache Kafka cluster and connecting it from Kpow using the OAUTHBEARER mechanism. We will walk through creating the necessary GCP resources, configuring a client virtual machine, and deploying a Kpow instance using Docker Compose to demonstrate examples of monitoring and managing Kafka brokers and topics.
About Factor House
Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions forw Apache Kafka® and Apache Flink®.
Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.
Explore our live multi-cluster demo environment or grab a free Community license and dive into streaming tech on your laptop with Factor House Local.
Create a Managed Kafka Cluster
We create GCP resources using the gcloud CLI. Once it is initialised, we should enable the Managed Kafka, Compute Engine, and Cloud DNS APIs as prerequisites.
gcloud services enable managedkafka.googleapis.com compute.googleapis.com dns.googleapis.com
To create a Managed Service for Apache Kafka cluster, we can use the gcloud managed-kafka clusters create command by specifying the cluster ID, location, number of vCPUs (cpu), RAM (memory), and subnets.
export CLUSTER_ID=<cluster-id> export PROJECT_ID=<gcp-project-id> export PROJECT_NUMBER=<gcp-project-number> export REGION=<gcp-region> gcloud managed-kafka clusters create $CLUSTER_ID \ --location=$REGION \ --cpu=3 \ --memory=3GiB \ --subnets=projects/$PROJECT_ID/regions/$REGION/subnetworks/default \ --async
Set up a client VM
To connect to the Kafka cluster, Kpow must run on a machine with network access to it. In this setup, we use a Google Cloud Compute Engine virtual machine (VM). The VM must be located in the same region as the Kafka cluster and deployed within the same VPC and subnet specified during the cluster's configuration. We can create the client VM using the command shown below. We also attach the http-server tag to the VM, which allows HTTP traffic and enables browser access to the Kpow instance.
gcloud compute instances create kafka-test-instance \ --scopes=https://www.googleapis.com/auth/cloud-platform \ --tags=http-server \ --subnet=projects/$PROJECT_ID/regions/$REGION/subnetworks/default \ --zone=$REGION-a
Also, we need to update the permissions of the default service account used by the client VM. To ensure that the Kpow instance running on the VM has full access to Managed Service for Apache Kafka resources, bind the predefined admin role (roles/managedkafka.admin) to the service account. This grants Kpow the necessary administrative privileges. For more fine-grained access control within a Kafka cluster, it is recommended to use Kafka ACLs. The Enterprise Edition of Kpow provides robust support for it - see Kpow's ACL management documentation for more details.
gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:[email protected]" \ --role=roles/managedkafka.admin
Launch a Kpow Instance
Once the client VM has started, we can connect to it using the SSH-in-browser tool provided by Google Cloud. After establishing the connection, install the Docker Engine and the Docker Compose plugin, as they are required to deploy the Kpow instance using Docker Compose. Refer to the official installation and post-installation guides for detailed instructions.
Kpow's Docker Compose environment (kpow-local), provides resources for setting up Kpow instances. In this post, we use some of the existing configurations to simplify the deployment process. To get started, clone the GitHub repository and create a Docker Compose (docker-compose.yml) file with the following contents in the project's root directory. This Compose file defines a service that runs a Kpow instance and maps port 3000 inside the container to port 80 on the host machine. This makes the Kpow UI accessible via a browser at http://<vm-external-ip>, without needing to specify a port number. The application's authentication and authorisation settings are managed through the mounted JAAS and RBAC files, while its main configuration values are provided via an environment file specified in the env_file section.
version: '2.1'
services:
kpow:
image: factorhouse/kpow:latest # 94.2 or higher
pull_policy: always
restart: always
ports:
- "80:3000"
env_file:
- resources/kpow/gcp-trial.env
mem_limit: 2G
volumes: ## Enterprise edition only
- ./resources/jaas:/etc/kpow/jaas
- ./resources/rbac:/etc/kpow/rbac
Next, create a file (resources/kpow/gcp-trial.env) to store Kpow's main configuration values. We use the pre-configured authentication and authorisation settings, adding our own license details. The connection information for the managed Kafka cluster is included in the second section. The ENVIRONMENT_NAME is a display label used within Kpow to identify the Kafka environment, while the BOOTSTRAP value specifies the Kafka bootstrap server address, which Kpow uses to establish a connection. Connection security is managed through SASL over SSL, as indicated by the SECURITY_PROTOCOL value. The SASL_MECHANISM is set to OAUTHBEARER, enabling OAuth-based authentication. To facilitate this, the SASL_LOGIN_CALLBACK_HANDLER_CLASS is configured to use Google's GcpLoginCallbackHandler, which handles OAuth token management for Kafka authentication. Lastly, SASL_JAAS_CONFIG specifies the JAAS login module used for OAuth-based authentication.
## AuthN & AuthZ - Enterprise Edition Only JAVA_TOOL_OPTIONS=-Djava.security.auth.login.config=/etc/kpow/jaas/hash-jaas.conf AUTH_PROVIDER_TYPE=jetty RBAC_CONFIGURATION_FILE=/etc/kpow/rbac/hash-rbac.yml ## Managed Service for Apache Kafka Cluster Configuration ENVIRONMENT_NAME=GCP Kafka Cluster BOOTSTRAP=bootstrap.<cluster-id>.<gcp-region>.managedkafka.<gcp-project-id>.cloud.goog:9092 SECURITY_PROTOCOL=SASL_SSL SASL_MECHANISM=OAUTHBEARER SASL_LOGIN_CALLBACK_HANDLER_CLASS=com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler SASL_JAAS_CONFIG=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required; ## Your License Details LICENSE_ID=<license-id> LICENSE_CODE=<license-code> LICENSEE=<licensee> LICENSE_EXPIRY=<license-expiry> LICENSE_SIGNATURE=<license-signature>
To deploy the Kpow instance, run the following command using Docker Compose:
docker compose up -d
Monitor and Manage Resources
Once logged in using default credentials (e.g., admin as both username and password), we can follow the example below to connect Kpow to a managed Kafka cluster. This walkthrough covers monitoring brokers, creating a topic and producing a message, then reading that message through the Kpow UI.
Conclusion
By following the steps outlined in this post, we have successfully established a Google Cloud Managed Service for Apache Kafka cluster and deployed a Kpow instance on a Compute Engine VM. With this setup, we can immediately start exploring and managing Kafka brokers and topics, giving us valuable insights into our Kafka environment and streamlining operations.
Kpow is packed with powerful features, and it also integrates seamlessly with Kafka connectors deployed on Google Cloud Managed Kafka Connect clusters. This opens up a world of possibilities for managing data pipelines with ease. Stay tuned as we continue to roll out more integration examples in the future, enabling us all to unlock even more value from our Kafka and Kpow setups.