Write data from Kafka to BigQuery by using Dataflow

This page shows how to use Dataflow to read data from Google Cloud Managed Service for Apache Kafka and write the records to a BigQuery table. This tutorial uses the Apache Kafka to BigQuery template to create the Dataflow job.

Overview

Apache Kafka is an open source platform for streaming events. Kafka is commonly used in distributed architectures to enable communication between loosely coupled components. You can use Dataflow to read events from Kafka, process them, and write the results to a BigQuery table for further analysis.

Managed Service for Apache Kafka is a Google Cloud service that helps you run secure and scalable Kafka clusters.

Reading Kafka events into BigQuery
Event-driven architecture using Apache Kafka

Required permissions

The Dataflow worker service account must have the following Identity and Access Management (IAM) roles:

  • Managed Kafka Client (roles/managedkafka.client)
  • BigQuery Data Editor (roles/bigquery.dataEditor)

For more information, see Dataflow security and permissions.

Create a Kafka cluster

In this step, you create a Managed Service for Apache Kafka cluster. For more information, see Create a Managed Service for Apache Kafka cluster.

Console

  1. Go to the Managed Service for Apache Kafka > Clusters page.

    Go to Clusters

  2. Click Create.

  3. In the Cluster name box, enter a name for the cluster.

  4. In the Region list, select a location for the cluster.

  5. Click Create.

gcloud

Use the managed-kafka clusters create command.

gcloud managed-kafka clusters create CLUSTER \
--location=REGION \
--cpu=3 \
--memory=3GiB \
--subnets=projects/PROJECT_ID/regions/REGION/subnetworks/SUBNET_NAME

Replace the following:

  • CLUSTER: a name for the cluster
  • REGION: the region where you created the subnet
  • PROJECT_ID: your project ID
  • SUBNET_NAME: the subnet where you want to deploy the cluster

Creating a cluster usually takes 20-30 minutes.

Create a Kafka topic

After the Managed Service for Apache Kafka cluster is created, create a topic.

Console

  1. Go to the Managed Service for Apache Kafka > Clusters page.

    Go to Clusters

  2. Click the name of the cluster.

  3. In the cluster details page, click Create Topic.

  4. In the Topic name box, enter a name for the topic.

  5. Click Create.

gcloud

Use the managed-kafka topics create command.

gcloud managed-kafka topics create TOPIC_NAME \
--cluster=CLUSTER \
--location=REGION \
--partitions=10 \
--replication-factor=3

Replace the following:

  • TOPIC_NAME: the name of the topic to create

Create a BigQuery table

In this step, you create a BigQuery table with the following schema: