Migrate from Kafka to Pub/Sub Lite

This document is useful if you're considering migrating from self-managed Apache Kafka to Pub/Sub Lite.

Overview of Pub/Sub Lite

Pub/Sub Lite is a high-volume messaging service built for low cost of operation. Pub/Sub Lite offers Zonal and Regional Storage along with pre-provisioned capacity. Within Pub/Sub Lite, you can choose zonal or regional Lite topics. Regional Lite topics offer the same availability SLA as Pub/Sub topics. However, there are reliability differences between Pub/Sub and Pub/Sub Lite in terms of message replication.

To learn more about Pub/Sub and Pub/Sub Lite, see What is Pub/Sub.

To learn more about Lite-supported regions and zones, see Pub/Sub Lite locations.

Terminology in Pub/Sub Lite

The following are some key terms for Pub/Sub Lite.

  • Message. Data that moves through the Pub/Sub Lite service.

  • Topic. A named resource that represents a feed of messages. Within Pub/Sub Lite, you can choose to create a zonal or regional Lite topic. Pub/Sub Lite regional topics store data in two zones of a single region. Pub/Sub Lite zonal topics replicate data within just one zone.

  • Reservation. A named pool of throughput capacity shared by multiple Lite topics in a region.

  • Subscription A named resource that represents an interest in receiving messages from a particular Lite topic. A subscription is similar to a consumer group in Kafka that only connects to a single topic.

  • Subscriber. A client of Pub/Sub Lite that receives messages from a Lite topic and on a specified subscription. A subscription can have multiple subscriber clients. In such a case, the messages are load-balanced across the subscriber clients. In Kafka, a subscriber is called a consumer.

  • Publisher. An application that creates messages and sends (publishes) them to a specific Lite topic. A topic can have multiple publishers. In Kafka, a publisher is called a producer.

Differences between Kafka and Pub/Sub Lite

While Pub/Sub Lite is conceptually similar to Kafka, it's a different system with a narrower API that is more focused on data ingestion. While the differences are immaterial for stream ingestion and processing, there are some specific use cases where these differences are important.

Kafka as a database

Unlike Kafka, Pub/Sub Lite doesn't currently support transactional publishing or log compaction, although idempotence is supported. These Kafka features are more useful when you use Kafka as a database than as a messaging system. If you use Kafka primarily as a database, consider running your own Kafka cluster or using a managed Kafka solution such as Confluent Cloud. If neither of these solutions are an option, you can also consider using a horizontally scalable database such as