Real-Time CDC Pipelines: Debezium + Kafka Production Patterns
CDC is the most-used least-documented streaming pattern. The schema, the failure modes, and what actually scales past 50k events/sec.
Change Data Capture is the most-used least-documented streaming pattern at substantial enterprises. The substantial appeal: capture row-level changes from operational databases (Postgres, MySQL, SQL Server, Oracle, MongoDB) and stream them to substantial downstream systems (data warehouses, search indexes, caches, event-driven services). Debezium + Kafka is the substantial dominant open-source pattern. Production deployments at substantial scale (50K+ events/sec) require substantial discipline. This post walks through what we’ve learned across substantial client engagements.
What CDC enables#
The substantial CDC value propositions:
Real-time data warehouse sync. Substantial near-real-time replication from operational DB to warehouse.
Search index synchronization. Updates in operational DB substantially flow to Elasticsearch/OpenSearch/similar.
Cache invalidation. Cache layers substantially update from CDC events rather than TTL-based.
Event-driven architectures. CDC events trigger substantial downstream services.
Data lake landing. Substantial historical change data lands in data lake for analytics.
Cross-region replication. Substantial geographic replication via CDC.
The Debezium stack#
Substantial Debezium architecture:
Source connectors. Substantial database-specific connectors — Debezium Postgres, MySQL, SQL Server, Oracle, MongoDB, Cassandra, plus the various.
Kafka Connect. Runtime hosting connectors; substantial operational substrate.
Apache Kafka. Event streaming platform; substantial event log.
Schema Registry. Confluent Schema Registry or alternatives; substantial schema management.
Sink connectors. Substantial database/system-specific connectors for downstream — Snowflake, Elasticsearch, S3, plus the various.
Substantial transformation. Single Message Transforms (SMTs) for substantial inline transformation.
The substantial production patterns#
Several patterns from substantial production deployments:
Snapshot plus streaming. Initial snapshot of database state plus continuous streaming. Substantial coordination needed.
Topic-per-table. Substantial standard pattern — one Kafka topic per source table. Substantial scaling considerations.
Substantial schema evolution. Source schema changes propagate; substantial discipline matters.
Substantial idempotency. Downstream consumers handle duplicate events; substantial design requirement.
Substantial ordering preservation. Per-partition ordering preserved; cross-partition ordering not guaranteed.
Substantial Outbox pattern. Application writes to outbox table; CDC publishes outbox events. Substantial transactional event publication.
Substantial dead letter queues. Failed events route to DLQ for substantial later investigation.
The substantial scaling considerations#
Several substantial considerations at scale:
Source DB impact. Substantial CDC adds load to source DB. Replication slot management on Postgres, binlog reading on MySQL, plus the various — substantial discipline.
Connector resource sizing. Substantial Kafka Connect workers need substantial sizing for throughput.
Substantial throughput limits. Single Debezium connector practical limit ~10-50K events/sec depending on database. Substantial sharding required beyond.
Substantial network bandwidth. Substantial events × substantial size = substantial bandwidth.
Substantial Kafka topic partitioning. Substantial partitioning for parallelism; substantial design matters.
Substantial backpressure. Substantial downstream consumers slower than producers; substantial buffering matters.
The substantial failure modes#
Several substantial production failure modes:
Replication slot bloat on Postgres. Substantial unconsumed slots cause substantial WAL accumulation; substantial database disk fills.
Substantial MySQL binlog retention. Substantial binlogs retained until consumed; substantial disk pressure.
Substantial schema drift. Source schema changes not anticipated by downstream consumers; substantial breakage.
Substantial event reordering. Substantial across-partition events arrive out of order; substantial downstream confusion.
Substantial transaction boundaries. Substantial multi-table transactions appear as substantial separate events; substantial atomicity lost.
Substantial connector failures. Substantial connectors restart; substantial event reprocessing.
Substantial schema registry issues. Substantial schema registry problems substantially break entire pipeline.
The substantial operational requirements#
Substantial production CDC operations require:
Substantial monitoring. Connector health, lag, throughput, error rates.
Substantial alerting. Replication slot bloat, lag thresholds, connector failures.
Substantial schema management. Schema evolution discipline.
Substantial documentation. Connector configuration, downstream contracts.
Substantial substantial DR planning. Substantial disaster recovery for connectors and Kafka.
Substantial substantial substantial cost monitoring. Substantial substantial CDC pipelines at scale have substantial cost.
The substantial alternatives#
Several substantial alternatives to Debezium + Kafka:
Fivetran with CDC. Substantial managed offering; substantial cost; substantial reduced operational burden.
Airbyte with CDC. Comparable to Fivetran with substantial open-source option.
AWS DMS. AWS-anchored substantial managed CDC.
Azure Database Migration. Comparable Azure offering.
Database-native replication. Substantial Postgres logical replication, MySQL replication, plus the various.
Cloud-vendor proprietary — Snowpipe Streaming, BigQuery streaming inserts, plus the various.
The decision framework#
For most teams in 2026:
Pick Debezium + Kafka when you have substantial CDC volume and substantial operational capability.
Pick Fivetran/Airbyte when managed simplicity matters more than cost; substantial common at smaller-to-mid scale.
Pick database-native replication for simple replication scenarios within same database family.
Pick AWS DMS / Azure DMS for cloud-anchored migration scenarios.
What we typically see at clients#
Common patterns:
Debezium + Kafka at substantial scale. Default for substantial enterprises with substantial operational capability.
Fivetran for smaller deployments. Substantial common managed choice.
Mixed deployments. Substantial enterprises with multiple CDC paths for substantial different sources.
Substantial operational challenges. Substantial production issues from substantial schema drift, replication slot bloat, plus the various.
Where pdpspectra fits#
Our data engineering practice builds production CDC pipelines with substantial operational discipline.
Related reading: the streaming SQL post, the Pulsar post, and the Fivetran vs Airbyte vs custom ELT post.
Production CDC is substantial operational discipline. Talk to our team about your data pipeline architecture.