TL;DR — Message Queue Backlog Growing

You run an event processing pipeline for a fintech platform. Kafka cluster with 3 brokers, a topic 'transaction-events' with 12 partitions, and a consumer group of 4 instances processing payment events. Each event triggers fraud checks, ledger updates, and notification delivery. Normal throughput is 5K events/second with consumer lag under 1K messages. Key features: All transaction events must be processed (no data loss). Fraud checks must run on every transaction.

MEDIUM35 min

Message Queue Backlog Growing

consumer-scalingpartitioningdead-letter-queuebackpressuremonitoring

Key Points

All transaction events must be processed (no data loss)
Fraud checks must run on every transaction
Ledger updates must maintain strict ordering per account

Key Constraints

Kafka partitions

Consumer instances

4 (current)

Normal throughput

5K events/sec

Hints (0/3)

Canvas

Build your design

Drag components from the palette to build your solution for "Message Queue Backlog Growing"

Message Queue Backlog Growing

▶3D Simulate

Intermediate35 min read+100 XP

TL;DR — Message Queue Backlog Growing

MEDIUM35 min

Message Queue Backlog Growing

consumer-scalingpartitioningdead-letter-queuebackpressuremonitoring

Key Points

All transaction events must be processed (no data loss)
Fraud checks must run on every transaction
Ledger updates must maintain strict ordering per account

Key Constraints

Kafka partitions

Consumer instances

4 (current)

Normal throughput

5K events/sec

Hints (0/3)

Canvas

Build your design

Drag components from the palette to build your solution for "Message Queue Backlog Growing"