Kafka interview questionsQuestion 2 of 2
Kafka interview question · Question 2 of 2
What is at-least-once delivery and what problems can it create?
Short answer
At-least-once delivery guarantees that every message is processed, but a message may be processed more than once. In Kafka it typically comes from processing a record and then committing its offset: if the consumer crashes after processing but before committing, it reprocesses that record on restart. The problem is duplicates, such as double-counted metrics or repeated side effects, so the processing and the sink must be idempotent, for example by upserting on a unique event id.
Detailed explanation
poll → process → write to sink → commit offset
↑ crash here → record is processed again after restartDuplicates also arise from producer retries (mitigated by idempotent producers, enabled by default in modern Kafka clients) and from rebalances.
Making duplicates harmless
- Idempotent writes: upsert or
MERGEon a unique event id. - Deduplication: keep a record of processed ids within a time window (streaming engines support watermark-bounded dedup).
- Side effects: record that an email or payment was sent, keyed by event id, before sending again.
Common mistakes
- Using auto-commit and assuming it gives exactly-once.
- Believing Kafka transactions make writes to an external database exactly-once.
- Counting events in a sink that appends blindly.
Progress is saved in this browser only. No account needed.