Kafka + Flink streaming blueprints

Real-time streaming systems are getting a clearer playbook: use Kafka as the event log, then use Apache Flink to keep state and score each event as it arrives. (nightlies.apache.org) Kafka is the durable queue in that setup, a running ledger that can retain and replay events after failures. Flink is the compute layer on top, designed for “stateful computations over unbounded and bounded data streams,” which means it can remember prior events while new ones keep coming. (nightlies.apache.org) (flink.apache.org) That memory matters in fraud-style pipelines, where one transaction is rarely enough to decide anything. Flink’s checkpointing system saves operator state and stream positions so a job can recover “with the same semantics as a failure-free execution” if a task or machine dies. (nightlies.apache.org) The reliability target in these designs is usually “exactly once,” meaning each event is processed one time even during retries or failover. Kafka documents three delivery levels — at most once, at least once, and exactly once — and frames exactly once as the mode that avoids both loss and duplicate reads. (docs.confluent.io) That guarantee depends on more than one switch. Kafka’s producer API says retries can create duplicates unless idempotence is enabled, and transactional producers extend that model so a stream processor can commit output in step with recovery. (kafka.apache.org) (docs.confluent.io) On the Flink side, state backend choices decide how much history a job can hold and how fast it can touch it. The EmbeddedRocksDBStateBackend stores state as serialized bytes in a local RocksDB database, supports very large state that can spill beyond memory, and always takes asynchronous snapshots. (nightlies.apache.org) That tradeoff is explicit in the Flink docs: RocksDB can keep far more state than heap memory backends, but reads and writes are slower because every access goes through serialization and may hit disk. Teams building low-latency scoring systems usually accept that cost when long windows, keyed joins, or session histories would otherwise overflow memory. (nightlies.apache.org) The same blueprint fits sensor and vessel feeds because those systems also need replay, ordering, and stitched history. Flink’s checkpointing docs say durable sources such as Kafka are a prerequisite for recovery, which is the core requirement when operators need to rewind a stream, rebuild state, and reproduce prior outputs after a fault. (nightlies.apache.org) The open question is latency versus guarantees. Confluent’s Flink documentation says exactly-once delivery in its cloud service is currently dominated by Kafka transaction commit intervals and lands at roughly one minute, while at-least-once can run much faster, so teams chasing sub-50 millisecond model scoring often separate online inference latency from slower end-to-end commit guarantees. (docs.confluent.io) What the recent engineering guides add is not a new engine, but a more concrete operating pattern: durable ingress, replayable logs, checkpointed state, and duplicate-safe writes. For teams handling fraud alerts, industrial telemetry, or Automatic Identification System tracks, that is the difference between a fast demo and a stream that can survive failure. (docs.confluent.io) (nightlies.apache.org)