Batch to Event-Driven Migration Patterns

Almost every enterprise still runs a nightly batch tier. Few have moved a workload to events without quietly rebuilding the batch semantics, badly, on top of a stream.

The request lands in roughly the same words every time. Leadership wants the 2 AM batch window gone, the four-hour reconciliation job replaced by something that reacts in seconds, and a data platform that feels live rather than a day behind. So a team stands up Kafka, points a connector at the database, and declares the batch job retired. Six weeks later the numbers on a downstream report disagree with the system of record, nobody can say why, and the old batch job gets switched back on "just to check." That parallel run becomes permanent. The technology worked. The guarantees did not survive the move.

The migration is a change of guarantees, not a change of tools

The framing that derails these projects is that batch to event-driven is a plumbing swap: retire the cron job, adopt a broker, done. It is not. A batch job is not just code that runs on a schedule. It is a bundle of implicit guarantees that the surrounding business logic has depended on for years, usually without anyone writing them down.

Think about what a nightly job actually hands you. It reads a consistent snapshot of the source, because the extract runs against a single point in time. It processes records in whatever order it chooses, because it holds the whole set. It has one obvious failure point and one obvious recovery: if the job dies, you rerun it, and rerunning is safe because the job overwrites yesterday's output. And it has a natural cutoff, because "yesterday's data" is a closed set by the time the job starts.

Event streams give you none of those without effort. You get a partitioned, append-only log with per-partition ordering, at-least-once delivery unless you work for exactly-once semantics, and events that arrive late, out of order, and forever. The migration that fails is the one that treats those four lost guarantees as details to sort out later. They are the actual work.

The counter-take: most of what you want to migrate should stay batch

Conventional modernization advice says event-driven is the mature end state and batch is legacy. We disagree, and we say so on the first call. A large share of the workloads teams try to convert were correct as batch and get worse as streams. Monthly billing close, regulatory reporting with a fixed cutoff, model training sets, any process whose correctness depends on operating over a complete, frozen set: these want a batch boundary, not a firehose. We routinely find that only 30% to 50% of a candidate migration backlog has a real latency requirement that batch cannot meet. The rest is being event-driven for fashion.

The counter-take we offer is to migrate the guarantee, not the job. Ask what latency the business actually needs. If the honest answer is "within an hour," the right first move is often incremental micro-batch triggered by an event, not a per-record stream: you keep the atomic-snapshot and rerun-safety properties of batch while cutting latency from a day to minutes. Full per-event streaming earns its complexity only when the requirement is seconds and the cost of a stale decision is high. Reserve it for those cases.

Four guarantees your batch job handed you at no cost

When a workload genuinely needs to move, name the four guarantees explicitly and decide how you will rebuild each one. Skipping any of them is how the numbers drift.

• The consistent snapshot. Batch read the source at one instant. A naive dual-write, where the application writes its database and then publishes an event, has no such instant: the two writes can diverge on any failure between them. Use the transactional outbox pattern instead. The application writes the business row and an event row in the same local transaction, and a change-data-capture reader ships the outbox. Debezium's outbox documentation describes the mechanism directly, and it is the single most important pattern for getting events out of a system of record without inventing inconsistency at the source.
• Ordering and a single retry point. Batch processed a whole set and retried the whole set. A log gives you ordering only within a partition. The Apache Kafka documentation is explicit that ordering is guaranteed per partition, not across a topic, so you must partition by the aggregate key whose order matters, the account, the customer, the order id, and accept that two different keys have no defined order between them. Get the partition key wrong and you have races that never showed up in batch.
• The natural cutoff. Batch had a closed set. Streams do not; events for "yesterday" keep arriving today. This is event-time processing, and it needs explicit watermarks and a stated policy for late data. The Apache Flink event-time documentation covers watermarks and lateness as first-class concepts, and the Dataflow Model paper (VLDB 2015) is the primary source for why you cannot get correctness, latency, and cost all at once on unbounded data. Decide what happens to an event that lands 3 hours after its window closed before you ship, not after a quarter-end number is wrong.
• Idempotency by rerun. Rerunning a batch job was safe because it overwrote its output. At-least-once delivery means a consumer will see some events more than once, so every consumer needs a dedup key and idempotent writes. This is not optional hardening you add if there is time. It is the price of the delivery model.

Patterns that survive contact with production

Beyond rebuilding the four guarantees, a handful of patterns separate migrations that finish from the ones that stall.

Run both and reconcile. Do not cut over. Keep the batch job running and stand the event pipeline up beside it, writing to a shadow output, for two to four weeks. Compare the two outputs continuously and set an explicit reconciliation tolerance, something like 99.9% row agreement, below which cutover does not happen. The reconciliation harness is not throwaway scaffolding. It is the artifact that tells you the new path is correct, and it is the thing most teams skip because it feels like extra work next to the exciting streaming code. It is the cheapest insurance in the project.

Keep the log and make replay a first-class operation. The best reason to move to a durable log is not lower latency. It is that the log lets you reprocess history by replaying it, which a batch pipeline could only do by rerunning against a source that may no longer hold the old state. Size retention for your real backfill needs, not the 7-day default, and rehearse a replay before you depend on one. A team that has never replayed its stream in anger does not actually have a replay capability.

Treat the schema at the seam as a contract. The event you publish is an interface, and every consumer downstream couples to it. A registry that rejects a breaking change at publish time turns a silent 3-week drift into a failed build the producer sees in a day. This is the same interface discipline that separates data platforms that compound from data platforms that calcify, and it matters more in streams than in batch because a bad event, once consumed, cannot be un-sent.

What this costs and how long it takes

For a single meaningful workload, moving from a nightly batch job to a production event-driven pipeline that has earned a cutover is a 3 to 6 month effort, not the two-sprint estimate that usually gets funded. The effort profile is lopsided in a way that surprises people: roughly 20% on the streaming machinery itself, the broker, the connectors, the topology, and the other 80% on the outbox, the dedup and idempotency work, the watermark and late-data policy, and the reconciliation harness. Teams that budget only for the 20% deliver a demo that works and a production system that drifts.

At portfolio scale the discipline is the same one strangler-fig migrations demand: sequence by workload, prove each cutover with reconciliation, and retire the batch job only when the shadow output has agreed for weeks. Retire it explicitly. A batch job left running "just in case" behind a live stream is not a safety net; it is a second source of truth that will eventually disagree with the first at the worst possible moment.

Where to start

1. Pick one workload with a real, stated latency requirement that batch cannot meet, and one that has caused visible pain. Do not pick an obscure job for a clean pilot; you will need the political capital the visible one earns.
2. Write down the four guarantees that workload's current batch job provides, and name how you will rebuild each one: outbox for the snapshot, partition key for ordering, watermark and late-data policy for the cutoff, dedup key for idempotency.
3. Stand up the event pipeline beside the batch job, not in place of it, writing to a shadow output. Set a numeric reconciliation tolerance for cutover before you write a line of consumer code.
4. Run both for two to four weeks, compare outputs daily, and fix every divergence to root cause rather than adjusting the tolerance to make the gap disappear.
5. Cut over only after the shadow output has held, then decommission the batch job on a dated plan with a named owner. A parallel run with no end date is a failed migration wearing a safety vest.