Batch versus stream processing

01When is the input complete?

A shop needs both yesterday's final sales report and a warning when suspicious purchases occur within seconds.

• A batch job processes a fixed input, such as yesterday's saved files or database snapshot, then finishes.
• A stream processor keeps handling an ongoing sequence of arriving sales and updates results as new records appear.

Batch can simplify large scans, checking totals and rerunning the same fixed input. Streaming provides earlier results but needs saved calculation state, progress tracking and rules for delayed data.

Neither is automatically complete or accurate. A batch misses sales if its source files are missing. A live total can look current while offline devices have not uploaded their sales. Streaming may use small internal batches; that implementation detail does not turn ongoing input into a final closed dataset.

02Time and state are part of correctness

A device records sale S9 at 12:00 but uploads it at 12:07. Separate the two clocks:

1. Event time is when the sale occurred according to its timestamp: 12:00.
2. Processing time is when the processor handles the record: around 12:07.
3. A window groups sales into an interval, such as 12:00–12:05. If the report means sales occurring then, late S9 belongs in that interval, not the interval when its upload arrived.
4. A watermark estimates how far event-time input has progressed. Passing 12:05 can let the processor publish a result under its assumptions, but does not prove an older device upload cannot appear later.
5. Declare late-data handling. Revise the original interval for an allowed period, create a separate adjustment, or close it under a stated rule. Tell users whether the number is preliminary or final.

Keep the same definition of a valid sale in both paths; otherwise comparing their totals does not reveal which one is wrong.

03Worked example: sales analytics

Assume 2,000 sales/s. A day contains 2,000 × 86,400 = 172.8 million events. At 200 bytes each, that is 34.56 GB/day raw, before copies, indexes and logs. Keep enough input to support required replay and checking.

Follow S9 through a five-minute revenue view:

1. Save the original sale facts. Retain sale ID, event time, amount and relevant version in replayable input.
2. The processor chooses S9's event-time window. It adds the sale to the corresponding preliminary total and writes a new result version.
3. S9 arrives at 12:07 for 12:00–12:05. For this example, accept corrections until ten minutes after the window ends, using the processor’s clock: 12:15 for this window. The 12:07 arrival therefore changes the original total. Do not count it as a sale occurring at 12:07.
4. A repeated S9 must not add revenue again. For a transactional output, a unique applied-sale/window record can be checked and saved with the total update. The processor checks the sale/window key; if absent, it inserts that key and increases the total in the same transaction. If present, it skips the increment. A unique key prevents concurrent attempts from both succeeding. Other processing engines need an equivalent rule covering both repeat recognition and the output change.
5. The worker crashes. Restore calculation state and its matching saved input position, then replay. An output already written just before the crash may repeat, so protect the destination too; checkpointing input alone is insufficient.

One million active shop/window keys at 200 bytes each need 200 MB of raw working state. Engine storage, checkpoints and overlapping windows add more. A small final chart does not imply a tiny processing system.

A nightly batch can check a closed reporting interval:

1. Read a saved stream checkpoint containing the interval’s total, the last included input position and its result version. That position is the batch’s input cutoff; that result version is its baseline version. Comparing different input cutoffs could mistake unprocessed sales for a counting error.
2. Recompute that interval from saved sales through the same cutoff. Give the correction a stable identity made from the interval, input cutoff and baseline version—not the checking job’s run ID.
3. In one database transaction, check whether that correction identity was already applied and whether the target total still has the baseline version and cutoff. If already applied, return the recorded result. If the target advanced, leave it unchanged and recompute against a newer checkpoint.
4. If both checks allow the change, replace the total with the checked total, increase its result version and save the applied-correction record together. Do not add the whole checked total as another sale. Two checking runs using the same baseline therefore cannot apply it twice.
5. Stream updates must use the same result-version check. If a correction changed their baseline, they reload the corrected total before adding the next sale; they cannot overwrite it with an old absolute total. The stream can continue processing other intervals while this check runs.

04Often the right answer combines them

Choose freshness and completeness deliberately:

• Use batch when periodic completion is acceptable and rerunning fixed input is valuable.
• Use streaming when prompt alerts or continuously changing results justify ongoing state and recovery work.
• Combine them when both paths use the same business definition: one gives an early view and the other checks completeness.
• Explain “exactly once” with participants. Saved input position, calculation state and outputs must agree. A transaction inside a message system does not automatically include an email or payment provider.
• Measure delay and drift. Track how far processing trails input, oldest unprocessed age, event-time progress, late arrivals and differences found during reconciliation.

In an interview, trace a normal sale, a late sale and a repeated sale after a crash. Name the actual record preventing double counting and when users may rely on the total as final.