Batch Processing
Batch processing is the technique of collecting and processing large volumes of data together at scheduled intervals rather than processing each record individually as it arrives.
Definition
Batch processing groups multiple records or transactions together and processes them as a single unit on a scheduled basis—typically nightly, hourly, or weekly. Instead of handling each customer order, log entry, or sensor reading individually as it arrives, a batch job collects thousands or millions of records and processes them efficiently in one pass. Batch processing is economical for large-volume work, simpler to implement than real-time processing, and ideal when freshness can be delayed by hours or days. Most traditional data warehouses and ETL jobs operate on batch schedules.
How It Works
1. Collect: Data accumulates in a queue, file, or staging table. 2. Schedule: At the designated time (e.g., 2 AM), the batch job starts. 3. Process: All accumulated records are processed together, often leveraging parallelization. 4. Load: Results are written to the destination. 5. Report: Logs capture success and any errors; alerts fire if the job fails.
When to Use It
Use batch processing for daily reports, nightly ETL jobs, end-of-month accounting, or any work where delayed freshness (hours to days) is acceptable. Batch is cost-effective for large volumes and simpler than real-time streaming. It's less suitable for user-facing analytics, fraud detection, or operational systems requiring sub-second latency.
Last updated: Jun 17, 2026