Why a large export should be a job Exporting millions of records inside one HTTP request can exceed web server timeouts and memory limits. A background exp...
Why a large export should be a job
Exporting millions of records inside one HTTP request can exceed web server timeouts and memory limits. A background export lets the user request the report, see its progress, and download a completed file later. The export must still apply authorization and data filters at execution time.
Define the requested columns, filter snapshot, timezone, output format, owner, and retention period. A filter based on "last month" should be resolved to exact timestamps when the request is created so a delayed job does not change the result unexpectedly.
Process records in bounded batches
Use a query scoped to the user's tenant and permissions. Select only needed columns and stream rows to a temporary file. Chunking or a cursor can bound memory, but choose a stable ordering key. If records change during the export, define whether the report is a snapshot or a best effort live view.
Do not call get() on the full dataset and then paginate the in-memory collection. That defeats the purpose of background processing.
Protect the file
Store the output privately and provide a short lived authorized download route or signed URL. The job's owner and tenant should be checked again before download. Apply retention and cleanup, since exports can contain more sensitive data than the ordinary UI.
For CSV, protect against spreadsheet formula injection when values may start with formula characters. A CSV file is data, but spreadsheet applications may interpret cells as formulas. Test exports in the actual target spreadsheet application.
Track progress and failure
Record queued, processing, completed, failed, and expired states. Expose a useful message if the report is too large or the filter is invalid. On retry, use a unique operation ID and replace or resume the temporary artifact safely; do not create multiple public files for one request.
Test with realistic volume
Test a report large enough to reveal memory issues, a worker restart, a revoked user, a tenant switch, download after expiry, and an export containing commas, newlines, Unicode, and formula-like values. Confirm the generated count and totals against an independent query.
Conclusion
Background exports require bounded reads, durable status, private storage, and a clear freshness definition. They are part of data analytics and reporting, not just a CSV formatting function.
References