I use a double-buffered queue to collect and segment telemetry events into groups bound by UTC hour. My pipeline leverages Cloudflare Workers, Queues, Workflows, and R2 to capture, batch, process, and store telemetry events while maximizing the benefits of Cloudflare's pricing.
Modulo math is used to route the payload to the active queue and again to drain the inactive queue. UTC is used to avoid DST handling.
Producer
The producer evaluates the payload's timestamp to route the message to the active queue:
Consumer
The consumer task triggers every hour on the half-hour to drain the inactive queue:
The consumer task looks back one hour to determine which queue is currently inactive and safe to consume:
Finally, the batch is compressed and written to R2 using a simple partitioning strategy:
This pattern transforms chaotic streams of telemetry into a compressed hourly archive. You get a partitioned, serverless data lake optimized for downstream analytics, while keeping your cloud storage bills incredibly low.