Summary

• add streaming to the existing sync dump to reduce peak memory pressure
• add async dump job flow for databases too large to export within the 30s worker timeout:
  • POST /export/dump kicks off a background job, returns { dumpId, statusUrl } immediately
  • GET /export/dump/:dumpId to poll status/progress
  • GET /export/dump/:dumpId/download streams the completed dump file from R2
• dump progress is persisted in DO storage; large files are written via R2 multipart uploads
• existing GET /export/dump (sync path) is unchanged — still works for small databases

Why

The current endpoint loads the whole database into memory and has to finish inside the 30-second Workers limit. That’s fine for small databases but breaks for anything approaching the DO storage limit. The async path runs the export in chunks via DO alarms that chain themselves until done, then writes the result to R2 for streaming download.

Demo

Full flow demonstrated:

1. Create test data (50 users + 30 products across 2 tables)
2. POST /export/dump — starts async export, returns dumpId + statusUrl
3. GET /export/dump/:id — poll status (shows complete with progress)
4. GET /export/dump/:id/download — stream valid .sql file from R2
5. Unauthenticated request returns HTTP 401
6. All 7 unit tests pass

Architecture

Client                    Worker (Hono)              Durable Object              R2
  │                           │                           │                       │
  ├─POST /export/dump────────►│                           │                       │
  │                           ├─startAsyncDump()─────────►│                       │
  │                           │                           ├─initiateDump()        │
  │                           │                           │  enumerate tables     │
  │                           │                           ├─createMultipartUpload─►│
  │◄──202 { dumpId }─────────┤                           │                       │
  │                           │                     alarm ├─processDumpChunk()    │
  │                           │                           │  read rows in batches │
  │                           │                           │  buffer → flush part──►│
  │                           │                           │  if time < 20s: loop  │
  │                           │                           │  else: save checkpoint │
  │                           │                     alarm ├─continue from checkpoint
  │                           │                           │  ...                  │
  │                           │                           ├─complete multipart───►│
  │                           │                           │                       │
  ├─GET /export/dump/:id─────►│──getAsyncDumpStatus()───►│                       │
  │◄──{ status: "complete" }──┤                           │                       │
  │                           │                           │                       │
  ├─GET .../download─────────►│──streamDumpDownload()───►│                       │
  │                           │                           ├─get object───────────►│
  │◄──stream .sql file────────┤◄──────────────────────────┤◄──ReadableStream──────┤

Key design decisions:

• DO alarms chain every 100ms, processing for up to 20s per cycle (well within 30s limit)
• R2 multipart with 5 MiB minimum part size, pending data buffered in DO storage (64 KiB chunks under 128 KiB value limit)
• Checkpoint/resume: table index + row offset saved on each alarm yield
• Concurrent dump rejection: only one active dump at a time per DO instance
• Callback URL: optional POST webhook on completion for downstream automation

Test

pnpm test --run src/export/dump-async.test.ts

/claim #59