Purpose

Fixes #59 by replacing memory-heavy export behavior with streaming + async job-based dump processing that can continue beyond request limits, with optional R2 artifact storage, callback notification, and alarm-driven continuation.

Closes #59 /claim #59

Tasks

• Implement streaming export primitives for SQL/CSV/JSON to avoid loading full tables in memory
• Add async dump flow: POST /export/dump, GET /export/dump/:jobId, GET /export/dump/:jobId/download
• Persist export job state in DO SQLite (tmp_export_jobs, tmp_export_job_tables, tmp_export_job_chunks)
• Add DO alarm continuation for long-running export jobs
• Add optional callback URL support on async dump completion/failure
• Add callback retry with exponential backoff and alarm scheduling
• Add optional R2 artifact support (EXPORT_R2_BUCKET) and stream downloads from R2 when available
• Add stale export metadata cleanup logic in alarm cycle
• Update docs for async dump usage and optional R2 binding
• Add and update unit tests for new/changed export behavior

Verify

• Run tests:

pnpm vitest run \
  src/handler.test.ts \
  src/do.test.ts \
  src/export/async-dump.test.ts \
  src/export/dump.test.ts \
  src/export/csv.test.ts \
  src/export/json.test.ts \
  src/export/index.test.ts

• Manual sanity checks:

# Start async export
curl --location --request POST 'https://starbasedb.YOUR-ID-HERE.workers.dev/export/dump' \
  --header 'Authorization: Bearer ABC123' \
  --header 'Content-Type: application/json' \
  --data '{"callbackUrl":"https://example.com/webhooks/export-complete"}'

# Poll status
curl --location 'https://starbasedb.YOUR-ID-HERE.workers.dev/export/dump/JOB_ID_HERE' \
  --header 'Authorization: Bearer ABC123'

# Download result
curl --location 'https://starbasedb.YOUR-ID-HERE.workers.dev/export/dump/JOB_ID_HERE/download' \
  --header 'Authorization: Bearer ABC123' \
  --output database_dump.sql

Before

Large DB dump requests could fail due to request timeout / memory-heavy accumulation.

After

• Exports stream and paginate data instead of aggregating full datasets in memory.
• Long-running dump operations continue asynchronously with persisted checkpoints.
• Status/download endpoints provide retrieval after completion.
• Optional callback notifications (with retry backoff) and optional R2 artifact storage are supported.