Purpose

Fixes: #72 /claim #72 Implement data synchronization functionality between external data sources and StarbaseDB’s internal SQLite database. This plugin enables close-to-edge replica creation by automatically pulling and synchronizing data from external sources (like PostgreSQL) at configurable intervals.

Tasks

• Implement core Data Sync Plugin

  • Configurable sync intervals via sync_interval option
  • Selective table synchronization via tables option
  • Incremental updates using created_at and id columns
  • PostgreSQL to SQLite type mapping
  • Metadata tracking for sync state

• Add plugin configuration support

  • Wrangler.toml configuration options
  • Environment variable support for sensitive data
  • Plugin registration system integration

• Implement sync mechanisms

  • Pull-based synchronization
  • Schema auto-detection and mapping
  • Incremental data fetching
  • Error handling and retry logic

• Create comprehensive demo and documentation

  • Testing environment with Docker PostgreSQL
  • Example endpoints for monitoring
  • Query hook demonstrations
  • Security best practices

Verify

1. Start a PostgreSQL instance:

docker run --name starbasedb-postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=demo -p 5432:5432 -d postgres:15

2. Configure the plugin in wrangler.toml:

[plugins.data-sync]
sync_interval = 300 # 5 minutes
tables = ["users", "products"]

3. Set up environment variables for database credentials

4. Test synchronization:

• Monitor sync status: curl http://localhost:8787/sync-status
• View synced data: curl http://localhost:8787/sync-data
• Check sync metadata: curl http://localhost:8787/debug

Before

StarbaseDB instances could only:

• Use internal SQLite database
• Connect to external databases for direct queries
• No automatic data synchronization
• No edge-replica capabilities

After

StarbaseDB now supports:

• Automatic data synchronization from external sources
• Configurable sync intervals and table selection
• Incremental updates based on timestamps and IDs
• Type mapping between PostgreSQL and SQLite
• Close-to-edge replica functionality
• Monitoring and debugging capabilities