Deployment
This guide covers deploying the Activation System to production. The architecture follows a layered proxy pattern: Cloudflare (edge) → Hono API (Node.js origin) → Supabase / Redis / Operator APIs (data layer).
Architecture Overview
┌─────────────────────────────────────────────────────────────────┐
│ Cloudflare (Edge) │
│ DNS · TLS 1.3 termination · WAF · DDoS protection · Caching │
└──────────────┬──────────────────────────┬──────────────────────┘
│ │
/api/* routes /admin/* routes
│ │
▼ ▼
┌──────────────────────┐ ┌──────────────────────┐
│ Hono API (Node.js) │ │ Next.js Admin/B2B │
│ Port 3001 │ │ Dashboard │
│ │ │ │
│ • BullMQ queues │ │ • Super Admin portal │
│ • Drizzle ORM │ │ • B2B company portal │
│ • Moyasar/Twilio │ │ • @supabase/ssr auth │
└──────┬───────┬────────┘ └──────────┬─────────────┘
│ │ │
▼ ▼ ▼
┌────────────┐ ┌────────┐ ┌────────────────────┐
│ Supabase │ │ Redis │ │ Supabase │
│ PostgreSQL │ │ 7.x │ │ (Auth/Storage/ │
│ (Drizzle) │ │(BullMQ)│ │ Realtime) │
└────────────┘ └────────┘ └────────────────────┘The Flutter mobile app communicates with the Hono API and is distributed through app stores (not deployed as a server).
Prerequisites
| Requirement | Version | Purpose |
|---|---|---|
| Node.js | 22 LTS | API server runtime (required for @hono/node-server + BullMQ TCP connections) |
| pnpm | 10+ | Monorepo package manager |
| Supabase project | Managed | PostgreSQL database, Auth, Storage, Realtime |
| Redis instance | 7.x | BullMQ job queues, caching, OTP storage, rate limiting |
| Cloudflare account | Managed | DNS, TLS termination, WAF, DDoS protection |
| Moyasar or Paymob (KSA) account | — | Payment processing (Mada, Visa, Mastercard, Apple Pay; STC Pay) |
| SMS provider account | — | Twilio or Taqnyat for OTP delivery |
Environment Configuration
Production uses the same Zod-validated environment variables as development (defined in apps/api/src/config/env.ts), but with critical differences. See the Configuration guide for the full variable reference.
Key Production Differences
| Variable | Development | Production |
|---|---|---|
NODE_ENV | development | production |
DATABASE_URL | Pooler on port 6543 | Same pooler URL, SSL enforced with rejectUnauthorized: true, TLS 1.3 minimum |
DATABASE_URL_DIRECT | Port 5432 for migrations | Same direct URL, SSL enforced, TLS 1.3 minimum |
REDIS_URL | redis://localhost:6379 | rediss:// with TLS |
PAYMENT_PROVIDER | mock fallback | moyasar or paymob fallback; owner must configure/enable saved plugin settings |
PUBLIC_API_URL | Optional; HTTPS tunnel for gateway testing | Pins the public HTTPS API base for /payment-return and Paymob callbacks (/webhooks/paymob); unset, each checkout uses the address its request reached |
ENCRYPTION_KEY | Placeholder | Securely generated 64-char hex key (AES-256) |
CORS_ORIGINS | http://localhost:3000,http://localhost:3001,http://localhost:8888 | Comma-separated list of real owned production origins |
Email, SMS and push providers are not in this table because they are not environment variables. They are stored in the database and changed at runtime in Admin → Plugins → Services. In production the development mock cannot be made active, and push starts off, so save the provider keys and pick each service's provider before go-live. See Service providers.
Required Production Secrets
These variables must be set in the deployment platform's secret manager — never in code or unencrypted config:
DATABASE_URLandDATABASE_URL_DIRECT— Supabase pooler and direct connection stringsSUPABASE_URL,SUPABASE_ANON_KEY,SUPABASE_SERVICE_ROLE_KEY— Supabase project credentialsSUPABASE_JWT_SECRET— JWT signing secret from Supabase dashboardENCRYPTION_KEY— Generate withnode -e "console.log(require('crypto').randomBytes(32).toString('hex'))"ENCRYPTION_KEY_PREVIOUS— Only during a key rotation. Holds the retired key so rows written under it stay readable, and is removed once those ciphertexts are goneREDIS_URLEXCHANGE_RATE_API_KEY— WhenFX_PROVIDER=exchangerate-api, which production requires whenever the eSIM store is enabled
The app crashes at startup if any required variable is missing (Zod fail-fast validation in apps/api/src/config/env.ts).
Database Deployment
The database is Supabase managed PostgreSQL. All schema changes go through Drizzle ORM migrations.
Migration Commands
Two migration systems run sequentially via packages/database/src/migrate.ts:
- Drizzle Kit migrations — Schema DDL generated by
drizzle-kit generate(stored inpackages/database/drizzle/) - Custom SQL migrations — Hand-written SQL for RLS policies, triggers, Realtime publication (stored in
packages/database/drizzle/custom/)
Run migrations against the direct connection (port 5432), not the pooler (port 6543):
# Generate a new migration from schema changes
pnpm db:generate
# Apply all migrations (Drizzle Kit + custom SQL)
pnpm db:migrate
# Verify the live Supabase database after migration
pnpm db:verifyThe db:migrate script (tsx src/migrate.ts) does the following:
- Runs
drizzle-kit migrateagainstDATABASE_URL_DIRECT(pgBouncer on port 6543 is incompatible with DDL) - Applies custom SQL migrations from
drizzle/custom/in order (0000_,0001_, etc.) - Tracks applied custom migrations in a
custom_migrationstable with content hashing (skips already-applied, re-applies and updates the stored hash when content changes) - Leaves live database verification to
pnpm db:verify, which checks RLS flags, expected policies, custom migration hashes, Supabase Realtime publication, required triggers, and PostgreSQL extensions.
Notification Review Rollout
Apply 0026_contract_readiness_episode.sql before deploying the new API code. It adds the durable readiness counter and backfills it above legacy numeric occurrence-key suffixes. Application rollback can retain this additive schema, but an old writer may reuse count-based keys; prefer rollforward. Do not automatically down-migrate or delete the counter values or notification history.
The compliance retention sweep (POST /compliance/retention/sweep, super_admin) must run to scrub eligible batch requests to channels-only metadata: finished/cancelled for at least 24 hours, with an expired or null replay window. The notification handoff sweep does not perform this scrubbing. Recipient params retain their existing 365-day account-deletion window.
RLS Policies
Row Level Security policies are enforced at the database level. The executable policy SQL lives in packages/database/drizzle/custom/0004_apply_rls_policies.sql. Key policies include:
consent_records_user_insert— Authenticated users can INSERT their own consent records.- User/order/payment ownership policies — End users can read their own records while staff roles have scoped access.
- Admin/support policies —
super_admin,support,activation_officer, andb2b_adminroles have role-specific access. - Auth trigger —
handle_new_user()syncsauth.userstopublic.userson signup withSECURITY DEFINERandpgcryptofor phone hashing. - Cascade delete —
handle_deleted_user()cleans up all public data when a user is deleted from Supabase Auth.
Run pnpm db:verify after every deployment to prove those policies are installed on the actual Supabase project.
Supabase Realtime
Custom migration 0003_enable_realtime.sql adds activations and activation_status_events to the supabase_realtime publication for WebSocket-based status tracking. The activations table uses REPLICA IDENTITY FULL so UPDATE/DELETE events include old values.
TLS Enforcement
Production database connections enforce TLS 1.3 minimum (defined in packages/database/src/utils/connection.ts):
ssl: process.env.NODE_ENV === 'production'
? { rejectUnauthorized: true, minVersion: 'TLSv1.3' }
: false,API Server Deployment
Build
The API compiles from TypeScript to JavaScript using tsc:
# From monorepo root — builds all workspace packages first, then the API
pnpm build
# Or build the API specifically
pnpm --filter @activation-sys/api buildThe build outputs to apps/api/dist/. The tsconfig.json in apps/api/ targets ES2022 with NodeNext module resolution, outputting declarations and source maps.
Start
NODE_ENV=production node apps/api/dist/index.jsThe server binds to the port specified by the PORT environment variable (default 3001). On startup, it:
- Validates all environment variables via Zod (crashes if invalid)
- Registers the production SMS provider (
registerProductionSmsProvider()) - Registers the production email provider (
registerProductionEmailProvider()) - Schedules the package sync job (every 5 minutes by default)
- After Redis connects, starts notification email, SMS, and orchestration workers with injected rendering, provider dispatch, and orchestration hooks alongside the activation, payment, and store workers, then schedules the startup and recurring handoff sweep
- Starts listening via
@hono/node-server
HTTP listening does not wait for Redis or worker startup to finish. Payment, activation, store, and store-recovery startup failures are logged and roll back started workers and notification producers, but do not stop the HTTP server. Notification startup failures are logged and stop only the notification workers. The notification producers, the business workers, and store recovery keep running, so sends from that process still queue work for another replica or for the next start. A healthy HTTP response therefore does not prove that notification delivery is running. There is no automatic worker-start retry in this startup path after a failed initial Redis connection. Restore connectivity and restart the API, then verify worker activity.
Notification Readiness
The three notification lanes replace the former shared stub and API-local status lane. They run inside the API process without a notification feature flag. Verify notification-email, notification-sms, and notification-orchestration, including the repeat-notification-reconciliation scheduler, before treating delivery as ready. In-app notifications do not require a channel worker, and activation Realtime updates are independent of all three lanes.
Notification email uses the active email plugin settings. Configure and verify SMTP before expecting real email. Notification SMS always uses the mock, regardless of the OTP provider selection. Push has no implementation. Mock callback ingestion is refused in production, and SMTP has no delivery callback, so successful SMTP notifications end at sent.
Monitor persisted delivery outcomes and notification queue metrics together. A finished batch means dispatch persistence completed, not that every recipient received a message. Check pending and failed delivery counts separately. The handoff sweep repairs eligible persisted deliveries and re-arms scheduled or running batches that stalled past the stale window. It also re-drives unprocessed callback receipts that were never queued or whose job is gone. See handoff reconciliation and provider limits, including the residual SMTP duplicate window.
Each API replica starts its own channel workers, with concurrency 5 for email and 2 for SMS. HTTP rate-limit counters are also per process and held in memory, with no shared application-wide counter. Do not treat that HTTP budget as a cluster-wide delivery limit.
Health Check
The GET /health endpoint checks database and Redis connectivity with timeouts (5s for database, 3s for Redis):
{
"status": "ok",
"version": "1.0.0",
"timestamp": "2026-05-05T08:10:54.000Z",
"checks": {
"database": "ok",
"redis": "ok"
}
}When a dependency check fails, the response returns HTTP 503 with "status": "degraded":
{
"status": "degraded",
"version": "1.0.0",
"timestamp": "2026-05-05T08:10:54.000Z",
"checks": {
"database": "error",
"redis": "ok"
}
}Implementation: apps/api/src/routes/health.ts.
Graceful Shutdown
The server handles SIGTERM and SIGINT signals (in apps/api/src/index.ts):
- Closes the HTTP server (
server.close()) - Stops in-process workers, then closes the three notification queues and their supplied Redis clients
- Disconnects the API Redis singleton (
await disconnectRedis()) - Closes the database connection pool (
await closePool()) - Exits with code 0 on successful cleanup or code 1 if a cleanup stage throws
Cleanup attempts later stages even if an earlier stage fails. Notification worker drains and notification producer cleanup each have five-second bounds, with forced disconnection where needed. Activation, payment, and store worker drains are awaited before the database pool closes. The shutdown path calls server.close() without awaiting its completion, so it does not guarantee that all in-flight HTTP requests finish before exit. Send SIGTERM with enough termination grace for worker and connection cleanup, and verify the behavior under load. The notification close helper does not close the older non-notification producers.
Admin Dashboard Deployment
The Next.js admin dashboard runs as a separate server behind Cloudflare.
Build
# From monorepo root
pnpm build
# Or build specifically
pnpm --filter @activation-sys/admin buildNext.js outputs the standalone build to apps/admin/.next/standalone/. For production, enable standalone output mode in next.config.ts:
const nextConfig: NextConfig = {
output: 'standalone', // Produces a self-contained server bundle
};Start
# Using the standalone server (smaller footprint, no node_modules required)
NODE_ENV=production node apps/admin/.next/standalone/server.jsThe standalone mode bundles only the necessary dependencies, reducing the production image size significantly compared to running next start with the full node_modules.
Auth
The admin dashboard uses @supabase/ssr for server-side authentication with cookie-based sessions in Next.js App Router. Ensure SUPABASE_URL and SUPABASE_ANON_KEY are available as environment variables.
Docker Deployment
Docker Compose
The project's docker-compose.yml at the monorepo root builds production app images and runs the apps as separate containers plus Redis 7.x. Each app container pins its own internal PORT; only the published host side is configurable. This prevents a deploy-level PORT=3001 from making every image listen on the API port.
services:
admin:
image: skyte-admin:latest
environment:
PORT: 3000
ports:
- "${ADMIN_HOST_PORT:-3000}:3000"
api:
image: skyte-api:latest
environment:
PORT: 3001
ports:
- "${API_HOST_PORT:-3001}:3001"
depends_on:
redis:
condition: service_healthy
web-b2c:
image: skyte-web-b2c:latest
environment:
PORT: 3002
ports:
- "${WEB_B2C_HOST_PORT:-3002}:3002"
site:
image: skyte-site:latest
environment:
PORT: 3003
ports:
- "${SITE_HOST_PORT:-3003}:3003"
docs:
image: skyte-docs:latest
environment:
PORT: 5173
ports:
- "${DOCS_HOST_PORT:-5173}:5173"
redis:
image: redis:7-alpine
ports:
- "${REDIS_HOST_PORT:-6379}:6379"
command: redis-server --appendonly yes
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
start_period: 5s
restart: unless-stoppeddocker compose up -d # Start built apps and Redis in background
docker compose down # Stop and remove containersProduction Docker for API Server
A Dockerfile for the API server follows this pattern:
FROM node:22-slim AS builder
WORKDIR /app
RUN corepack enable && corepack prepare pnpm@10 --activate
COPY pnpm-lock.yaml pnpm-workspace.yaml package.json ./
COPY apps/api/package.json apps/api/
COPY packages/database/package.json packages/database/
COPY packages/queue/package.json packages/queue/
COPY packages/shared/package.json packages/shared/
RUN pnpm install --frozen-lockfile
COPY . .
RUN pnpm build --filter @activation-sys/api
FROM node:22-slim AS runner
WORKDIR /app
COPY --from=builder /app/apps/api/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/apps/api/package.json ./
ENV NODE_ENV=production
EXPOSE 3001
CMD ["node", "dist/index.js"]BullMQ and Redis
BullMQ (in packages/queue/) requires persistent TCP connections to Redis via ioredis. This is why the API runs on Node.js rather than Cloudflare Workers — Cloudflare Workers' V8 isolates do not support persistent TCP sockets.
Production Redis should use a managed provider with TLS (rediss:// URLs). The ioredis client is configured with maxRetriesPerRequest: null as required by BullMQ.
Cloudflare Configuration
Cloudflare sits in front of both the API origin and the admin dashboard, providing TLS termination, WAF, DDoS protection, and caching. The API itself runs on a Node.js origin server — Cloudflare is the proxy, not the compute platform.
DNS Setup
Point the application domain to Cloudflare nameservers. Configure DNS records for the API and admin origins:
| Record | Type | Target | Proxy |
|---|---|---|---|
<production-host> | A | API origin IP | Proxied (orange cloud) |
<admin-production-host> | A | Admin origin IP | Proxied (orange cloud) |
SSL/TLS Termination
- Set SSL mode to Full (strict) — Cloudflare terminates TLS 1.3 for clients, then connects to the origin over HTTPS with a valid certificate
- Minimum TLS version: TLS 1.3 (per security requirements)
- Enable Always Use HTTPS to redirect HTTP → HTTPS
- Enable Automatic HTTPS Rewrites
WAF Rules
Configure Web Application Firewall rules to protect against common attacks:
- SQL injection — Block requests with SQL patterns in query parameters
- XSS — Block requests with script injection patterns
- Bot management — Challenge or block known bot user agents
- Country access — Restrict admin dashboard access to expected regions (Saudi Arabia + team locations)
Rate Limiting
Cloudflare provides rate limiting at the edge, complementing the application-level rate limiting in apps/api/src/middleware/rate-limit.ts:
- Global rate limit — Protect the origin from traffic spikes
- API-specific rules — Stricter limits on
/auth/*endpoints to prevent brute-force OTP attempts - Admin dashboard — Moderate limits for authenticated admin users
Origin Server Protection
- Authenticated Origin Pulls — Ensure only Cloudflare can connect to the origin server by verifying Cloudflare's client certificate
- IP Access Rules — Allow only Cloudflare IP ranges to reach the origin server
- Configure the origin server's firewall to accept inbound HTTPS only from Cloudflare IP ranges
Monitoring and Health Checks
Health Endpoint
GET /health — Returns system status with dependency checks.
Healthy response (HTTP 200):
{
"status": "ok",
"version": "1.0.0",
"timestamp": "2026-05-05T08:10:54.000Z",
"checks": {
"database": "ok",
"redis": "ok"
}
}Degraded response (HTTP 503):
{
"status": "degraded",
"version": "1.0.0",
"timestamp": "2026-05-05T08:10:54.000Z",
"checks": {
"database": "error",
"redis": "ok"
}
}Each check has a timeout (5s for database, 3s for Redis). A timed-out check is reported as "error".
Application Logging
The API uses Pino for structured JSON logging (pino@^9.0.0 in apps/api/package.json). All log entries include:
- Timestamp
- Log level (trace, debug, info, warn, error, fatal)
- Request ID (from Hono middleware)
- Structured fields (userId, route, method, statusCode, duration)
Production logs should be shipped to a centralized logging service for querying and alerting.
Analytics
PostHog (posthog-node for API, posthog_flutter for mobile) tracks:
- Activation funnel events (package selection → identity verification → payment → activation)
- Payment events and conversion metrics
- Feature flag evaluation
Recommended Monitoring
| Check | Method | Alert Condition |
|---|---|---|
| API health | GET /health every 30s | Status 503 or timeout |
| Database connectivity | Health check — database field | "error" status |
| Redis connectivity | Health check — redis field | "error" status |
| API error rate | Pino structured logs | 5xx rate > 1% over 5 minutes |
| BullMQ queue depth | BullMQ metrics | Queue depth > 1000 or jobs stuck > 10 minutes |
| SSL certificate expiry | Automated check | < 30 days to expiry |
Post-Deployment Verification
After deploying, verify each component is functioning correctly:
API Server
- [ ]
GET /healthreturns HTTP 200 with"status": "ok"and both checks passing - [ ]
GET /healthreturns the expectedversionfield - [ ] CORS headers present on preflight requests (
OPTIONS) - [ ] API responds within 2 seconds at the 95th percentile
- [ ] Graceful shutdown works: send
SIGTERM, confirm process exits cleanly
Database
- [ ] Migrations applied successfully with
pnpm db:migrate - [ ] Live Supabase database verification passed with
pnpm db:verify - [ ] RLS policies installed on all expected public tables
- [ ] TLS 1.3 enforced on production connections
- [ ] Supabase Realtime publication includes
activationsandactivation_status_events - [ ] Connection pooling active on port 6543 (verify via
DATABASE_URL)
Admin Dashboard
- [ ] Admin dashboard loads at the production URL
- [ ] Supabase Auth login works (cookie-based sessions via
@supabase/ssr) - [ ] Admin, B2C web, and landing site pages render without console errors
Cloudflare
- [ ] DNS resolves correctly for
<production-host> - [ ] TLS 1.3 is the minimum negotiated version (check with
curl -vvv https://<production-origin>) - [ ] WAF blocks test payloads (e.g.,
?id=1' OR '1'='1) - [ ] Rate limiting triggers on burst requests
- [ ] Origin pull authentication passes (only Cloudflare can reach origin)
- [ ] HTTP requests redirect to HTTPS
Redis / BullMQ
- [ ] Redis connection uses TLS in production (
rediss://URL) - [ ] BullMQ can enqueue and process test jobs
- [ ] Package sync job runs on schedule (check logs for
[API] Package sync job scheduled) - [ ] Notification email, SMS, and orchestration workers process jobs, with per-queue metrics and startup/recurring handoff reconciliation activity
- [ ] SMTP notification outcomes reach
sent, with provider message IDs when returned, without expecting a delivery callback - [ ] Batch pending/failed counts are checked independently of
finishedstatus and HTTP health - [ ] A termination test confirms notification workers and their dedicated producer clients close
Payment Integration
- [ ] Moyasar webhook endpoint receives test events (
/webhooks/moyasar) - [ ] Moyasar webhook secret matches the selected plugin environment
- [ ] Moyasar sandbox checkout and confirmation succeed without real fulfillment before enabling production
- [ ] Paymob: the API is reachable at a public HTTPS address (checkouts send their own callback addresses; no dashboard Webhook or Redirect URL is needed)
- [ ] Paymob production: API key saved; HMAC secret optional (old dashboard → Settings → Account info)
- [ ] Paymob sandbox checkout and confirmation succeed without real fulfillment before enabling production
SMS Provider
- [ ] OTP delivery works in production (Twilio or Taqnyat)
- [ ] SMS logs confirm delivery (check Pino structured logs)
Rollback Procedure
If a deployment causes issues:
- API rollbacks — Redeploy the previous Docker image tag or revert the git commit and rebuild
- Database rollbacks — Drizzle does not auto-generate down migrations. Write a manual SQL migration to revert schema changes and add it to
drizzle/custom/ - Admin dashboard rollbacks — Redeploy the previous standalone build or use the hosting platform's rollback feature
- Cloudflare — Use the Cloudflare dashboard to revert rule changes; DNS TTL changes may take up to 48 hours to propagate
Always verify the GET /health endpoint returns "ok" after any rollback.
Frontend Deployment Surfaces
| Workspace | Build output | Deployment notes |
|---|---|---|
apps/admin | .next/** | Deploy as the admin/B2B Next.js app behind authenticated routes. |
apps/web-b2c | .next/** | Deploy as the customer web activation app; configure API origin and secure cookie domain intentionally. |
apps/site | dist/** | Deploy as a static Vite landing site through CDN/edge hosting. |
docs | .vitepress/dist/** | Deploy separately from customer-facing surfaces; keep internal docs access-controlled where required. |
Production CORS must include only deployed frontend origins that call apps/api. TLS termination remains at Cloudflare, and all sensitive activation, payment, identity, and compliance logic stays server-side.
Payment rollout
Configure Admin → Plugins → Payments with matching production keys and webhook secret (Paymob: public key, secret key, API key and card integration ID; the HMAC secret is optional and callback addresses need no setup); saved settings override PAYMENT_PROVIDER. Unconfigured production starts disabled. Verify sandbox checkout, API confirmation, webhook delivery and the customer SDK integration before accepting real payments. Mock remains explicitly selectable but does not charge customers. See payment setup.
Paymob reconciliation shares the existing moyasar-reconciliation queue, so Moyasar jobs queued before the upgrade keep draining and no idle window is needed. Jobs queued before Paymob existed carry no provider and are treated as Moyasar's.