Queues

Overview

We use BullMQ backed by Redis for async processing. Eight queues handle the core async workflows.

Queue Configuration

Queue	Name	Location	Purpose
Activation	activation	packages/queue	Drives operator activation steps (initiate, verify_identity, complete)
Payment	payment	packages/queue	Settles completed payments by enqueuing activation handoff
Notification	notification	packages/queue	Delivers activation and payment status notifications
Store Fulfillment	store-fulfillment	packages/queue	Issues eSIM profiles for paid store orders
Package Sync	package-sync	apps/api	Periodic sync of telecom operator packages
eSIM Product Sync	esim-product-sync	apps/api	Periodic sync of eSIM provider catalogs
FX Refresh	fx-refresh	apps/api	Scheduled fetch of USD exchange rates from the configured provider
Status Notifications	status-notifications	apps/api	Delivers realtime activation status events to connected subscribers

The packages/queue queues use shared constants from @activation-sys/shared. The apps/api queues configure attempts, backoff, removeOnComplete, and removeOnFail directly at queue creation time, and share the API's Redis singleton without a BullMQ prefix.

For the packages/queue four, connections do not use ioredis keyPrefix, because BullMQ requires unprefixed Redis connections. Their namespaces are set with BullMQ's prefix option through getQueuePrefix(queueName). This preserves the previous effective Redis key layout, such as activation:bull:activation:* and payment:bull:payment:*, without a deploy-time namespace migration.

Usage

import { addActivationJob, addPaymentJob, addNotificationJob } from '@activation-sys/queue';

// Add an activation job
await addActivationJob({
  type: 'activation',
  activationId: '...',
  userId: '...',
  operatorSlug: 'stc',
  step: 'initiate',
});

// Add a payment job
await addPaymentJob({
  type: 'payment',
  paymentId: '...',
  orderId: '...',
  userId: '...',
  gateway: 'stc_pay',
});

// Add a notification job
await addNotificationJob({
  type: 'notification',
  userId: '...',
  channel: 'sms',
  templateId: 'activation_complete',
  data: { iccid: '8996601212345678901' },
});

FX Refresh Queue

The fx-refresh queue fetches USD exchange rates from the configured provider. It uses 3 attempts with exponential backoff starting at 5000ms and concurrency of 1. A repeat scheduler runs every FX_REFRESH_INTERVAL_MS plus a startup job triggers an immediate fetch. The queue is gated by FEATURE_ESIM_STORE. A store-disabled boot also clears any repeatable scheduler left in Redis by an earlier store-enabled boot (both the FX and eSIM sync lanes), and logs a warning if that cleanup fails so a leftover scheduler is visible. Transient provider fetch failures throw so the queue retries. Validation and jump-guard rejects keep stored rates and complete the job.

Catalog Sync Engine

One shared engine (catalog-sync-engine.ts) drives both the eSIM product sync and the local package sync. Each lane provides a thin adapter around the engine. Every provider or operator gets its own DB transaction. One owner failing does not stop the rest. When all owners fail the engine throws so the queue retries.

Store Fulfillment State Machine

The store worker drives a provider-agnostic issuance lifecycle on esim_store_orders.issuance_state. It stamps in_flight before every provider call and persists the provider's own order reference as soon as it is known. A retry that finds in_flight, reconciliation_required, or pending_verification goes through the plugin's reconcile capability instead of calling issue again — the double-billing guard for providers without idempotency keys.

Classified provider failures map to queue behavior. Transient errors reset to pending and rethrow so the 3-attempt backoff engages — the failure reason is recorded only on the final attempt. Out-of-stock and terminal errors mark the order failed and stop retries with an unrecoverable error. Ambiguous results park the order as reconciliation_required. An unclassified throw keeps in_flight so the next attempt reconciles. A pending_verification outcome records the customer verification URL and completes the job without completing the order.

The eSIM catalog sync worker also refreshes each registered provider's health reading after every run — the data behind the admin providers list and the checkout circuit breaker.

Payment Queue Deduplication

Re-adding a settlement job first removes a previously failed job with the same id.

Worker Lifecycle

apps/api/src/index.ts starts the payment, activation, and store workers inside the API process after Redis connects. This runs in development and in production. Only NODE_ENV=test skips worker startup. It is what carries a settled payment through to activation completion on the local lane and to store fulfillment on the store lane, once the gates are satisfied: order paid and identity verification completed. Running the workers as their own deployment is a tracked follow-up.

The block below is the packages/queue entry point a standalone worker service would use.

import { initializeQueues, shutdownQueues } from '@activation-sys/queue';
import { esimProviderPluginManager } from './plugins/index.js';
import { sendStoreEsimDeliveryEmail } from './services/esim-delivery.js';

// Start queues and workers. Store fulfillment starts only when the
// eSIM provider manager is supplied — without it, paid store orders
// wait unissued until a store worker attaches.
await initializeQueues({
  startWorkers: true,
  storeProviderManager: esimProviderPluginManager,
  onStoreIssued: sendStoreEsimDeliveryEmail,
});

// Graceful shutdown
await shutdownQueues();

Redis Requirements

• Redis 7+ with TLS 1.3 in production (rediss://)
• Each packages/queue queue gets a dedicated connection with a BullMQ prefix via getQueuePrefix(queueName). The apps/api queues share the API's Redis singleton, unprefixed
• maxRetriesPerRequest: null required for BullMQ compatibility