Notifications API

The notification module provides an owner-scoped inbox, channel opt-outs, and administrative sends. A notification is one recipient's durable record. Each enabled channel has its own delivery outcome. Acceptance never means that email or SMS reached the recipient.

The interactive reference at /docs reads the hand-maintained /openapi.json. The operation IDs below match that reference.

What clients need

The backend now provides realtime inbox-change signals through SSE. Clients must integrate the stream to receive them; adding backend support does not automatically update existing apps.

Concern	Required behavior
Connection	Open one authenticated GET /notifications/stream connection while the signed-in notification experience is active.
Initial data	Wait for ready, then fetch /notifications and /notifications/count.
Updates	On notifications.changed, refetch inbox/count. Coalesce bursts and preserve hints received during an in-flight fetch.
Disconnection	Reconnect with backoff and fetch a new snapshot after ready; there is no replay.
Logout	Close the stream and discard the previous user's inbox state.
Polling	No periodic client polling is required. REST remains necessary for snapshots, hints, and explicit user actions. Heartbeat comments do not trigger a refetch.
Dependencies	Use the core API only. Clients need no Supabase subscription, Redis connection, or notification-provider credentials.
Rollout	Apply migration 0027_notification_signals.sql, deploy the updated API, and verify unbuffered SSE through the proxy before enabling client integration.

This delivers in-app change hints while connected, not operating-system push or background delivery. Time-only expiry emits no hint; the expiry rule takes effect on the next REST read. Backend recovery sweeps remain enabled and are separate from client polling.

Routes

Inbox routes require authentication. Every administrative route requires super_admin or an admin holding the notifications area, at View or Manage level depending on the route (see View vs Manage). Ownership and sender identity come from authenticated context, never the body.

Method	Path	Operation ID	Access
GET	/notifications	listNotifications	Own inbox
GET	/notifications/count	getUnreadNotificationCount	Own inbox
GET	/notifications/stream	streamNotifications	Own inbox hints (SSE)
POST	/notifications/:id/read	markNotificationRead	Own inbox
POST	/notifications/read-all	markAllNotificationsRead	Own inbox
POST	/notifications/seen	markNotificationsSeen	Own inbox
POST	/notifications/:id/archive	archiveNotification	Own inbox
GET	/notifications/opt-outs	getNotificationOptOuts	Own preferences
PUT	/notifications/opt-outs	updateNotificationOptOuts	Own preferences
POST	/admin/notifications	sendAdminNotification	super_admin, notifications (Manage)
GET	/admin/notifications/recipients	searchAdminNotificationRecipients	super_admin, notifications (View)
GET	/admin/notifications/batches	listAdminNotificationBatches	super_admin, notifications (View)
GET	/admin/notifications/batches/:id	getAdminNotificationBatch	super_admin, notifications (View)
POST	/admin/notifications/batches/:id/cancel	cancelAdminNotificationBatch	super_admin, notifications (Manage)
GET	/admin/notifications/delivery-rate	getAdminNotificationDeliveryRate	super_admin, notifications (View)
POST	/webhooks/notifications/:channel/:provider	receiveNotificationProviderCallback	Provider signature

See notification callbacks for the inbound route.

Inbox

Realtime change hints

Open GET /notifications/stream with Accept: text/event-stream and the existing session cookie or bearer token. No query parameters are accepted; the recipient comes from authentication. The response uses Cache-Control: private, no-store, no-transform and X-Accel-Buffering: no.

event: ready
data: {"version":1}

event: notifications.changed
data: {"version":1}

: heartbeat

ready follows subscriber registration. Every event has only the fixed version payload: no notification IDs, titles, bodies, counts, categories, or personal data. Redis publication is not client delivery. Duplicate/coalesced hints are permitted; there are no event IDs, replay guarantees, or exactly-once semantics. Last-Event-ID does not resume history.

Client adoption sequence (documentation only; this change does not edit clients):

1. Open the authenticated stream and wait for ready.
2. Fetch GET /notifications and GET /notifications/count as the initial snapshot.
3. If a hint arrives during a fetch, remember it and refresh once more when that fetch completes. Do not clear that pending hint when replacing the snapshot.
4. Coalesce bursts into one refresh. Refresh on explicit user actions as needed.
5. On disconnect or 503, reconnect with bounded exponential backoff and jitter; after the next ready, fetch a fresh snapshot. Refresh expired credentials through the existing login/session flow. Do not depend on replay.
6. Close on logout. Do not add a periodic inbox-fetch timer.

Browser EventSource supports cookies but cannot set an Authorization header; bearer clients need a streaming HTTP reader. Never put tokens or recipient IDs in the URL.

CORS: native mobile versus browser clients

Native Android/iOS Flutter requests through Dio are not subject to browser CORS. The phone needs no fixed DNS name, public IP, or entry in CORS_ORIGINS. It connects outbound to the API's HTTPS URL using Authorization: Bearer <token> and, for SSE, Accept: text/event-stream. The current mobile API client already attaches the bearer token and X-Client-Type: mobile; an SSE integration still needs streaming response handling. Do not invent an Origin header for the phone. Wi-Fi/cellular changes can break an open stream: reconnect with the current token and refetch after ready.

For browser clients (including Flutter Web or JavaScript requests inside a WebView), configure the page's exact origin, including scheme and port, in CORS_ORIGINS. This is the frontend origin, not the user's IP, device domain, API URL, or DNS resolver. The API returns that origin in Access-Control-Allow-Origin only when it is allowlisted and returns Access-Control-Allow-Credentials: true. Unlisted origins receive no allow-origin header, so browsers deny response access. Requests without Origin still require normal authentication; CORS is not API authorization or an IP firewall.

Example deployment: a page at https://app.example.com calls https://api.example.com; allowlist https://app.example.com. For development, http://localhost:3000 and http://localhost:3001 are different origins and must be configured accordingly.

For browser cookie authentication, open the stream with:

const stream = new EventSource(`${apiBaseUrl}/notifications/stream`, {
  withCredentials: true,
});

Use credentials: 'include' for cross-origin login/session and REST snapshot requests too. The existing cookie is HttpOnly, SameSite=Strict, and Secure in production. Thus cookie authentication works across same-site origins such as HTTPS subdomains of the same site, or different localhost ports using the same scheme. CORS permission alone does not make that cookie available across unrelated sites; use a same-site/same-origin deployment or an appropriate bearer-client flow. This change does not weaken cookie policy or allow arbitrary origins.

Startup waits at most 5 seconds for Redis connections to become ready and at most 5 seconds for scheduler registration before cleaning up and retrying. Long-lived JWT expiry timers are chained beyond Node's timer limit so streams close at the actual credential expiry.

New visible rows, read/seen changes, read-all, archive, content changes, visibility restoration, and removal/scrubbing of visible rows create durable invalidations. Idempotent no-op updates and maintenance of already hidden rows do not. Retention/deletion-request timestamps alone do not change inbox visibility and do not signal. Expiry is evaluated on REST reads: time passing does not write a row or produce a hint. Rendering still uses current locale and referenced domain data at read time; changes outside the notification row are not a notification-state event.

Streams expire at JWT expiry. The existing Redis session blocklist is checked before sending a change hint and every 15 seconds; failures close the stream. Limits are 5 connections per user per API process, 1000 per process, and 8 buffered frames per connection, plus the Node transport's bounded socket buffering. A slow reader is disconnected when the frame buffer fills; heartbeat writes reap idle peers. A single in-flight authorization check coalesces bursts. Subscriber disconnects close local streams immediately; a missing dispatcher health message closes them within 35 seconds. Streams remain unavailable until a fresh end-to-end health message arrives.

Realtime deployment and recovery

Apply Drizzle migration 0027_notification_signals.sql before starting the updated API using the existing pnpm db:migrate workflow. It adds the API-only notification_signals pending outbox and an AFTER INSERT OR UPDATE OR DELETE trigger. RLS is enabled; public, anon and authenticated roles have no access. The API database role must retain its existing inbox rights and have select/insert/delete access to the outbox (and RLS bypass/table ownership), as with the other backend-only tables. This implementation does not require Supabase Realtime, webhooks, shared queue changes, or infrastructure edits.

Every API process starts a dedicated subscriber and an API-local BullMQ notification-signals worker, independent of provider/delivery workers. The notification-signal-recovery scheduler runs every second. Each job locks at most 100 pending rows with FOR UPDATE SKIP LOCKED, coalesces recipient IDs, publishes through Redis, and removes acknowledged rows in the same dispatcher transaction. Notification writes have already committed before they can be claimed. A failed publication/acknowledgement rolls back that batch; later runs retry it. A crash after publishing may duplicate hints. PostgreSQL remains authoritative, and existing delivery/recovery sweeps continue unchanged.

Redis commands for publication/auth checks time out after 2 seconds; dispatcher SQL statements time out after 5 seconds. BullMQ retries a failed job 3 times with exponential backoff starting at 1 second. Durable rows survive exhausted job retries and are picked up by later scheduled jobs. Startup retries every 5 seconds after failure. Redis connections reconnect automatically; a dispatcher health message each successful tick proves database access, publishing, and subscription before streams become available. Shutdown closes streams and dedicated Redis/queue connections. Monitor structured notification.signal_* lifecycle events, notification.realtime_start_failed, and pending outbox count/oldest age. A growing backlog calls for measuring and adjusting the 100-row batch/one-second scheduler capacity.

Keep proxy buffering and compression buffering disabled for this route. Allow streaming response chunks and set ingress read/idle timeouts above the 15-second heartbeat interval (at least 60 seconds recommended). Any hard maximum connection duration may close a healthy stream; clients must reconnect and reconcile. Headers request no buffering but do not replace proxy configuration. Production proxy behavior must be verified during rollout.

Rollback: stop/revert the SSE API runtime first. Preserve the outbox while deciding whether to resume publication. Remove the trigger and function before removing the table only if abandoning realtime; existing notification REST state and delivery records are unaffected. No production migration is applied by the implementation tests.

Backend verification harness

Run focused HTTP/lifecycle and REST regression tests:

pnpm --filter @activation-sys/api test -- tests/notification-stream.test.ts tests/notification-realtime-lifecycle.test.ts tests/notification-routes.test.ts tests/auth-middleware.test.ts

For actual PostgreSQL/Redis/SSE verification, supply disposable loopback PostgreSQL and Redis instances (defaults: PostgreSQL 15432, Redis 16379). The PostgreSQL login needs database/role creation rights. The harness creates and drops its own uniquely named database, applies the notification baseline and new migration, and uses synthetic JWTs/content. It never reads configured application service credentials. Use an empty Redis dedicated to this harness because the real scheduler/channel names are exercised.

$env:NOTIFICATION_REALTIME_INTEGRATION = '1'
$env:NOTIFICATION_TEST_DATABASE_URL = 'postgres://postgres@127.0.0.1:15432/postgres'
$env:NOTIFICATION_TEST_REDIS_URL = 'redis://127.0.0.1:16379'
pnpm --filter @activation-sys/api test -- tests/notification-realtime.integration.test.ts

The harness checks commit/rollback, visibility/no-op/expiry/scrub semantics, RLS, bounded concurrent claims, failed publication recovery, HTTP authentication/recipient isolation, fixed event payloads, two independently connected runtime subscribers, REST reconciliation, cancellation, credential expiry/revocation, and subscriber-outage recovery. Lifecycle tests cover startup recovery/cancellation, scheduler timeout, health timeout, per-user/process limits, heartbeat reaping, and destruction of actual sockets when a paused TCP reader exceeds its buffer. A server-initiated close can appear as a transport reset; handle it as a disconnect and reconnect through the sequence above.

Feed and Count

GET /notifications returns { data, nextCursor, hasMore }, without total. Rows sort newest first by (created_at, id). Pass nextCursor back unchanged to preserve timestamp microseconds.

Query	Default	Bounds
cursor	Absent	Full-precision timestamp, then |, then UUID
limit	20	Integer from 1 to 50
unreadOnly	false	true or false
archived	false	true or false

Each item has id, eventKey, category, templateKey, severity, title, body, actionUrl, imageUrl, degraded, batchId, createdAt, readAt, seenAt, expiresAt, and archivedAt. Optional stored values are returned as null. Timestamps are RFC 3339 in UTC on every inbox response, including the read and archive responses. The default view only ever returns rows with a null archivedAt, and the archived view only ever returns rows with one set, so the field doubles as a view marker on top of stating when the row was archived. The cursor is opaque and is not a timestamp to parse.

The default feed hides archived, expired, and scrubbed rows. Passing archived=true returns archived rows only, newest first, with their own cursor, and nothing else. Expired and scrubbed rows stay hidden in both views. GET /notifications/count uses the same expiry and scrub predicates as the default feed view and returns { count } for unread rows only.

Inbox text is rendered on each read using the recipient's stored locale, not Accept-Language. Arabic locale tags select Arabic. Other values fall back to English. Changing the stored locale changes how history renders (the stored locale is updated via PUT /profile). Error messages still use Accept-Language. A render failure preserves the row and page boundary with a category title, empty body, no links, and degraded: true. Every other row carries degraded: false.

Read, Seen, and Archive

Request	Body	200 Response	Behavior
POST /notifications/:id/read	None	{ id, readAt }	Preserves the first read timestamp on repeats
POST /notifications/read-all	None	{ updated }	Marks visible unread rows only
POST /notifications/seen	{ notificationIds: [...] }	{ updated }	Accepts 1 to 200 UUIDs and counts newly seen rows
POST /notifications/:id/archive	None	{ id, archivedAt }	Preserves the first archive timestamp on repeats

Seen does not mean read. Duplicate seen IDs collapse. If any seen ID is foreign, missing, or scrubbed, the entire request returns 404 NOTIFICATION_NOT_FOUND without writes. Single-item read, seen, and archive operations allow owned expired or archived rows. Scrubbed rows are never writable through these routes. All inbox responses use Cache-Control: private, no-store.

Channels and Opt-outs

GET /notifications/opt-outs returns { channels, optOuts }. Every channel entry contains id, bilingual label, enabled, and optOutAllowed.

Channel	Enabled	Opt-out Allowed	Delivery
in_app	Yes	No	Written directly as delivered
email	Yes	Yes	Queued and sent through the selected email provider
sms	Yes	Yes	Queued and sent through the mock provider
push	No	Yes	Declared only, with no worker or delivery row

PUT /notifications/opt-outs accepts { "channels": ["email", "sms"] } as the full replacement list. An empty array clears all opt-outs. Duplicates collapse. push can be stored as a preference despite being disabled. in_app is rejected with 400 VALIDATION_ERROR.

Category	Transactional	Honors Opt-outs
activation	Yes	No
payment	Yes	No
esim_delivery	Yes	No
company	Yes	No
admin	Yes	No
system	Yes	No
announcement	No	Yes

Workers recheck account activity, opt-outs for non-transactional sends, and the current address before dispatch. An opt-out does not erase inbox history.

Administrative Send

POST /admin/notifications accepts exactly one of content or template. Unknown body fields are rejected.

Field	Required	Contract
category	Yes	One of the seven categories above
audience	Yes	One of the audience forms below
channels	Yes	Nonempty channel array, with duplicates collapsed
severity	No	info, warning, or critical, for presentation only
urgency	No	normal by default, or urgent, for queue priority only
expiresAt	No	Offset-bearing ISO timestamp or null, hides the inbox item after expiry
scheduledAt	No	Offset-bearing ISO timestamp or null, must precede expiresAt when both exist
content	One branch	Bilingual custom content
template	One branch	{ key, params } from the event registry

Audience	Shape	Resolution
Explicit users	{ kind: 'users', userIds: [...] }	1 to 1,000 UUIDs, deduplicated. Missing accounts create no rows. Inactive accounts retain in-app delivery but suppress external channels
Company	{ kind: 'company', companyId: UUID }	Active users with admin membership in that company, not all members or company contact fields
Role	{ kind: 'role', role: ... }	Active users with the named application role
Everyone	{ kind: 'all' }	All active users

Role values are end_user, b2b_admin, activation_officer, super_admin, support, and admin. Audiences resolve in chunks of 500 at execution time. A future batch does not freeze its recipient list at acceptance. Explicit-recipient counts include distinct submitted IDs even when an account no longer exists.

Custom Content

content.title and content.body each require nonempty en and ar strings. Whitespace is trimmed. Each title is capped at 160 UTF-8 bytes per language. Each body is capped at 1,000 UTF-8 bytes per language.

Optional content.actionUrl and content.imageUrl accept null, a root-relative path, or an HTTPS URL up to 2,048 characters. Protocol-relative URLs, root-relative paths that start with a backslash or contain control characters, and other schemes are rejected. Custom content is stored separately for each recipient, so storage grows with audience size.

{
  "category": "announcement",
  "audience": { "kind": "all" },
  "channels": ["in_app"],
  "content": {
    "title": { "en": "Maintenance", "ar": "صيانة" },
    "body": { "en": "Brief downtime tonight.", "ar": "انقطاع موجز الليلة." }
  }
}

Templates

The category must match the template's registered category. Parameter objects are strict. IDs are UUIDs and occurrence counters are nonnegative integers.

Template Key	Category	Parameters	Enabled Channels
activation.completed	activation	activationId, packageName of 1 to 255 characters, optional packageNameAr of 1 to 255 characters	in_app, email, sms
activation.failed	activation	activationId, eventId, errorCode from the activation failure codes	in_app
payment.succeeded	payment	paymentId, orderId	in_app, email
company.approved	company	companyId, approvalAuditId	in_app, email
contract.ready	company	companyId, contractId, readinessEpisode	in_app, email
admin.invited	admin	userId, inviteSequence	in_app, email, sms
esim.issued	esim_delivery	source: 'activation' with activationId, or source: 'store' with storeOrderId	in_app, email
ticket.opened	system	ticketId, reference	in_app
ticket.assigned	system	ticketId, reference	in_app
ticket.replied	system	ticketId, reference	in_app
ticket.resolved	system	ticketId, reference	in_app
ticket.refund_requested	system	ticketId, reference	in_app

Template availability does not mean every automatic producer uses the module. Activation completion, payment success, admin invites, company approvals, contract-readiness messages, and eSIM QR emails all go through the trigger with a dedupe key, so a distinct event occurrence is persisted once per recipient. Verified gateway (Moyasar or Paymob) sandbox payments are isolated from production fulfillment and payment.succeeded notifications; production and mock qualifying successes keep the existing best-effort, deduplicated notification behavior. OTP, magic-link delivery, and the email test send remain outside the module. The eSIM renderer supports QR attachments and describes a profile as ready to install, not installed.

activation.failed tells activation officers that an activation failed. The dispatch that runs after a committed move to failed triggers it with the role audience activation_officer, which resolves at send time to the active accounts holding that role. The parameters are ids and a code only, never reason text or personal data, because parameters are stored for each recipient. The in-app text is built when the notification is rendered, from the last four ICCID digits, the operator name, and a label for the code. The dedupe key is activation.failed:<eventId>, so each failed status event produces at most one notification per recipient. The template renders only for an active account whose role is activation_officer or super_admin, and refuses any other recipient. replayActivationFailed in apps/api/src/services/activation-notifications.ts triggers the notification again for failed events that never produced one. It runs after every activation recovery scan over the last 24 hours (see Recovery).

Ticket events are automatic triggers too. ticket.opened targets active support-role users; ticket.assigned targets the selected officer; ticket.replied and ticket.resolved target the customer; ticket.refund_requested targets the active users holding the refund decider role, and fires only when a refund request is created, not when an existing live request is returned instead. All five are category system, use only the in-app channel, and persist only { ticketId, reference } as parameters. Their dedupe keys are ticket.opened:<ticketId>, ticket.assigned:<auditId>, ticket.replied:<commentId>, ticket.resolved:<auditId>, and ticket.refund_requested:<ticketId>. They are transactional and do not honor opt-outs. Notification failure is best-effort and does not undo the ticket write.

Compliance Guard

In-app is always added. A disabled requested channel, including push, returns 400 NOTIFICATION_CHANNEL_DISABLED. A requested channel unsupported by the selected template returns the same error.

For non-transactional sends to all or role end_user, the server narrows the effective channels to in_app. Promotional email and SMS require prior consent that this endpoint does not capture. This rule does not narrow explicit users, company audiences, or other roles. Announcement opt-outs still apply for every audience. Transactional categories, including system, bypass opt-outs.

Acceptance

The route returns 202 with batchId, status, recipientsTotal, deliveriesQueued, deliveriesSuppressed, fullyEnqueued, scheduledAt, and effective channels.

Status	Meaning of the Acceptance Body
finished	Notification persistence completed. Counts come from the trigger, or persisted rows when recovering a missing snapshot. Channel handoffs can still be incomplete
scheduled	Only the future batch job was enqueued. Counts are zero and fullyEnqueued is true
running	Immediate execution failed and the same batch was queued for retry. Counts are zero and fullyEnqueued is false

deliveriesQueued counts eligible fresh or recovered deliveries, not confirmed provider sends. fullyEnqueued: false can also indicate a handoff-stamp or audit failure. Use batch delivery counts to inspect outcomes. A persistence failure that enters the retry queue returns 202 with status: running and fullyEnqueued: false, not an error body. 503 SERVICE_UNAVAILABLE with Retry-After: 1 is returned only when batch creation fails, when the fallback enqueue fails, or when a replay finds no stored response to recover the handoff.

Idempotency

The optional Idempotency-Key header is 1 to 255 characters and must not be blank. Its value is not trimmed. Scope is the sending administrator plus the exact key. Omitting the header provides no request replay protection.

The request fingerprint uses the validated body after defaults, trimming, and channel deduplication, but before channel narrowing. Object keys are sorted. Array order is preserved, so reordering an audience or channel array can produce a conflict.

A matching replay returns the stored 202 body for 24 hours after that response was built. When two identical requests overlap, the first stored body wins and both requests return it. It does not recompute delivery progress. A different validated request returns 409 NOTIFICATION_IDEMPOTENCY_CONFLICT. After expiry, the key can start a new send, even if the old batch has a future schedule.

An unfinished response reserves the key for 24 hours from batch insertion. Writing the acceptance body restarts the 24-hour window. Concurrent identical requests check for that body six times with five 20-millisecond waits. If it is still absent and the batch is unfinished, the route retries the same batch job with its stored schedule and effective channels. Unfinished-batch recovery returns zero counts and fullyEnqueued true only for a scheduled batch. A finished batch with finishedAt set and no cancellation reconstructs and stores a 202 snapshot without rerunning or enqueueing the batch; a concurrent snapshot winner remains authoritative. Recovery counts persisted notification rows for recipientsTotal, non-suppressed email/SMS delivery rows for deliveriesQueued, and suppressed delivery rows for deliveriesSuppressed. Any eligible email/SMS row missing queuedAt conservatively makes fullyEnqueued false, even if its outcome has advanced. A cancelled batch without a stored response still returns 503.

This header convention is separate from the payment body-field convention. Recipient/occurrence deduplication is also separate from this 24-hour replay window.

Recipient Search

GET /admin/notifications/recipients resolves a search term to picker rows for the composer's audience fields. It takes q (up to 100 characters, trimmed before validation) and kind. A company search needs at least 2 characters in q. A member or staff search may omit q or leave it empty, which lists the set. companyId is required when kind is member and ignored otherwise.

Kind	Matches	Scope
company	Company names	Every company
member	Portal user names	The company named by companyId, its active users whose role is not end_user
staff	Internal staff names	Active accounts whose role is super_admin, admin, activation_officer, or support

The response is { data: [{ id, name }] }, in name order and capped at 100 rows, the size of one company roster, so a term that matches more is narrowed by whoever typed it rather than paged through. A member or staff read with no term is bounded to that cap in the database; a read with a term covers the whole set, because names are matched after decryption. End users are never returned by name. They are addressed through the everyone or role audiences instead, never chosen individually. Company matching runs as a database query. Member and staff matching decrypts each candidate's name and filters in memory, because names are encrypted with a random initialization vector and cannot be matched by the database, and the candidate set for each is short and bounded by construction — one company's roster, or the fixed internal staff roles.

{
  "data": [
    { "id": "b2c3d4e5-f6a7-8901-bcde-f12345678901", "name": "Acme Logistics" }
  ]
}

Scheduling and Cancellation

A future scheduledAt creates a scheduled batch and a delayed job. An omitted, null, or past schedule runs immediately. Queue load can delay execution beyond the requested time. Expiry hides an inbox item but is not a delivery cancellation deadline.

POST /admin/notifications/batches/:id/cancel conditionally changes only scheduled to cancelled. The stored status decides a race with execution. Removing the delayed job is best-effort cleanup. A repeat returns the original { id, status: 'cancelled', cancelledAt }. A running or finished batch returns 409 NOTIFICATION_BATCH_NOT_CANCELLABLE. An unknown batch returns 404 NOTIFICATION_BATCH_NOT_FOUND.

Audit

Three actions in the audit trail cover the notification module.

Action	Entity	Written	Values
NOTIFICATION_TRIGGERED	notification_batch or notification_event	After a staff send (createdBy set) persists at least one notification. Automatic events write no row; their counts are in the notification and delivery rows	Event or template key, category, audience kind, and the trigger's recipient and delivery counts
NOTIFICATION_BATCH_CREATED	notification_batch	When an admin send accepts and inserts its batch row, before the handoff runs	Category, audience kind, channels, and scheduled time
NOTIFICATION_BATCH_CANCELLED	notification_batch	When a cancel's conditional update wins the race with execution	The settled status

None of the three rows carry message content. A failed NOTIFICATION_TRIGGERED write is one of the causes of fullyEnqueued: false on the acceptance response above. A failed NOTIFICATION_BATCH_CREATED or NOTIFICATION_BATCH_CANCELLED write is logged and does not change the response, since the batch row itself already committed.

Batch Reports

GET /admin/notifications/batches accepts cursor, limit from 1 to 100 with default 20, optional status, and optional category. Status values are scheduled, running, finished, and cancelled.

The response is { data, total, nextCursor, hasMore }. total is the page length, not the collection size. Each summary has id, category, status, title, severity, recipientsTotal, counts, createdBy, createdByName, createdAt, scheduledAt, startedAt, finishedAt, and cancelledAt. title is the message title in both languages and is null for a template send or a batch whose request has been scrubbed by retention. severity is the message importance and is null when absent or when the batch request has been scrubbed by retention. counts holds the same sent, delivered, failed, suppressed, and pending fields as the batch detail counts below, computed with one grouped read for the whole page rather than one query per row. createdByName is the sending administrator's display name, joined from their user row, and is null when the batch has no creator or that row is gone. Ordering and cursors use full-precision (created_at, id) descending.

GET /admin/notifications/batches/:id returns { batch, counts, failures }. It accepts cursor and limit for the failure list, with the same bounds. batch has the summary fields above, including its own nested counts and createdByName. The top-level counts repeats the same sent, delivered, failed, suppressed, and pending figures nested in batch.counts. Both come from the same single read of the delivery rows.

failures has { data, total, nextCursor, hasMore } and includes both failed_terminal and suppressed rows. Each item contains deliveryId, notificationId, channel, outcome, provider, attempts, errorCode, errorText, suppressionReason, and failedAt. No recipient address or message content is returned. Batch reads and send responses use Cache-Control: private, no-store.

Delivery Outcomes

Outcome	Meaning
queued	Persisted and awaiting a worker, possibly before Redis handoff
sending	A worker has claimed the delivery. Retry bookkeeping can record a failed attempt without changing this outcome
sent	The provider accepted the message, not proof of delivery or reading
delivered	In-app persistence succeeded, or a supported callback confirmed delivery
failed_terminal	Permanent rejection, exhausted attempts, or unrecoverable worker failure
suppressed	No send due to a recipient or provider gate

The progress order is queued to sending to sent to delivered. Callbacks can skip intermediate steps but never move backwards. delivered, failed_terminal, and suppressed never advance again. sent can advance to delivered or failed_terminal.

Suppression Reason	Meaning
account_inactive	Recipient is missing or inactive when the worker checks
channel_opted_out	Recipient opted out of a non-transactional channel
no_address	No usable email address or phone number for that channel
provider_unsupported	Selected provider has no notification send capability

Disabled channels create no delivery row, rather than a suppression row. Email under SMTP ends at sent because the adapter has no delivery callback. SMS uses the mock regardless of the OTP provider selection. Push has no implementation.

Retries and Recovery

Queue	Attempts	Exponential Base	Concurrency Per Worker	Queue Rate Limit
notification-email	5	5 seconds	5	30 jobs per 60 seconds
notification-sms	3	5 seconds	2	10 jobs per 60 seconds
notification-orchestration	5 by default	1 second	2	None

Callback jobs override the orchestration defaults with nine attempts and a 10-second exponential base. Urgent channel jobs use priority 1 and normal jobs use 5. Urgency does not bypass rate limits or introduce quiet hours.

Retryable failures record attempts, errorCode, errorText, and failedAt separately from the outcome. Permanent failures stop retries. Workers skip already accepted or terminal deliveries. Delivery IDs are deterministic channel job IDs. Notification rows also deduplicate by recipient and occurrence, independently of Redis job retention.

At startup and every minute, reconciliation selects up to 200 email/SMS deliveries. It selects queued rows or sending claims older than 60 seconds, with no handoff stamp or one older than 15 minutes. It repairs missed enqueueing, lost jobs, and abandoned claims. It then selects unfinished batches (scheduled or running) that stayed stale past the same 15-minute window, counting a scheduled batch as stale only once its scheduled time has also passed that window, and re-enqueues their run_batch job with the original scheduled time, so a batch that lost its worker is picked up again. It also re-drives unprocessed callback receipts whose enqueue stamp is missing or older than the same window. See the queue guide for selection and recovery limits.

SMTP can accept mail before a process dies or loses the acknowledgement. If acceptance was not recorded, recovery can send the message again. The 20-second SMTP deadline and claim fencing do not eliminate this duplicate window. Exactly-once email delivery is not guaranteed.

The HTTP rate counter is separate from queue limits. It is in memory per API process, with a default 100 requests per 15 minutes per IP. Notification callbacks remain subject to it even when valid browser credentials are supplied. It is not a shared cross-process budget.

Delivery Rate

GET /admin/notifications/delivery-rate requires offset-bearing from and to timestamps with from < to. Optional channel accepts any declared channel. Optional provider is a lowercase slug of 1 to 64 characters matching ^[a-z0-9][a-z0-9_-]*$.

The report filters delivery creation time using [from, to), then reads current outcomes. It is not a historical reconstruction. The response has from, to, buckets, and failureReasons.

Every bucket contains channel, nullable provider, total, sent, delivered, failed, suppressed, pending, and rate. sent includes delivered. pending includes queued and sending. rate = sent / total, with zero for an empty denominator. Suppressed and pending rows remain in the denominator. Counts are deliveries, not recipients. In-app success is immediate, so a combined rate is not proof of external delivery.

Failure reasons group terminal error codes and suppression reasons by channel and provider. Each entry has channel, provider, reason, and count. A missing reason is unknown. Successful retry history is not a terminal failure reason.

Retention and Export

GET /profile/export includes all owned notifications, including archived, expired, and scrubbed rows, with their stored parameters, state timestamps, and delivery outcomes. It includes channel opt-outs separately. It does not render inbox copy. Scrubbed parameters are empty. Batch requests and callback receipts are not included.

Account deletion marks existing notifications for the 365-day retention period. A later retention sweep clears eligible params and stamps deletedAt. It keeps notification rows, dedupe keys, and delivery rows. Expiry and archive alone do not trigger recipient-content scrubbing. Independently of recipient deletion, the sweep replaces a batch's request with channels-only metadata when it has been finished or cancelled for at least 24 hours and replayExpiresAt has expired or is null. It preserves the batch row, response snapshot, fingerprint, and dedupe history, and does not remove callback receipts or opt-outs. See notification data lifecycle for the retained fields and limits.

Rows created after a deletion request, for example a live event or a custom send reaching an account that already asked for deletion, receive that request's own deletionRequestedAt and retainUntil at write time. The request is recognised through the deletion audit event, because the user row keeps no request timestamp, so an account that was only deactivated by an administrator receives no stamps. The recipient read, the audit lookup and the insert run in one transaction, and the deletion request commits its flag, its stamps and its audit event together, so a request racing the send cannot leave a row unstamped.

Errors

HTTP Status	Code	Condition
400	VALIDATION_ERROR	Invalid body, parameters, cursor, schedule, template category mismatch, template parameters, language content, or idempotency header
400	NOTIFICATION_CHANNEL_DISABLED	Disabled channel or unsupported template channel
401	AUTH_REQUIRED	Missing or invalid authentication on protected routes
403	AUTH_FORBIDDEN	Caller is not super_admin and lacks the required notifications area level on an administrative route
404	NOTIFICATION_NOT_FOUND	Missing, foreign, or scrubbed inbox item
404	NOTIFICATION_BATCH_NOT_FOUND	Unknown batch
409	NOTIFICATION_BATCH_NOT_CANCELLABLE	Batch execution has already started
409	NOTIFICATION_IDEMPOTENCY_CONFLICT	Same administrator and key with a different validated body
429	RATE_LIMIT_EXCEEDED	Applicable HTTP request budget exceeded
503	SERVICE_UNAVAILABLE	Administrative acceptance or recovery could not complete, with Retry-After: 1

The shared catalogue also declares NOTIFICATION_INVALID_CUSTOM_CONTENT, NOTIFICATION_UNKNOWN_EVENT, and NOTIFICATION_OPT_OUT_NOT_ALLOWED. Current HTTP schemas reject these invalid inputs as VALIDATION_ERROR before those catalogue codes can be returned. See all notification error messages and callback errors.