Card-Platform Job Spine

Work that should not run inside an HTTP request — SMS sends, slow third-party polls, partner callbacks, daily reports — gets shared rails: a producer writes a job to a durable queue and answers immediately; a separate workers service does the slow part, with retries, dead-lettering, and tracing built in. Every background feature is then a queue definition and a handler.

Waiting for the rails, roughly in order:

The first real consumer is Wallet Provisioning OTP delivery — until it ships, the spine is exercised end-to-end by a temporary hello smoke path (covered below).

Three parts, no new infrastructure: producers (today events-ingest) enqueue with a send-only client; the queue is pg-boss tables in the pgboss schema of the card-platform Postgres — the same database the domain rows live in; the workers service, a separate deployable, polls and runs handlers.

Why the queue lives in the domain database

Because the queue is just tables, enqueueing can join the domain transaction — pg-boss's fromPrisma adapter (re-exported by @setldpay/queue) runs the enqueue INSERT on the same Prisma transaction as the domain write:

import { fromPrisma } from '@setldpay/queue'

// the pattern every real producer will follow — enqueue inside the
// domain transaction, acknowledge the caller only after commit
await prisma.$transaction(async (tx) => {
  const row = await persistNotification(tx, parsed)
  await sender.send(someQueue, { notificationId: row.id }, { db: fromPrisma(tx) })
})

Rollback, and the job vanishes with it — no ghost job pointing at a row that never committed. Commit, and the job is durably committed too — no lost job from a process dying between persist and enqueue. A transactional outbox without building one.

This section is pg-boss machinery rather than our code, but it is the runtime contract every producer and handler inherits. The wrapper pins the defaults; per-queue overrides go on the worker registration.

The job state machine

Net effect: delivery is at-least-once — expiration redelivery means a job can run twice, so every handler must be idempotent: check whether the work is already done before doing it.

Claiming work — why concurrency is safe

WITH next AS (
  SELECT id FROM pgboss.job
  WHERE name = 'hello' AND state < 'active'  -- created and retry sort below active
    AND start_after < now()
  ORDER BY priority DESC, created_on LIMIT 1
  FOR UPDATE SKIP LOCKED                       -- row already locked? skip it, never wait
)
UPDATE pgboss.job SET state = 'active', started_on = now(), … RETURNING *

The worker loop — what “every 2 s” means

One Node thread per replica: handlers that await I/O interleave across all queue loops; a CPU-bound handler stalls every loop in the process.

The knobs

Only the workers service runs pg-boss's supervisor (supervise: true) — expiration, retention, and queue monitoring happen there. Senders run none of it.

DLQ jobs keep their payload, so they can be inspected and replayed from the dashboard once the bug is fixed.

packages/shared/queue wraps pg-boss in three primitives, each owning one side of the contract. Services never import pg-boss directly.

defineQueue — the typed registry

Declared once in card-platform/common and imported by both sides, so payload typing is end-to-end: queue name, TypeBox schema, retry options. Every queue gets <name>-dlq unless it opts out with deadLetter: false.

// packages/card-platform/common/src/queues.ts — the context's queue registry
export const helloQueue = defineQueue({
  name: 'hello',
  schema: Type.Object({
    message: Type.String({ minLength: 1, maxLength: 256 }),
    /** When true the handler throws — exercises retry → DLQ → warning. */
    fail: Type.Optional(Type.Boolean()),
  }),
  options: { retryLimit: 2, retryDelay: 1, retryBackoff: true },
})  // deadLetter defaults to 'hello-dlq'

createQueueSender — send-only, migrate-free

Deliberately powerless: no migration, no maintenance, no scheduling. It validates the payload before sending (fail loud at the producer) and starts lazily on first send. Deployed before the workers service has created the schema? Sends fail — and the sender rebuilds its pg-boss instance per send, so it self-recovers once the worker migrates, no restart.

createWorkerRuntime — owns the schema, runs the handlers

Sole owner of the pgboss schema. Start: migrate (advisory-lock protected) → provision every queue and DLQ, converging options on existing ones → poll. Per job: re-validate the payload (invalid → DLQ), wrap the handler in an OTel span (queue.process <name> with job id, queue, attempt), hand it a child logger with the same context. stop() drains in-flight jobs up to 25 s — under the ECS 30 s stopTimeout.

Rolling deploys make producer/consumer version skew routine — jobs enqueued under yesterday's shape meet today's consumer, and vice versa.

When a rule is violated anyway, the payload fails consume-side validation and routes to the DLQ intact — visible, alertable, replayable. Never silent loss.

One knob exercises the whole machine: the payload is message plus an optional fail flag that makes the handler throw.

mise run dev includes the spine: workers (nodemon hot reload, no published port — health probed in-container) and dashboard, the official pg-boss dashboard on host port 3013. The dashboard starts only after workers is healthy, because workers is what migrates the pgboss schema.

mise run dev                  # full local stack — postgres, services, workers, dashboard
mise run queue:dashboard      # pg-boss dashboard against the local compose DB (port 3013)

# against a deployed env — requires --owner until read-only grants land,
# and owner mode refuses to start without the basic-auth env vars
mise run queue:dashboard -- --env dev --owner

Without --env the dashboard points at local compose; with --env <env> it pulls the deployed database URL from Secrets Manager and runs on your dev box — no exposed admin surface. Two interim constraints: --owner is required until read-only grants land, and owner mode refuses to start without basic-auth credentials (PGBOSS_DASHBOARD_AUTH_USERNAME / PGBOSS_DASHBOARD_AUTH_PASSWORD).

Two ordering rules and one configuration invariant keep producers and the consumer in step:

The schema-agreement rule is why both services default QUEUE_SCHEMA to pgboss and treat overriding it as a deliberate, paired change.