Scheduled jobs

API commerce recovery and independent diagnostic and operator-history retention.

The API Worker owns hourly commerce jobs (0 * * * *); the storefront has no scheduler. The backoffice is a deliberate exception: its own hourly handler cleans the local operator-access ledger in BackofficeDatabase. It never exposes operator sessions to the API.

The five jobs run in this order:

1. recover_pending_cancellations
2. recover_financial_operations
3. expire_agent_invitations
4. clean_conversations
5. clean_dpop_proofs

Each acquisition writes job_leases(job, holder, lease_id, acquired_at, expires_at) atomically. The lease lasts 30 minutes. An unexpired row prevents another invocation from running that job; an expired row can be taken over. A job renews its fenced lease before each batch. A previous owner cannot renew or release a successor’s lease. A lost lease stops that job and the scheduler continues with later jobs.

Cancellation and financial recovery have renewal checkpoints between batches. Financial recovery also expires unconfirmed account intents, recovers deleted-account payments and pending returns, redacts terminal deleted orders and retries card operations. Invitation expiry only removes expired invitations; invitation delivery and delegated-session provider cleanup are explicit requests. Retention is one bounded SQL deletion of at most 100 conversations and needs no loop checkpoint. clean_dpop_proofs deletes expired rows of the shared DPoP replay record (dpop_proofs) in batches of 1,000, renewing its lease before each batch, for at most 100 batches per run.

A job failure releases its lease and rethrows. Account email and deletion provider work and notification delivery do not run in cron: a pending account operation resumes only through the customer’s explicit retry on the signed-out status page or the operator recovery endpoint. If an isolate running a scheduled job dies, a later hourly invocation can acquire that job after its lease expires; this does not resume account provider work. Existing per-operation authority and provider idempotency remain in force.

Diagnostic and history retention

The API independently attempts bounded diagnostic cleanup alongside financial recovery, waits for both and preserves the original financial error. A recovery failure cannot skip retention. Diagnostic chunks select at most 1,000 expired parents, cascade refs/header/token children and release recorded retained charges once. Minute write charges are not refunded. Customer activity also expires after 30 days; orders/payment records remain unchanged. Backoffice ledger cleanup runs separately. Every reader applies live expiry while deletion catches up, including after restore.

See diagnostic investigations for capture health, default-off flags and credential policy.

Diagnose a run

Inspect scheduled_job logs by worker, stage (job name) and outcome. Outcomes are ran, skipped_leased, lease_lost, and failed. No request or notification payload is logged.

On the API’s D1 binding, read:

SELECT job, holder, acquired_at, expires_at FROM job_leases ORDER BY job;

Successful runs release their rows, so an empty table after completion is expected. Capture a live row during a run to observe its holder; use completion logs to prove the owner of completed runs. Do not expect holder='api' rows to remain until the following hour.

To stop new commerce scheduled work during an incident, disable the API cron trigger. This also interrupts its retention branch; the independent backoffice cron remains configured. Reads still hide expired data. A job already running keeps its lease until completion or expiry.