---
title: Scheduled jobs
description: API commerce recovery and independent diagnostic and operator-history retention.
---

The API Worker owns hourly commerce jobs (`0 * * * *`); the storefront has no scheduler. The backoffice is a deliberate
exception: its own hourly handler cleans the local operator-access ledger in BackofficeDatabase. It never exposes
operator sessions to the API.

The five jobs run in this order:

1. `recover_pending_cancellations`
2. `recover_financial_operations`
3. `expire_agent_invitations`
4. `clean_conversations`
5. `clean_dpop_proofs`

Each acquisition writes `job_leases(job, holder, lease_id, acquired_at, expires_at)` atomically. The lease lasts 30
minutes. An unexpired row prevents another invocation from running that job; an expired row can be taken over.
A job renews its fenced lease before each batch. A previous owner cannot renew or release a
successor's lease. A lost lease stops that job and the scheduler continues with later jobs.

Cancellation and financial recovery have renewal checkpoints between batches. Financial recovery also expires
unconfirmed account intents, recovers deleted-account payments and pending returns, redacts terminal deleted orders and
retries card operations. Invitation expiry only removes expired invitations; invitation delivery and delegated-session
provider cleanup are explicit requests. Retention is one bounded SQL deletion of at most 100 conversations and needs no
loop checkpoint. `clean_dpop_proofs` deletes expired rows of the shared DPoP replay record (`dpop_proofs`) in batches
of 1,000, renewing its lease before each batch, for at most 100 batches per run.

A job failure releases its lease and rethrows. Account email and deletion provider work and notification delivery do
not run in cron: a pending account operation resumes only through the customer's explicit retry on the signed-out
status page or the [operator recovery endpoint](/runbooks/account-recovery). If an isolate running a scheduled job
dies, a later hourly invocation can acquire that job after its lease expires; this does not resume account provider work. Existing per-operation authority and provider idempotency remain in force.

## Diagnostic and history retention

The API independently attempts bounded diagnostic cleanup alongside financial recovery, waits for both and preserves
the original financial error. A recovery failure cannot skip retention. Diagnostic chunks select at most 1,000 expired
parents, cascade refs/header/token children and release recorded retained charges once. Minute write charges are not
refunded. Customer activity also expires after 30 days; orders/payment records remain unchanged. Backoffice ledger cleanup
runs separately. Every reader applies live expiry while deletion catches up, including after restore.

See [diagnostic investigations](/runbooks/audit-investigations) for capture health, default-off flags and credential policy.

## Diagnose a run

Inspect `scheduled_job` logs by `worker`, `stage` (job name) and `outcome`. Outcomes are `ran`,
`skipped_leased`, `lease_lost`, and `failed`. No request or notification payload is logged.

On the API's D1 binding, read:

```sql
SELECT job, holder, acquired_at, expires_at FROM job_leases ORDER BY job;
```

Successful runs release their rows, so an empty table after completion is expected. Capture a live row during a
run to observe its holder; use completion logs to prove the owner of completed runs. Do not expect `holder='api'`
rows to remain until the following hour.

To stop new commerce scheduled work during an incident, disable the API cron trigger. This also interrupts its retention
branch; the independent backoffice cron remains configured. Reads still hide expired data. A job already running keeps its lease until completion or expiry.