Diagnostic investigations

Follow observed actions across applications and providers, with bounded demo inspection and explicit gaps.

Diagnostic events explain observed application, authentication and provider work. They supplement customer activity and the separate operator-access ledger; they are not a lossless trace or proof of whole-operation success. The feature is locally implemented. These instructions do not claim deployment, active tenant streams or live provider verification.

Setup and source ownership

The API retains five diagnostic tables in existing ShopDatabase: events, exact references, request headers, typed token attachments and fixed-cardinality health. BackofficeDatabase separately retains operator sessions and its access ledger. Storefront, MCP and backoffice emit through their own named private API producer capabilities; the storefront uses that capability even though it has business access to ShopDatabase. Proxy Auth0 gets no app recorder or database capability.

Apply the normal ShopDatabase and backoffice migration chains. Configure API AUDIT_ENABLED=true, positive AUDIT_DATABASE_CAPACITY_BYTES, AUDIT_ENVIRONMENT and exact AUDIT_TENANT. Missing capacity disables admission. Alchemy forwards the configuration and producer-specific bindings. Preserve environment/tenant when disabling capture so cleanup can continue. A tenant name is not inferred from a custom login domain.

Genuine settings also require configured log/event stream IDs and exact CloudEvents source; AUTH0_AUDIT_EVENT_MODE defaults disabled. Prefer direct with a separate audit Event Stream, preserving existing Auth0 proxy lifecycle handling. If the tenant cannot provide a second destination, use forward with the Auth0 proxy’s exact AUTH0_AUDIT_FORWARD_URL and independent matching AUTH0_AUDIT_FORWARD_TOKEN. Synthetic delivery uses PROXY_AUTH0_AUDIT_API_ORIGIN. Do not reuse lifecycle/service secrets. Check tenant Log Streaming entitlement and Event Stream capacity before provisioning; local fixtures prove neither. Check deployed bindings, cron and actual delivery separately after owner activation.

The local API generator forwards the dedicated synthetic token separately from genuine credentials. Proxy Auth0 local bindings keep business secrets required while forward/synthetic/tenant/demo/correlation settings are independently optional. Leaving diagnostics unconfigured does not require dummy audit secrets or activate tenant streams.

Investigate observed facts

Open Diagnostic events as an auditor. Filter to a time/source/service or exact resource reference before opening the right-hand sheet. It contains Overview, Event JSON, Headers, Tokens, Actor chain and Timeline. Timeline offers Events and Flow. Metadata never contains retained raw credentials. The local operator ledger remains separately readable.

Source framing, producer, simulation and execution are different facts. Genuine direct Auth0 and “Auth0 · forwarded by Auth0 proxy” are distinct from simulated proxy-auth0. An event collected by the API may have executed in Storefront, MCP, backoffice, an actual Auth0 proxy/Shop Durable Object, a scheduled job or the provider. Runtime class/instance/invocation and business workflow run/step/known attempt remain distinct. An ordinary checkout function is not a Cloudflare Workflow. Original provider HTTP headers/invocation are unknown; webhook delivery headers cannot replace them.

Verified OAuth actors are current-first, with intermediate/initiating identities and represented account separately labelled. Native Poppy nested-actor refusal is unchanged. Target or public subject claims are not verified authority. Existing customer activity may project a narrower business attribution than the diagnostic actor chain.

Actor-chain coverage is explicitly complete, incomplete or invalid; historical absence means not supplied. Completeness describes structure, not token verification. Supported delegation includes at most four ordered actors plus separately represented account; malformed/oversized arrays are omitted whole instead of displaying a complete-looking prefix. Safe coverage reasons include malformed input, actor limit and reference limit.

Primary controls are Action ID / Service / Event. Time from and through use Europe/Brussels with Apply time. Additional groups cover Source and result, Execution and intent, Network and Exact references (N/8). References are exact AND predicates; Clear filters starts fresh. A selected eventId is URL-only: it preserves current filters/page/scroll, never enters list/API input, and loads detail independently without another list read. Close removes selection and returns focus. Timeline keeps its own bounded cursor history and disables navigation while pending.

A non-secret owned action ID survives child calls and validated continuations. Distinct later commands remain separate, with explicit links; sharing a user, session, order or cart does not merge them. Public trace/correlation strings stay claims. Exact private sender evidence plus actual API response request ID and matching transport facts can establish traversal. Parent causality, provider association and related resource are different edges. Missing or ambiguous evidence stays missing. Do not infer edges from timestamps alone.

The query contract supports exact source/producer/simulation, service, category/channel, event/outcome, action intent/ID, runtime/trigger, direct client/provider/either IP, ASN/country, times and exact references. Either-IP matches the directly observed client or provider once; all other supplied conditions remain ANDed. At most eight supplied exact references are allowed; absent namespace is exact empty namespace, never a wildcard. Actor/session, runtime class/instance, workflow name/run/step/known attempt and business resource references retain their recorded namespaces and trust labels. Shared normalization canonicalizes country/IP/reference conjunctions before cursor hashing, rejects unknown keys and preserves boolean false. Matching a reference does not establish verified authority.

Pages are bounded keysets (default 50, maximum 100 events); action traversal stops at 128 identities. Unloaded/expired parents remain placeholders. Refresh for late arrivals or newly resolved evidence before the current cursor. Expiry and complete query eligibility apply before paging. A partial page cannot establish whole-action success or duration.

Optional AUTH0_LOGIN_CORRELATION_VERIFIED and AUTH0_MANAGEMENT_CORRELATION_VERIFIED default disabled. Enable only after independently verifying the tenant capability. A successfully prepared protected login URL is not a provider HTTP attempt. A Management write handed to transport is not proof of provider log delivery. A later exact tenant/client log echo may associate the action, without proving an exact hop. Direct grants/refresh/MFA/reads have no universal external mapping; Event Stream transaction-state matching remains unsupported.

When actor lookup is unavailable

The legacy actor-index migration uses only slack already reserved by older events, preserving snapshots, hashes, expiry and budget totals. Exceptional unfunded retained rows make actor-filter queries return 503 audit_query_unavailable until they expire, instead of partial results that look complete. Non-actor investigations remain available. A narrower time range cannot make an incomplete retained deployment index trustworthy; keep that gap visible.

Demo credential inspection

DEMO ONLY: Capture retains up to 2 KiB of incoming headers, excluding cookies and known cookie-forwarding headers, plus selected access and ID tokens. Other retained credentials are unredacted. Never enable it in production: readers may reuse live tokens or service credentials. Use isolated demo credentials.

AUDIT_CAPTURE_DEMO_CREDENTIALS=true is independently required at producer and collector. Disabling it also denies old raw reads. The whole serialized header entry array is at most 2,048 UTF-8 bytes; oversized entries are omitted whole. Cookie/Set-Cookie and Guest-Session/poppy-browser-session forwarding headers are excluded before capture/hash. Each receiving HTTP invocation owns one snapshot; RPC/alarm has none. Incoming tokens cannot be recovered from omitted headers.

Typed response capture selects eligible user/agent/operator-user access and ID fields only: at most eight intact 32,768-byte tokens. Refresh tokens, codes, state, cookies, arbitrary bodies/queries and response headers are excluded. Machine/cache reuse is metadata-only. Local JWT decode is unverified and fetches no URLs; verification labels reflect actual evidence.

Raw release requires both the awaited local access ledger and an admitted value-free API release event. A failed gate withholds values. Sheet cache is ephemeral, reused across tabs and cleared on selection/close/logout/expiry/disabled capture. No raw values belong in ordinary JSON, lists, SSR, URLs, telemetry or persisted browser storage.

Access auditing does not prevent replay. Access tokens keep scoped/delegated resource authority. An ID token may reveal personal claims but is not automatically API authority. A retained backoffice machine bearer can make operator reads and claim staff attribution; storefront/agent-forwarder credentials keep existing guest/forwarding permissions. A lifecycle bearer can forge supported revocations, a stream bearer can forge events in its own namespace, and a Hub loopback bearer can call internal turn/timeout routes. Dedicated provider ingress does not capture its delivery bearer as original provider headers. These are capabilities of values if retained, not a claim that every configured secret is captured.

Health, budgets and retention

Application invocations allow 32 observations. Overflow records bounded lower-bound health, not another failure event; interrupted delivery or failed health storage leaves additional loss unknown. No healthy collector can prove every producer delivered. Business work is never retried by diagnostics; a total shared database outage can still stop commerce.

Current ceilings are 100,000 retained events, charged bytes at most min(100 MiB, 20% of configured database capacity), and1,000 new events/5 MiB charged writes per fixed minute. Bytes may bind first. These are allocations, not physical free-space or sustained throughput guarantees. First accepted events are immutable; identical retries do not recharge, changed retries cannot replace children. Provider audit ingress persists before acknowledgement and requests retry on storage failure. There is no queue/outbox. Local 100,000-event query fixtures do not prove production D1 ingestion capacity.

Charges cover serialized snapshot/scalar copies, parent allowance, each final reference and raw child. Short references share pages; references wider than 768 serialized bytes retain overflow allowance. The actual formula and reproducible scripts are in the repository operational guide. Six separate 100-admission shapes charged 1.287–2.206 times their measured local file growth, with unchanged ceilings and isolated clocks advancing across buckets. This validates those shapes, not sustained ingestion or universal physical free space.

The 100,000-event query fixture includes 5,000 exact 2 KiB snapshots and 1,000 typed token children, but deliberately bypasses admission and occupies about 920 MiB locally, above production allocation. It is query/cleanup evidence, not 100,000 accepted runtime events. A local 50-row either-IP query took about 1 ms; an evidence-resolved action 110 ms. Direct 10,000-action lookup varied 747–1191ms across runs. These are hardware/cache-sensitive measurements, not guarantees or live D1 throughput. Sparse IP union and selective evidence joins avoid unrelated scans; category-only measurements justified its own index. A partial unavailable-actor index bounds legacy coverage checks. No index-per-filter or universal intersection is promised.

Diagnostics, customer activity and the local operator ledger expire after 30 days. Reads hide expired rows immediately, including direct IDs and linked raw parents. Replay cannot restart the clock. Diagnostic cleanup cascades children and releases retained charges once; minute charges are not refunded. API cleanup runs independently of financial recovery; backoffice has its own hourly ledger cleanup. See scheduled jobs. Orders/payments retain their existing business records; customer history deletion cannot erase operator/security evidence. Restores must keep current-time expiry and resume bounded cleanup before exposing history. Ordinary D1 backups remain accepted; there is no backup-erasure guarantee.

Verify locally

Repository operational details live in docs/audit-investigations.md and its linked catalogs. Audit tests use real migrated SQLite; native API/Auth0 proxy fixtures use workerd; browser/operator fixtures use deterministic local provider responses. Source-derived stream fixtures are sanitized examples, not live tenant captures. Run the affected tests, builds and mocked deployment-configuration checks; apply migrations and perform tenant/deployed activation preflight separately.

See components, callers and tokens, backoffice request flow and deployment.

Genuine Auth0 ingestion produces one API-local receiving observation even on rejection; imported provider events never inherit its headers. Optional raw capture can retain the ingestion bearer, whose authority is submission to that diagnostic destination. Keep production credentials out of demo capture. Independent refreshes have separate actions, lifecycle effects retain bounded per-event provider parents, and exact service-pair investigations traverse either endpoint without changing local action IDs or128-action/100-event bounds. Callback provenance and actor structural completeness remain distinct.

Review corrections for execution and retention

The canonical runtime is cloudflare-workflow only for an actual Cloudflare Workflow; ordinary scheduled orchestration remains worker. API diagnostic cleanup attempts at most ten 1,000-event chunks each hour, stopping on a short batch and resuming backlog next hour. It runs independently of financial recovery and has its own scheduled execution step. Backoffice ledger cleanup records one bounded started/completed Worker execution pair through its private producer binding; diagnostic sink failure never gates cleanup.

Genuine Auth0 and synthetic Auth0 proxy ingestion each produce one API receiver observation, without recursive ingestion instrumentation. Storefront and agent API transports supply explicit trace context; public trace headers remain claims. Idle retained-header/token sheets do not poll audited health. Raw cache still clears on close, selection, leaving/hiding the page, expiry, observed access failure and disabled-capture responses. Explicit releases retain both awaited access gates.