Golden path: diagnose a failed billing webhook

When to use

Use this path when provider delivery returns non-2xx, a customer paid but local access did not change, a lifecycle email is missing, a duplicate appears, payment recovery is wrong, or local status disagrees with the selected provider. Treat this as a financial and access-control incident.

Do not manually toggle users.is_premium as the first response. Establish the signed event, ordering, correlation, transaction, and entitlement facts.

Files and boundaries

packages/billing/src/providers/<selected>.ts: signature verification, provider SDK parsing, and normalization.
apps/server/src/domains/billing/routes.ts: public webhook endpoint and HTTP response.
apps/server/src/domains/billing/service.ts: claim transaction and lifecycle processing.
apps/server/src/domains/billing/repository.ts: event ledger and provider-neutral persistence.
apps/server/src/domains/billing/entitlements.ts: access result after state is stored.
billing_webhook_events: successfully committed (provider, event_id) claims.
subscriptions.provider_modified_at: provider ordering authority.
application logs/provider delivery history: request ID, event ID/type, response, retry.

Provider SDK objects do not cross into product domains. Email/notification work happens after the billing transaction and cannot determine whether the webhook is acknowledged.

Procedure

Capture the provider environment, event ID, event type, delivery time, response status, local request ID, subscription/customer IDs, and expected customer outcome. Never copy signatures, tokens, complete payloads, email addresses, or card data into a ticket.
Classify the failure:
- 403: missing/invalid standard webhook ID or signature headers;
- 503: webhook integration is intentionally unconfigured;
- 5xx: verified event failed correlation, plan mapping, or state mutation and should retry;
- 2xx with no new state: duplicate/stale/unhandled event or a defect;
- state correct but message absent: post-commit notification failure.
Check billing_webhook_events for the exact provider/event ID. Presence means the state transaction committed. Absence after a 5xx means the claim rolled back and retry is safe.
Compare provider modifiedAt with subscriptions.provider_modified_at. A lower timestamp is stale and should be claimed without overwriting state. Never compare provider time with local updated_at.
Verify application-user correlation from provider customer externalId or signed metadata.user_id. Confirm that the UUID names an active local user. Repair provider metadata and retry the same event rather than inventing a local customer.
Verify the provider product maps to exactly one configured plan and agrees with signed metadata.plan_type. Repair deployment mapping or provider metadata; do not default to monthly.
Inspect the local subscription status/period and evaluate it through the documented policy. past_due denies paid access but keeps billing recovery available; a canceled subscription grants access only before its period end.
Retry from the provider only after the cause is corrected. Reusing the same event ID proves transactional retry; replaying a committed event must be a no-op.
If state committed but notification failed, resend the communication through a deliberate support path. Do not replay or mutate billing merely to trigger email.
Add a redacted regression fixture for any new failure class, then run the full DB-enabled suite.

Verification

vp check
vp run --filter @app/server typecheck
RUN_DB_INTEGRATION_TESTS=1 \
  DATABASE_URL='postgres://localhost:5432/app_billing_webhook_proof?sslmode=disable' \
  bun test --cwd apps/server src/domains/billing/service.integration.test.ts
vp run -r test
git diff --check

The integration proof must retain invalid signatures, concurrent identical delivery, rollback and same-ID retry, missing-user repair, unknown product rejection, out-of-order provider timestamps, payment failure, and entitlement-cache assertions.

Security constraints

Never bypass signature verification or expose the webhook secret.
Do not paste raw provider payloads or customer data into git, Linear, chat, or fixtures.
Query by opaque IDs and redact email/customer metadata from shared evidence.
Do not manually grant access before establishing whether the provider reports payment.
Keep recovery endpoints reachable while paid product APIs are denied.
Use sandbox events for reproducible tests; never replay production events into local databases.

Failure modes

Looking only at users.is_premium and ignoring the subscription and event ledger.
Treating a 2xx duplicate as evidence the original mutation succeeded without checking the claim.
Retrying an invalid signature rather than fixing endpoint/secret configuration.
A missing user is acknowledged and permanently lost.
Local updated_at is mistaken for provider event ordering.
An unknown product silently maps to an existing plan.
Replaying billing state to repair a post-commit email.
Making an ad hoc SQL update with no provider reconciliation or regression test.

Rollback and diagnosis

If a code release caused failures, keep provider retries pending, restore the last verified webhook processor, and replay only unclaimed events after the fix. Do not delete committed ledger rows to force a replay; create an explicit reconciler if already-committed state was wrong.

If a migration caused the failure, follow the failed-migration runbook. Preserve evidence before rollback and ensure schema/application versions agree. For a compromised webhook secret, rotate it in the provider and the deployment, reject the old value, and do not log either secret.

Acceptance criteria

The incident has a redacted event ID/type, expected outcome, actual status, and root cause.
Signature, ledger claim, correlation, plan mapping, provider ordering, local state, and entitlement were checked.
Retry or replay behavior is deterministic and does not duplicate state.
Customer access and billing recovery match Decision 0003.
A regression test covers any newly discovered class.
Root checks and a generated-buyer diagnostic run pass with interventions recorded.

Agent prohibitions

Do not disable signature verification or transactional event claiming.
Do not delete event-ledger rows or toggle premium as a shortcut.
Do not leak payloads, signatures, tokens, email addresses, or payment data.
Do not compare provider timestamps to local write time.
Do not acknowledge an uncorrelated paid subscription as successfully processed.
Do not claim resolution until provider state, local state, access, and retry behavior agree.

Diagnose a failed webhook