API E2E reliability gate

Purpose and authority

The API E2E reliability gate is the mechanical release boundary for changes to the Rust API, tenant schema, authentication, authorization, billing, respondent flows, and form lifecycle. It is intentionally disposable and must never target production. The workflow creates its own PostgreSQL 18 database, Valkey 8 instance, five runtime roles, signed RLS context key, admin fixture, Rust API, and loopback provider doubles.

The executable contract is:

• workflow: .github/workflows/api-e2e.yml
• provisioning: tests/api-e2e/ci/provision.ts
• orchestration: tests/api-e2e/ci/run-gate.sh
• black-box suites and lower-level setup contract: tests/api-e2e/
• controlled-run contract: tests/api-e2e/controlled-agent-contract.ts
• reviewed adversarial plan: tests/api-e2e/agent-scenarios/baseline-security.json
• controlled-run orchestration: tests/api-e2e/ci/run-controlled-agent.sh

The gate does not read GitHub environments, deployment secrets, production credentials, or provider credentials. API_E2E_ALLOW_DISPOSABLE_DATABASE=true is required. Before role creation or migration, the provisioner validates all nine application, fixture, privileged, and migration DSNs: each must use a numeric loopback host, explicit port and user, no URL options or fragment, exactly one decoded database name, and a distinct ci, e2e, or test segment. The API, Valkey, and provider targets also require parsed numeric loopback URLs with explicit ports; userinfo host tricks, query/fragment overrides, and non-loopback destinations are rejected.

Mechanical PR gate

Every pull request and push to main runs the twelve files below explicitly and sequentially. Directory discovery is deliberately avoided: adding a test file does not silently make it a release blocker without an accompanying review of its isolation, timeout, fixtures, and provider behavior.

1. current-stack.contract.test.ts — health/OpenAPI, Better Auth signup and verification, tenant/workspace/form CRUD, public and respondent submission, embed/PAPI, admin, storage, AI, and OAuth contracts.
2. auth-role-isolation.test.ts — separated app/auth credentials, global identity confidentiality, session forgery resistance, and agent challenge lifecycle.
3. phase1-security.test.ts — storage authorization, role grants, RLS signing key isolation, purge capability, and SECURITY DEFINER boundaries.
4. tenant-boundary.test.ts — cross-team forms, workspaces, submissions, respondents, API keys, themes, branding, and team switching.
5. privilege-escalation.test.ts — admin, team/workspace role, billing/quota, API-key scope, and unauthenticated escalation attempts.
6. workspace-rbac-invariants.test.ts — canonical role behavior and durable owner/admin invariants, including concurrent removal.
7. submission-editing.test.ts — editor/viewer authorization, optimistic conflicts, append-only audit rows, tenant boundaries, and server-side filtering before pagination.
8. submission-attachments.test.ts — private immutable uploads, exact submitter binding, concurrent object verification, quota reclamation, signed downloads, and retention cleanup.
9. respondent-registration-security.test.ts — canonical password registration/login, assigned-form access, takeover, production mail receipt, recipient throttling, one-time link races/replay/expiry, SECURITY DEFINER boundaries, and tenant/form/team revocation.
10. form-deletion-lifecycle.test.ts — immediate undeploy, restoration, audited purge, quota recovery, and concurrent restore limits.
11. billing-lifecycle.test.ts — deterministic Stripe checkout/webhook/portal, reconciliation, deletion cleanup, and billing races.
12. race-conditions.test.ts — ownership limits, concurrent mutation, replay, and tenant lifecycle races.

Test files share fixed loopback provider ports and some intentionally adversarial database state, so files must not run in parallel. Within a file, the Bun test runner retains the suite's declared semantics.

Required thresholds

• All enabled assertions in all twelve files pass; skipped opt-in coverage is a gate configuration defect, not an acceptable result.
• Every current journaled migration applies to a blank PostgreSQL 18 database. The provisioner derives the expected count from the committed Drizzle journal, so a future migration automatically raises the threshold.
• All nine database URLs, including MIGRATION_DATABASE_URL, pass the same exact numeric-loopback/disposable-name parser before any role or DDL work.
• Runtime roles match their exact attributes: only kinoforms_service has BYPASSRLS; no KinoForms role is superuser, inherits, creates roles/databases, replicates, or belongs to another runtime role; kinoforms_owner cannot log in.
• Every public application table is owned by kinoforms_owner, and no runtime login can read rls_context_signing_keys.
• On a blank CI database, migration 0025_partition_ownership_hardening must leave every actor_request_events child owned by kinoforms_owner. Existing deployments use the separate privileged deploy-time repair and fail-closed owner verification because the sealed migration login cannot re-own a legacy partition that is still owned by postgres.
• The API reaches a database-connected health state before tests begin.
• Each file finishes within 300 seconds, API startup within 90 seconds, and the complete GitHub job within 45 minutes. A timeout is a failure.
• No external network provider is used by a test. Resend, R2, OpenRouter, Google OAuth, and Stripe behavior is supplied by loopback doubles with deterministic fixtures.

The job is fail-fast after the first failing file. Continuing against a state left by a failed destructive or race suite can produce misleading secondary failures.

Artifacts and failure triage

The workflow uploads artifacts/api-e2e/ and the bootstrap logs in artifacts/api-e2e-bootstrap/ on success and failure for 14 days. They contain:

• role provisioning, migration, and post-migration invariant logs;
• summary.tsv with suite, PASS/FAIL/TIMEOUT, and elapsed time;
• one console log per attempted suite;
• for an opted-in controlled run, controlled-agent-run.json with the reviewed plan, effective replay seed, schedule fingerprint, ordered operation/status evidence, and result;
• api.log and fake-resend.log (email bodies and one-time tokens are never logged by the fake provider);
• final /health output; and
• database diagnostics containing only server version, role attributes, migration count, table/RLS counts, and signing-key count.

Triage in this order:

1. If provisioning or migration fails, inspect the migration output and role invariant error before reading suite logs.
2. If startup fails, inspect api.log, then health and Valkey service status.
3. If a suite times out, treat it as a deadlock/resource-lifetime defect. Do not merely increase the file limit without a measured, reviewed reason.
4. For an assertion failure, reproduce that exact file in a fresh disposable stack. Do not rerun a later file against the failed job's database.
5. If the failure is flaky, retain at least three fresh-stack run artifacts and identify the nondeterministic state, port, clock, or race before changing the assertion. A green rerun alone does not close the defect.

Controlled agent-driven runs

Mechanical CI proves a fixed, reviewable regression contract. A controlled agent-driven run is additive and is appropriate before a release, after a security-sensitive migration, or while investigating an incident. It may:

• generate new adversarial tenant/RBAC/race sequences within a disposable stack;
• repeat selected races under varied concurrency and ordering;
• inspect logs and database state to propose a minimal new deterministic regression; and
• exercise a candidate migration from a sanitized production-shaped snapshot when that snapshot has been separately approved.

An agent-driven result is evidence, not a replacement gate. A newly discovered critical journey becomes mechanical only after its fixture, expected outcome, timeout, cleanup, and provider isolation are encoded in a reviewed test file and the explicit suite list is updated. Agents must not weaken RLS, grant runtime superuser access, reuse production data, or call external providers to obtain a green result.

Executable controlled scenario

The executable suite is intentionally less powerful than an unrestricted test agent. An operator or agent may choose a replay seed, but it cannot supply a command, URL, request body, or arbitrary plan. The only accepted selector is baseline-security (or its exact repository-relative filename), and the plan must be a regular non-symlink file with no unknown keys. Adding an operation or another plan requires a reviewed code change to the allowlist and contract tests.

The baseline runs three seeded rounds at concurrency seven. Its ten operations cover cross-tenant form read/patch/delete, submission read/delete, foreign team switching, a team-scoped API key against a foreign form, viewer form mutation, viewer submission deletion, and a seven-client public-submit idempotency race. After the schedule it verifies both forms and fixture submissions are intact, tenant ownership did not move, the viewer is still a team member/workspace viewer, active team contexts did not change, positive-control owner/viewer/API key reads still work, and every idempotency key resolves to one canonical row with the expected tenant, form, and payload.

The seed deterministically fixes the round ordering and batching. The report's scheduleFingerprint is a compact replay check; use the report's exact plan.seed as API_E2E_AGENT_SEED to reproduce one schedule. Response timing and the ordering in which concurrent requests reach PostgreSQL are expected to vary, but the requested schedule and evidence sequence do not.

The controlled runner fails before database access unless all of these are true:

• API_E2E_ALLOW_DISPOSABLE_DATABASE=true and API_E2E_AGENT_RUN_ACK=controlled-disposable-only are explicit;
• NODE_ENV=production exercises production response behavior;
• the API origin and fixture PostgreSQL URL use numeric loopback (127.0.0.1 or ::1) with explicit ports;
• the API URL has no credentials/path/query/fragment, the database URL has no query/fragment override, and its database name has a distinct ci, e2e, or test segment; and
• artifacts remain under the repository's artifacts/ directory through regular directories, never absolute paths, traversal segments, or symlinks.

The suite uses the already-running API and privileged fixture connection owned by the mechanical gate. It does not provision a server by itself. The package script is therefore additive:

API_E2E_AGENT_SEED=incident-123-replay-1 \
API_E2E_AGENT_SCENARIO=baseline-security \
bun run test:api:e2e:controlled

All other disposable-stack variables must match the parent gate. The normal and recommended entry point is Actions -> API E2E reliability gate -> Run workflow, with controlled_agent_run enabled and a URL-safe agent_seed. That dispatch creates three independent PostgreSQL/Valkey/API stacks. It runs the complete mechanical gate first on each stack, then derives the three effective seeds <agent_seed>-1, <agent_seed>-2, and <agent_seed>-3 for the additive adversarial run. Pull requests and pushes run one mechanical stack and do not opt into the controlled scenario.

The JSON artifact is written atomically with owner-only permissions and excludes tenant/user/form/submission IDs, request bodies, emails, session cookies, API keys, claim tokens, and database URLs. The suite log can contain assertion and server error text and must retain the same access policy as the other CI diagnostics.

The wrapper removes any old regular report before starting Bun and rejects a symlink or non-regular report target. A zero test-process exit is not sufficient for PASS: the current invocation must create a mode-600 JSON report whose version/result/completion timestamps, allowlisted invocation plan and effective seed, all 30 evidence entries, seeded ordering, operation-specific statuses, and recomputed schedule fingerprint pass tests/api-e2e/ci/verify-controlled-report.ts. Only then does the wrapper append PASS to summary.tsv.

For security-sensitive changes, controlled acceptance requires all three fresh workflow-dispatch stacks to pass the affected mechanical files and adversarial scenario; any failed stack resets the evidence set. For ordinary functional changes, one full fresh-stack mechanical pass is required. A locally green controlled run proves the harness works but does not by itself satisfy the three-stack release threshold.

Intentionally manual or provider-backed checks

The following remain outside PR CI because a local double cannot establish the real provider, browser, operational, or production property:

• real Resend delivery, suppression/bounce behavior, and inbox rendering;
• real Google OAuth consent, account policy, and callback configuration;
• real Stripe test-mode Checkout/Portal pages, webhook delivery/retry order, tax behavior, and dashboard reconciliation;
• real R2 credentials, bucket CORS, lifecycle rules, public-domain caching, and object deletion;
• real OpenRouter model availability, latency, billing, and output quality;
• browser accessibility/visual checks, responsive layout, and end-user copy;
• load/stress baselines, long-duration soak, backup/restore, rollback, and disaster recovery; and
• production migration, custodian repair, deploy activation, DNS/TLS/Caddy, service supervision, and post-deploy smoke checks.

Those checks require their own approved staging/provider credentials or production change authorization. Their absence from PR CI must be visible in a release decision; a green mechanical gate does not imply provider or production readiness.

Capacity mechanisms are governed separately by the measurable production gates in ../operations/scaling-decision-policy.md. Load evidence complements this correctness gate and never relaxes it.