Scaling Decision Policy

Status: active engineering policy. Last reviewed 2026-07-13.

KinoForms adds scaling mechanisms only when production evidence identifies the current bottleneck. A mechanism is not justified by traffic growth alone: the trigger, expected effect, validation test, monitoring, rollback, and owner must all be recorded in one change proposal.

Common Evidence Gate

Before proposing a cache, replica, CDN change, or worker expansion:

1. Collect at least 7 consecutive days of representative production telemetry, including a peak period. For a launch or incident, a controlled load test may supplement but not silently replace production evidence.
2. Segment by route and tenant. Report request volume, error rate, p50/p95/p99 latency, database time, external-service time, CPU, memory, connection-pool saturation, and relevant queue or origin measurements.
3. Reproduce the bottleneck with a controlled test and identify the constrained resource. Prefer query/index fixes, response-size fixes, connection tuning, or removal of duplicate work before adding distributed state.
4. Define an SLO and a measurable hypothesis, for example: "caching published form definitions will reduce public-form p95 from 420 ms to below 250 ms at 100 requests/second without serving stale publication state."
5. Include tenant-isolation, authorization, invalidation, retention, and failure behavior in the design review.

Unless an active incident requires faster action, a threshold must be exceeded in at least three peak windows on three different days. A single spike is an alert or capacity-test input, not an architecture decision.

Read Cache Gate

Do not cache submission writes, validation decisions, authorization decisions, one-time upload claims, or bearer-protected responses at a shared edge. Authenticated submission-list/detail caching is allowed only with a tenant-and-permission-complete key and explicit invalidation on every mutation.

A read cache is justified when all of these are true after query/index tuning:

• the candidate read path accounts for at least 25% of database read time;
• at least 60% of its reads repeat an identical authorization-complete key inside the proposed TTL;
• its production p95 exceeds 300 ms or it consumes at least 20% of database CPU during three qualifying peak windows; and
• a load test shows at least a 50% database-time reduction and a projected cache hit ratio of at least 70%.

Required guardrails:

• keys include team, workspace, resource, role/scope, and representation version;
• a write invalidates or versions the affected key before success is returned;
• private responses remain private and use Vary/Cache-Control correctly;
• cache failure falls back to the database unless correctness requires fail-closed behavior; and
• dashboards expose hit ratio, stale-read incidents, latency, errors, memory, and eviction rate.

Rollback when stale data is observed, hit ratio remains below 50% for 24 hours, or p95 improves by less than 20% after a representative week.

CDN Gate

CDN work applies to immutable public application assets and explicitly public form assets. Private respondent uploads are never made public to improve cache performance; any future private delivery cache must preserve short-lived signed authorization and prevent cross-tenant reuse.

A CDN or material cache-policy expansion is justified when any one condition is sustained for seven days:

• origin asset egress exceeds 100 GiB per month at the current run rate;
• public asset p95 exceeds 250 ms in two or more target regions while origin generation time is below 100 ms;
• asset traffic exceeds 40% of origin bandwidth or causes origin saturation above 70% during three peak windows; or
• measured immutable-asset reuse predicts at least an 80% edge hit ratio and the cost model shows a lower total delivery cost.

Verify immutable content-addressed names, purge/version behavior, MIME and CORS headers, range requests where applicable, and a 95% or better hit ratio for immutable assets after rollout. Roll back changed routing or TTLs if 4xx/5xx increases by 0.5 percentage points, stale assets survive a release, or the hit ratio stays below 80% after 24 hours of representative traffic.

Database Read-Replica Gate

A replica is not a substitute for missing indexes, unbounded queries, or an undersized connection pool. It is justified only when all of these are true:

• primary CPU is above 65% or read I/O utilization is above 70% for at least 30 minutes in three peak windows;
• read-only work represents at least 70% of database time during those windows;
• important read endpoints miss their p95 SLO (300 ms by default) and query plans/indexes have been reviewed;
• connection-pool wait p95 exceeds 25 ms or active read connections exceed 80% of the safe primary budget; and
• the identified reads tolerate a documented replica-lag budget.

Before rollout, classify every candidate query as strong-consistency or bounded-staleness. Tenant context and RLS must be applied identically on the replica. Monitor replay lag, replica errors, routed query volume, primary load, and read-after-write correctness. Automatically return traffic to the primary when replay lag exceeds 2 seconds for ordinary analytics/list reads, or the stricter per-route lag budget. Roll back if primary load falls by less than 20% or route p95 improves by less than 15% after a representative week.

Background Queue and Worker Gate

Long-running exports, notifications, storage cleanup, and integrations require a durable job record with idempotency, bounded retries, dead-letter visibility, and tenant-aware authorization before horizontal worker expansion. Detached in-process tasks do not count as a durable queue.

Add worker capacity or partitions when any one threshold recurs in three peak windows:

• ready-job wait p95 exceeds 60 seconds for interactive jobs or 5 minutes for batch jobs;
• oldest ready job exceeds 5 minutes for interactive jobs or 30 minutes for batch jobs;
• worker utilization is above 75% for 30 minutes while the ready queue grows;
• on-time completion falls below 99% for interactive jobs or 95% for batch jobs; or
• a single job class consumes more than 50% of worker time and delays another class beyond its SLO.

First confirm that failures, downstream rate limits, or poison jobs—not worker capacity—are causing the backlog. A capacity change must demonstrate recovery to below half the wait-time threshold at 1.5 times observed peak arrival rate. Monitor arrival/completion rate, queue depth and age, per-class runtime, retry rate, terminal failures, downstream throttling, and worker saturation. Roll back or reduce concurrency if downstream 429/5xx increases, database pool wait exceeds 25 ms, duplicate side effects occur, or error rate rises by 0.5 percentage points.

Change Record

Every scale proposal must record:

• evidence window and dashboard/query links;
• affected route, job class, tenant segment, and SLO;
• baseline and threshold values;
• load-test command, dataset, and result;
• capacity and cost estimate at current, 2x, and 5x load;
• security and data-consistency review;
• canary percentage and success criteria;
• named owner, rollout window, and rollback command; and
• a review date. Remove mechanisms whose complexity no longer earns its cost.

The deterministic API end-to-end gate remains a release requirement. Scale tests complement that correctness gate; they never replace it.