AI document chatbot scaling and launch checklist

• Pagination: Conversation lists, thread loads, document lists and usage drill-downs use keyset pagination on indexed (sort_key, id); thread loads fetch the newest 50 messages first and older ones on demand; no OFFSET.
• Idempotency: Every model call carries a request_id; usage_events, budget_reservations and assistant messages are unique on it, chunks are unique on (document_id, ordinal), embeddings are deduped by content_hash, and webhooks are idempotent, so retries and replays never double charge or duplicate rows.
• Rate limits: Order of operations for every chat request: authenticate -> rate limit (per user and per org, token bucket) -> reserve budget atomically (user then org) -> retrieve -> call gateway -> stream -> settle. Any failed step before the provider call refuses the request; nothing reaches the provider without a reservation.
• Rate limits: Per-user and per-org limits on chat requests per minute, concurrent streams, uploads per hour, documents per org, chunks per org and search requests; limits scale with plan and are returned in RateLimit headers; over-limit gets 429 with Retry-After, never a queue.
• Caching: Cache answers for history-free temperature-0 questions keyed by org, model, prompt version, normalised question and retrieval fingerprint, invalidated by collection version; cache query and chunk embeddings by content hash; static pages and the docs UI are edge cached, answers are never cached at the edge.
• Search: Retrieval is a single query combining HNSW vector top-40 and GIN full-text top-40 fused by reciprocal rank fusion, always filtered by collection_id; hnsw.ef_search and iterative scan are set per session so filters do not starve results; target recall@8 is measured on the golden set before any index parameter change.
• Storage: Context is bounded everywhere: max RETRIEVAL_TOP_N chunks, a total context token budget, a history token window, max_output_tokens, and per-document caps on bytes, pages and chunks; per-org storage and chunk quotas keep the vector index inside memory.
• Realtime: Streaming uses SSE with abort propagation: closing the connection cancels the upstream call and settles actual tokens; one stream per user and a per-org stream cap bound concurrent upstream calls and node memory.
• Data growth: usage_events and messages are partitioned by month with retention; chunks partition by collection hash at large size; caches have TTLs and row caps; raw usage is rolled up before expiry.

• Metric time to first token ms: histogram by model and cache hit; the core QoE number.
• Metric tokens and usd per org hour: from usage_events; drives the spend anomaly alert and the budget dashboard.
• Metric budget refusals: count of refused_budget outcomes per org and user; sustained refusals mean a limit is mis-set or abused.
• Metric retrieval quality: recall@k and groundedness from nightly golden-set runs; empty-retrieval rate; thumbs-down rate.
• Metric ingestion queue depth and age: oldest queued document stage and failure rate per stage.
• Log structured request logs without content: request id, hashed ids, model, token counts, chunk ids, latency; never prompts, answers or document text.
• Trace chat request trace: spans: auth, rate limit, reserve, retrieve, gateway, stream, settle, with the request_id.
• Slo chat availability: 99.5% of chat requests return a first token within 5 s or a clear refusal; error budget reviewed monthly.

• parse_document: runs queue (ingest_jobs stage=parse). Guard: resource-limited sandbox (CPU, memory, wall clock, no network), max MAX_DOC_MB and MAX_DOC_PAGES, OCR off unless the plan enables it and is capped, attempt cap 3.
• chunk_document: runs queue (ingest_jobs stage=chunk). Guard: cap MAX_CHUNKS_PER_DOC and MAX_CHUNKS_PER_ORG, idempotent by (document_id, ordinal), replaces prior chunks in one transaction.
• embed_chunks: runs queue (ingest_jobs stage=embed). Guard: reserves embedding budget before each batch of <= 64, skips chunks found in embedding_cache, resumes from last embedded ordinal, stops and fails the document if the org embedding budget is exhausted.
• settle_reservations: runs cron 1m. Guard: releases budget reservations open longer than 10 minutes, reconciles with usage_events, bounded batch.
• rollup_usage: runs cron 5m. Guard: recomputes usage_daily only for days touched since the last run; upsert by (day, org, user).
• run_eval: runs queue (admin starts a run). Guard: at most EVAL_MAX_CASES cases, own USD cap reserved up front, concurrency 1 per org, no retries on model errors.

• Read replicas: Route retrieval, document lists and usage dashboards to a read replica when primary CPU or retrieval latency crosses the trigger; the ingestion write path and the budget counters stay on the primary.
• Search: Move to a dedicated vector search service or a sharded pgvector layout when a tenant exceeds the stated chunk count or HNSW build and memory costs dominate; retrieval is already behind one interface.
• Fanout: Add a capped rerank step over fused candidates and a per-request fan-out limit (at most one rerank call and one rewrite call per question) when precision on the golden set needs it.

Forge64 matches your description to one of 12 reference architectures, writes the plan and data model into the project, generates the app, and runs a quality gate over the code before you see it. Each app gets its own Postgres database and sign-in with no per-user fee. Hosting on Forge Cloud and Forge Payments are rolling out.

Every metered resource on Forge64 has a hard cap, so a traffic spike stops at your limit instead of becoming a surprise bill. This is a reference design for the category, not a promise that every item is generated today. Built-ins such as email, background jobs, file uploads and search are coming soon, so anything here that needs them is something you add yourself for now.

When should I worry about scale for an AI document chatbot?

Not before a measured trigger fires. For example, for "Cross-encoder reranker and query rewriting": recall@8 < 0.85 on the golden set after tuning.

What performance targets should an AI document chatbot aim for?

Time to first token (cache miss, 8 chunks): p95 < 1.5 s excluding provider queue time; retrieval p95 < 150 ms. Retrieval (hybrid, 1M chunks per collection): p95 < 120 ms; recall@8 >= 0.85 on the golden set. Budget reserve + settle overhead: p99 < 15 ms, 2 statements.

Does Forge64 cap my usage costs?

Yes. Every metered resource has a hard cap, and paid plans add an overage spend cap that you set.

• How to build an AI document chatbot
• AI document chatbot architecture
• Example prompt for an AI document chatbot
• AI document chatbot security checklist
• Often combined: SaaS dashboard
• Often combined: CRM
• All app types