AI document chatbot architecture: the decisions that matter

Situation: Model calls bill per token with no ceiling; one abusive user, prompt loop or bug can produce a large bill before any dashboard alert is read.

Choice: Every call atomically reserves estimated tokens and USD against both the user and org counters; no reservation means no call; settlement records actual usage and a reaper releases abandoned reservations; defaults apply when no limit is configured.

• Rejected: Post-hoc metering with alerts: cannot stop spend, only report it.
• Rejected: Gateway-level limits only: protects the account but not per-user fairness inside an app.
• Rejected: Pre-paid credits only: safe but blocks plans that include a monthly allowance.
• Consequence: Users sometimes see a 'budget reached' message instead of an answer, which is the intended behavior.
• Consequence: Estimates are conservative so a small portion of budget is briefly held.
• Consequence: Counter rows are hot, so they are single-statement updates and can be sharded later.

Situation: Pure vector search misses exact terms (SKUs, names, error codes); pure keyword search misses paraphrases; weak retrieval makes the model guess.

Choice: Fuse vector top-40 and full-text top-40 with RRF in one query, send a bounded top-N within a context budget, and refuse to answer from thin evidence when the fused score is below the threshold.

• Rejected: Vector-only: simpler but poor on exact-match queries.
• Rejected: Keyword-only: cheap but poor on meaning.
• Rejected: Always answer from model knowledge: fluent but ungrounded.
• Consequence: Quality is measured with recall@k and groundedness on a golden set.
• Consequence: Some questions get 'not found in your documents', which is shown honestly.
• Consequence: A reranker can be added later behind the same interface.

Situation: Documents are large and untrusted, parsing and embedding are slow and billable, and users re-upload the same files.

Choice: Presigned upload, then staged jobs (parse, chunk, embed) with SKIP LOCKED, sandboxed parsing, content-hash dedupe at document and chunk level, batch embedding under an org budget and resumable stages.

• Rejected: Inline processing on upload: times out and exposes the web tier.
• Rejected: Re-embed everything on change: simple but wastes embedding spend.
• Rejected: Third-party ingestion pipeline: extra vendor and uncapped usage fees.
• Consequence: Documents show clear per-stage status and errors.
• Consequence: Re-uploads and edits cost only the changed chunks.
• Consequence: Embedding model changes require a new collection and a budgeted reindex.

Situation: Documents and web content can contain instructions that hijack the model; combined with tools or markdown rendering this becomes data exfiltration or unauthorised actions.

Choice: Delimit and label untrusted context, strip exfiltration vectors from output, give the assistant no side-effecting tools and a single step in phase 1, and add canary detection and ingest-time heuristics.

• Rejected: Rely on the system prompt alone: easily overridden.
• Rejected: Full agent autonomy with tools: powerful but the largest cost and safety risk.
• Rejected: Block all external content: removes the product's value.
• Consequence: The assistant can answer but not act, which is a stated non-goal until confirmations and budgets exist.
• Consequence: Output sanitisation may remove legitimate links outside citations.
• Consequence: Safety behavior is regression-tested with an injection test set.

Indexes are chosen from the questions the app asks most often. Each line below names a table and the question its main index answers.

• conversations: (owner_id, updated_at DESC, id) — conversation list keyset page.
• messages: (conversation_id, created_at, id) — load a thread in order and build the history window.
• collections: (org_id, name) — list knowledge bases for an org.
• documents: (collection_id, created_at DESC) — document list keyset page.
• chunks: (embedding) — HNSW vector_cosine_ops (m=16, ef_construction=64) for approximate nearest neighbour.
• ingest_jobs: (status, run_at) where status = 'queued' — worker claim with SKIP LOCKED.
• usage_events: unique (request_id, kind) — metering is idempotent per call, retries never double count.
• usage_daily: unique (day, org_id, user_id) — rollup upsert is idempotent.

Forge64 matches your description to one of 12 reference architectures, writes the plan and data model into the project, generates the app, and runs a quality gate over the code before you see it. Each app gets its own Postgres database and sign-in with no per-user fee. Hosting on Forge Cloud and Forge Payments are rolling out.

This is a reference design for the category, not a promise that every item is generated today. Built-ins such as email, background jobs, file uploads and search are coming soon, so anything here that needs them is something you add yourself for now.

What is the most important architecture decision for an AI document chatbot?

Reserve-then-settle budgets that fail closed before the provider call. Every call atomically reserves estimated tokens and USD against both the user and org counters; no reservation means no call; settlement records actual usage and a reaper releases abandoned reservations; defaults apply when no limit is configured.

Do I have to make these decisions myself?

No. Forge AI plans them from your description and writes the decisions into the project as architecture records you can read and change.

What do the alternatives cost me?

For example, for "Hybrid retrieval with reciprocal rank fusion and a grounding gate": Vector-only: simpler but poor on exact-match queries. Keyword-only: cheap but poor on meaning.

• How to build an AI document chatbot
• Example prompt for an AI document chatbot
• AI document chatbot security checklist
• AI document chatbot scaling checklist
• Often combined: SaaS dashboard
• Often combined: CRM
• All app types