How to build an AI document chatbot with AI

These are the parts a AI document chatbot needs before it is useful. Each one is something the plan covers.

• Conversations and messages: Persist chat history per user, with titles, branching-free linear threads and safe deletion.
• Model gateway client and metering: One choke point for every model and embedding call so cost is observable and capped.
• Token and USD budgets (fail closed): Guarantee no user, org or runaway loop can generate an unbounded model bill.
• Document ingestion pipeline: Turn uploaded files and pasted text into searchable, embedded chunks reliably and cheaply.
• pgvector store with tenant isolation: Fast approximate nearest-neighbour search that can never cross tenants.
• Hybrid retrieval and reranking: Retrieve the right passages for keyword-heavy and meaning-heavy questions.
• Streaming responses (SSE): Show tokens as they arrive, stop cleanly and bill honestly.
• Prompt-injection defences and PII handling: Keep hostile documents and users from hijacking the model or leaking data.
• Citations and grounding: Let users verify answers and trust them.

• Response and embedding caches: Avoid paying twice for identical work.
• Feedback, eval and quality loop: Measure answer quality and improve retrieval and prompts safely.

Each table below becomes part of the Postgres database that belongs to your app.

• conversations: title generated from the first question by a capped cheap-model call or by truncation; status: active|archived|deleted; delete cascades and writes a deletion record for the audit trail.
• messages: role: user|assistant|system_notice; content may contain redacted placeholders; finish_reason: stop|length|aborted|refused_budget|error; assistant rows are written once at stream end.
• collections: visibility: private|org; embed_model/embed_dim are fixed at creation, changing the model means a new collection and re-embedding; version increments on any document change and invalidates response_cache.
• documents: status: uploaded|parsing|chunking|embedding|ready|failed|deleted; caps on bytes, pages and chunks per document checked before work is queued.
• chunks: RLS plus mandatory collection_id filter; embedding dimension fixed per collection (EMBED_DIM); tsv built from content with the collection's language config; the HNSW index is built concurrently and ef_search tuned per session.
• ingest_jobs: attempt cap 3 with exponential backoff, dead-letter status=failed with error surfaced to the user; embedding stage resumes from the last embedded ordinal.
• usage_events: kind: chat|embed|rerank|eval|title; outcome: ok|refused_budget|error|aborted; insert-only, written by the gateway wrapper.
• usage_daily: dashboards read only this table; rebuilt from usage_events by cron.
• budget_counters: reserve: UPDATE ... WHERE used + reserved + est <= limit; settle: move est from reserved to used at actual cost; a counter row per user and per org makes both checks one statement each; hot org rows can be sharded by shard_id later.

• No direct provider SDK calls from app code; every call goes through the metering gateway.
• No autonomous agent loops or side-effecting tools in phase 1 (max_steps = 1, read-only).
• No fine-tuning or training on customer data.
• No unbounded retries, fan-out or background generation.
• No claim of HIPAA/SOC 2 compliance or of guaranteed-correct answers.
• Cross-encoder reranking and query rewriting: plan for it, build it later.
• Source connectors and tool use: plan for it, build it later.

Forge64 matches your description to one of 12 reference architectures, writes the plan and data model into the project, generates the app, and runs a quality gate over the code before you see it. Each app gets its own Postgres database and sign-in with no per-user fee. Hosting on Forge Cloud and Forge Payments are rolling out.

This is a reference design for the category, not a promise that every item is generated today. Built-ins such as email, background jobs, file uploads and search are coming soon, so anything here that needs them is something you add yourself for now.

What does an AI document chatbot need first?

Conversations and messages, model gateway client and metering, token and usd budgets (fail closed), document ingestion pipeline, pgvector store with tenant isolation and hybrid retrieval and reranking. The plan for a AI document chatbot covers all of these before it generates code.

What should I leave out of the first version of an AI document chatbot?

No direct provider SDK calls from app code; every call goes through the metering gateway. No autonomous agent loops or side-effecting tools in phase 1 (max_steps = 1, read-only). No fine-tuning or training on customer data.

Which products define the AI document chatbot category?

Well-known examples include ChatGPT, Perplexity, NotebookLM, Intercom Fin and Glean. Use them to decide which features your users expect, then describe your own version.

• AI document chatbot architecture
• Example prompt for an AI document chatbot
• AI document chatbot security checklist
• AI document chatbot scaling checklist
• Often combined: SaaS dashboard
• Often combined: CRM
• All app types