Project Overview and Business Context
The client was a 180-person technology business with documentation spread across Notion (product and engineering), Confluence (HR and finance), and Google Drive (sales and marketing). New employees spent a disproportionate amount of their onboarding time searching for information that existed somewhere but was not findable quickly. Experienced employees frequently answered the same questions repeatedly in Slack, despite the answers being documented. The brief was to build a Slack bot that employees could ask questions and receive cited answers from the internal documentation corpus, without having to know which tool the answer lived in or which department owned the relevant page. The agent needed to respect access controls: all employees could access general company documentation, but department-specific documents (HR policies, finance procedures, engineering architecture decisions) should only be visible to members of the relevant department. UK GDPR obligations apply to any personal data in company documents. HR documents containing personal data about employees require additional access controls and are handled separately from the general knowledge corpus. The access control design separates documents into tiers: public (all employees), department-restricted (Slack workspace role-based), and HR-restricted (HR team only).
Technical Architecture and Stack Decisions
The system has two main parts: the ingestion and indexing pipeline and the Slack bot query interface. The ingestion pipeline connects to Notion, Confluence, and Google Drive via their respective APIs, pulling document content and metadata (title, owner, department tag, last updated date) on a nightly schedule. Documents are chunked semantically, embedded using Anthropic Claude's embedding API, and stored in Weaviate with access tier metadata. The ingestion pipeline is incremental: only new or updated documents since the last run are re-processed, keeping nightly processing time under 10 minutes for the client's 4,000-document corpus. The Slack bot is a Slack App that listens for @mentions or direct messages. When a question is received, the bot retrieves the sender's Slack profile to determine their department membership and constructs an access filter for the Weaviate query. The query retrieves the top 8 document chunks relevant to the question, filtered to the user's access tier. The retrieved chunks are passed to Claude with the user's question and a system prompt instructing it to answer using only the provided document excerpts, to cite every claim with the document title and section, and to explicitly note when the provided documents do not contain a clear answer to the question. The response is formatted as a Slack message with inline citations, each linking to the source document. Supabase stores query logs (anonymised, no response content) for usage analytics and system performance monitoring. The Next.js admin interface allows the knowledge management team to manage document access tiers, review ingestion status, and trigger manual re-ingestion for specific documents.
Key AI and ML Components
Anthropic Claude handles both embedding generation and answer synthesis. Using Claude for embeddings ensures semantic consistency between the indexed document vectors and the query vectors, improving retrieval relevance. The same embedding model processes both document chunks at ingestion and user queries at question time. Claude was selected for the answer synthesis layer because of its strong performance on multi-document reasoning: given 8 retrieved chunks from different documents, Claude reliably synthesises a coherent answer while correctly attributing claims to their source documents. The system prompt includes strict grounding instructions: answer only from the provided excerpts, never extrapolate beyond what the documents state, and flag uncertainty when the evidence is thin or contradictory. This grounding requirement is important for internal knowledge tools, where hallucinated answers to questions about company policy, HR procedures, or technical architecture can cause real operational problems. Weaviate is the vector database, chosen for its access control filtering capability (metadata filters applied at query time) and its hybrid search (combining vector similarity with BM25 keyword search). Hybrid search is important for internal knowledge retrieval because employees often search using specific terms (product names, project codes, acronyms) that are more accurately retrieved by keyword than by semantic vector similarity.
Challenges Solved and How
Document freshness is a persistent challenge in knowledge agents. Company documentation changes constantly. The nightly ingestion schedule handles most updates, but time-sensitive changes (a policy update, a new product spec) may need to be indexed within hours. The admin interface includes an on-demand re-ingestion trigger per document and an import webhook that can be triggered by Notion or Confluence automation when a document is published. Slack response formatting required significant iteration. Claude's default response format is prose paragraphs. Slack users expect concise, scannable responses with clear structure. The response formatting layer post-processes Claude's output: breaking long answers into short paragraphs, converting lists to bullet points, and placing citations as numbered superscripts linked to source documents rather than inline URLs. Department access control testing required systematic boundary verification. The system was tested with users in each department trying to access documents from other departments, confirming that the access tier filtering prevented cross-department document retrieval before production deployment. GDPR for HR documents required the HR corpus to be in a separate Weaviate collection with strict access controls, handled by a separate ingestion pipeline with HR team-only API credentials.
Outcome and Measurable Results
The client measured three outcomes over the first 12 weeks. Slack question volume to experienced employees about documented topics fell by 37%, calculated by comparing the frequency of common question categories in public Slack channels before and after deployment. Employee-reported time spent searching for information in a weekly survey fell from an average of 3.2 hours per week to 1.1 hours per week. New employee onboarding satisfaction scores (measured at the 4-week mark) improved from a mean of 3.6/5 to 4.3/5, with onboarding coordinators attributing part of the improvement to the knowledge agent reducing the time new employees spent on basic information-finding tasks. The knowledge management team reported that the usage analytics showed 23% of queries found no clear answer in the existing corpus, which was used to prioritise documentation creation for the gaps identified.
Lessons for Similar Projects
Citation quality is the trust mechanism. An internal knowledge agent that gives answers without sources is difficult to trust because employees cannot verify the answer or find the primary source. Every answer must link back to the source document. If the agent cannot cite a source, it should say so rather than synthesise an uncited answer. Access controls must be designed before the data model, not after. Retrofitting access tier filtering to an existing vector store is hard. Design the access control schema into the Weaviate metadata structure from the first ingestion run. Invest in the document quality assessment. Poor quality documents, outdated pages, contradictory policies, and duplicate content produce poor agent answers. Run a documentation audit before ingestion, prioritising the high-query-frequency topics for quality review. Source document management is ongoing. The agent is only as useful as the documents it indexes. Build a process and assign ownership for keeping high-priority documents current. An agent that cites outdated information is worse than no agent, because it gives employees false confidence in incorrect answers.