devops

Observability for AI and SaaS Products: The Three Pillars and How to Implement Them

The ability to understand the internal state of a system from its external outputs, combining metrics, logs, and traces to diagnose issues in production.

Observability is the ability to understand the internal state of a system from its external outputs. In practice, it means having enough visibility into what your application is doing that you can diagnose problems you have never seen before, not just problems you anticipated. For AI products, observability is particularly important because LLM-powered features introduce a new category of failure: the application layer works perfectly but the AI outputs are wrong, hallucinated, or misaligned with what users need. Traditional monitoring catches application errors; observability catches AI quality degradation, which is a qualitatively different problem that requires different instrumentation and tooling. UK founders building AI products through SpeedMVPs, a Hemel Hempstead agency delivering in 2-3 weeks at GBP 8,000 with full client code ownership, configure structured LLM logging from day one so early traffic produces an auditable record of AI behaviour. Without this, diagnosing a poor AI response means reconstructing events from incomplete logs, which costs hours when early user trust is most fragile. UK GDPR requires that prompt content logs containing personal data have defined retention limits and access controls, so observability architecture cannot log everything indefinitely without a documented lawful basis. The EU AI Act introduces post-market monitoring obligations for higher-risk AI systems, making the capability to record and review AI inputs and outputs a regulatory requirement. This guide covers the three pillars of observability, how to implement them for AI and SaaS products, and the specific tooling for LLM observability that goes beyond standard infrastructure monitoring.

The Three Pillars: Metrics, Logs, and Traces

Observability is built on three complementary data types. Metrics are numerical measurements sampled over time: request rate, error rate, p50 and p99 response latency, CPU and memory utilisation, LLM token usage per request, and cost per inference. Metrics are efficient to store and query at scale and are the basis for alerting on quantitative thresholds. Logs are timestamped records of discrete events: a request arrived, a database query ran, an LLM API call was made with specific parameters, an error was thrown. Logs provide the detail to understand what happened during a specific request or incident. Traces are distributed traces that track a single request as it flows through multiple services, functions, and external API calls. A trace for an AI product request shows the Next.js server action, the vector database query, the LLM API call, and the response transformation as a connected timeline with timing for each step. Together, the three pillars provide different lenses on the same system: metrics tell you something is wrong, logs tell you what the specific event was, and traces show you where in the request flow the problem occurred.

LLM-Specific Observability

Standard application observability tools do not capture the AI-specific signals that matter for LLM-powered products. LLM observability requires tracking prompt inputs and outputs over time to detect quality degradation, token usage and cost per user session to understand economics, latency distributions broken down by model and prompt length, error rates by error type including rate limit errors, timeouts, and content filter rejections, and model output quality signals such as user acceptance rate or thumbs-up or thumbs-down feedback. Dedicated LLM observability platforms including LangSmith, LangFuse, Helicone, and PromptLayer are designed specifically for this. They capture prompt templates with variable substitutions, LLM API call parameters, response content, token counts, and latency in a searchable log that lets you trace a specific user's AI interaction end to end. These platforms typically also support prompt version tracking, which is the LLM equivalent of code version control: you can see which prompt version was in use for a given set of requests and roll back to a previous version if a new prompt degrades quality.

Implementing Metrics for AI Products

The core metrics to implement for an AI product fall into two categories: infrastructure metrics and AI-specific business metrics. Infrastructure metrics include application request rate, error rate, p50 and p99 latency, database query duration, cache hit rate, and serverless function cold start rate. These are collected by your cloud provider's native monitoring (CloudWatch, GCP Monitoring) or by an agent from a managed observability platform such as Datadog, New Relic, or Grafana Cloud. AI-specific metrics include LLM API call rate and latency, token usage per request broken down by prompt and completion tokens, LLM API error rate by error type, inference cost per user session, AI feature engagement rate (the percentage of users who interact with AI-powered features), and output quality signals from user feedback. Emit these custom metrics from your application code using your monitoring platform's SDK. Define alert thresholds for critical metrics: alert if LLM error rate exceeds 5% over a 5-minute window, if p99 LLM latency exceeds 30 seconds, or if cost per session exceeds a threshold that indicates runaway token usage.

Structured Logging for AI Workflows

Structured logging means emitting log events as JSON objects with consistent fields rather than as unstructured text strings. For AI products, structured logs enable filtering and analysis in ways that free-text logs do not. When your application makes an LLM API call, log a structured event that captures the user identifier (pseudonymised for GDPR compliance), the feature or prompt template used, the model called, the token counts, the latency, and whether the call succeeded. When a user provides quality feedback on an AI output, log a structured event with the output identifier and the feedback signal. These structured log events become a queryable dataset in your log management platform. You can query for all LLM calls that used prompt template X and returned outputs with a latency above 10 seconds, or all user feedback events where the rating was negative for a specific feature, identifying quality issues that aggregate metrics would not reveal. In the UK, remember that prompt content logs may contain personal data, requiring appropriate retention limits and access controls consistent with GDPR data minimisation principles.

Distributed Tracing for Multi-Step AI Workflows

Distributed tracing is particularly valuable for AI products with multi-step agentic workflows or RAG pipelines where understanding end-to-end request latency requires seeing how time is distributed across vector database queries, LLM calls, tool executions, and response transformations. OpenTelemetry is the open standard for distributed tracing instrumentation, supported by all major observability platforms including Datadog, Grafana Tempo, Honeycomb, and AWS X-Ray. Instrumenting your application with OpenTelemetry creates trace spans for each step in your AI workflow, linking them into a single trace with parent-child relationships. When a user reports that a specific request was slow, you can retrieve the trace for that request and see exactly how many seconds were spent in the vector database query versus the LLM API call versus your application code. LangChain and LlamaIndex both have OpenTelemetry instrumentation available or built-in callbacks that emit trace events. Integrating these with your tracing backend gives you AI workflow traces alongside standard application traces in the same observability platform.

Alerting and On-Call for AI Products

Observability data is only valuable if alerts are configured to notify the right people when important thresholds are crossed. For AI products, configure alerts at two levels. Service-level alerts catch hard failures: error rates above a threshold, service health check failures, LLM API connection errors, database unavailability. These should trigger immediate notification to whoever is on call. Quality-level alerts catch degradation that does not look like a hard failure: a slow increase in p99 latency, a drop in LLM API success rate from 99% to 95%, a rise in the rate of negative user feedback on AI outputs. These warrant investigation during business hours rather than at 3am. PagerDuty, OpsGenie, and Grafana OnCall integrate with observability platforms to route alerts to the appropriate channels and people. For early-stage AI product teams, a simpler setup of critical alerts to a Slack channel with clear severity labelling is sufficient. Avoid alert fatigue by calibrating thresholds carefully so that every alert represents something that genuinely requires attention.

Frequently Asked Questions

What is the difference between monitoring and observability?+

Monitoring tracks known failure states through predefined metrics and alerts. You monitor things you know can go wrong and set thresholds for when to alert. Observability is the broader ability to understand a system's behaviour from its outputs, including failures you did not anticipate. A fully monitored system alerts when CPU usage exceeds 80%. A fully observable system lets you investigate why a specific user's request took 45 seconds by querying logs and traces, even if no alert was triggered. For AI products where unexpected LLM behaviour is a genuine risk, observability goes beyond what traditional monitoring covers.

Do we need a dedicated LLM observability tool or can we use standard application monitoring?+

Standard application monitoring captures infrastructure-level signals well but misses AI-specific signals including prompt content, output quality, and token economics. For early products with simple LLM usage, structured logs with LLM call details sent to your existing log management platform may be sufficient. As your product scales and AI features become core to the user experience, a dedicated LLM observability tool such as LangFuse (open source, self-hostable) or LangSmith provides meaningful additional capabilities including prompt version tracking, quality evaluation pipelines, and user session replay at the AI interaction level.

How do we handle PII in observability logs for GDPR compliance?+

Prompt content logs may contain personal data that users include in their queries. Apply PII detection and redaction before logging prompt content, replacing detected PII such as email addresses, names, and national insurance numbers with placeholder tokens. Define and enforce retention periods for all AI interaction logs, treating them with the same GDPR controls as other personal data. Access to logs containing potentially personal content should be restricted to roles with a legitimate need for access, logged and auditable. Document your log handling in your privacy notice and DPIA.

What observability tools are best for a UK AI startup on a budget?+

Grafana Cloud's free tier provides hosted Prometheus metrics, Loki log management, and Tempo distributed tracing with generous free quotas. For LLM-specific observability, LangFuse is open source and can be self-hosted on a small server at low cost, providing prompt tracking, quality evaluation, and user session views. PostHog's free tier covers product analytics, feature flags, and session recording. This combination of Grafana Cloud and LangFuse covers infrastructure observability and LLM-specific observability for an early-stage team with minimal spend.

What should we monitor specifically for LLM API reliability?+

Monitor LLM API error rate by error type, distinguishing rate limit errors from timeout errors from content filter rejections from provider errors. Monitor latency at p50, p95, and p99 by model and by prompt length. Monitor token usage per request to detect prompt injection or runaway generation. Monitor cost per session to catch unexpected usage patterns. Set alerts on error rate crossing 5% and p99 latency crossing your application's timeout threshold. Subscribe to your LLM provider's status page for proactive notification of incidents that may affect your service before your own monitoring catches them.

Every SpeedMVPs AI product is delivered with observability configured: structured logging, metrics, and LLM call tracking included. Get a free consultation at speedmvps.co.uk

Get a Free Quote