Why Zero-Downtime Deployment Matters for AI Products
Traditional web applications can tolerate brief downtime during deployments because most user actions complete in seconds. If a deployment takes a server offline for 30 seconds, users who happen to make a request during that window see an error and can immediately retry successfully. For AI products with long-running LLM inference requests, the situation is different. A user who asks your AI product to analyse a 50-page document may be waiting 45 seconds for the response. If a deployment terminates the server handling their request, they receive an error for an operation that cannot be quickly retried, losing both the wait time and potentially the AI output they needed. Beyond the user experience concern, modern engineering culture treats production downtime during deployments as unnecessary risk. The tools and patterns to eliminate it are mature and accessible. For UK AI product teams whose enterprise customers may have availability SLAs requiring 99.5% or higher uptime, planned downtime for routine deployments is not compatible with those commitments.
Rolling Deployments
A rolling deployment gradually replaces old instances with new ones, running both versions simultaneously during the transition. The load balancer removes one old instance from the pool, the new version is deployed to that instance, health checks confirm it is healthy, and it is added back to the pool to receive traffic. The process repeats for each instance until all are running the new version. Rolling deployments provide zero downtime because the load balancer always has healthy instances in the pool. The trade-off is that two versions of your application run simultaneously during the rollout, which requires the new version to be backwards-compatible with the old schema, the old API contracts, and any shared state. For AI products where you are changing prompt templates or model configurations, both versions may serve users simultaneously during the transition, which can produce inconsistent experiences if not managed carefully. Rolling deployments are simpler than blue-green because they do not require a second complete environment, but they offer less control over the transition and slower rollback compared to blue-green.
Canary Deployments
A canary deployment routes a small percentage of production traffic, typically 1-10%, to the new version while the majority continues on the old version. Metrics, error rates, and user behaviour are monitored on the canary population. If the new version performs well on the canary slice, the percentage is gradually increased: 1% to 5% to 25% to 100%. If problems appear on the canary, it is rolled back, having affected only a small fraction of users. Canary deployments are particularly well-suited to AI products because they allow quality comparison between versions in production. If you are changing a prompt template, rolling out a new LLM model, or changing generation parameters, a canary lets you measure the impact on a small real user population before committing to the full rollout. Monitor AI-specific quality signals on the canary: user acceptance rate, task completion, regeneration rate. If the canary version shows quality degradation, stop the rollout before it reaches the majority of users. The tooling for canary deployments on AWS uses ALB weighted target groups. On Kubernetes, it uses Deployment replicas and service weights. On Vercel, manual traffic splitting or an edge middleware approach can approximate canary behaviour.
Connection Draining for Long LLM Requests
The specific challenge of zero-downtime deployment for AI products is in-flight requests. A user's LLM inference request that started two seconds before deployment should complete successfully. Connection draining, also called deregistration delay, solves this. When an instance is taken out of service during a rolling or blue-green deployment, the load balancer is told to stop sending new requests to that instance but to allow existing in-flight requests to complete before fully deregistering it. The draining duration must be set longer than your maximum expected request duration. For an AI product where LLM inference can take up to 60 seconds, set connection draining to at least 90 seconds. During those 90 seconds, the old instance continues serving its in-progress requests while the load balancer directs all new requests to the new instances. When draining completes or the timeout expires, the old instance is deregistered and shut down. On AWS ALB, the deregistration delay setting on the target group controls this. On Kubernetes, the terminationGracePeriodSeconds setting on the Pod spec and the preStop lifecycle hook together achieve the same result, allowing in-flight requests to complete before the container is killed.
Schema Migration and Backwards Compatibility
Zero-downtime deployment is straightforward for code changes that do not touch the database schema. When schema changes are involved, backwards compatibility requirements add complexity. During a rolling or blue-green deployment transition, two versions of your application may run simultaneously and both query the same database. If the new version adds a required column, the old version fails when it reads rows created by the new version. If the new version renames a column, the old version cannot find it. The expand-contract pattern solves this. The expand phase adds the new column or table without removing the old structure, and updates the new version of the application to write to both old and new locations. After the deployment stabilises, the contract phase runs a follow-up migration to remove the old column or table once no instances are reading from it. This means schema changes require at least two deployment cycles: the expand migration and new code in the first cycle, the contract cleanup in the second. For AI products, prompt template changes stored in the database, model configuration tables, and conversation history schemas all require this careful migration approach when modified.
Zero-Downtime Deployment on Vercel and Managed Platforms
Modern managed platforms handle zero-downtime deployment automatically, eliminating the configuration burden for teams that build on them. Vercel deploys each new build to an immutable deployment URL. Production traffic is only switched to the new deployment when it is healthy, and the switch is atomic: there is no period where some users see the new version and others see the old version within the same deployment event. Rollback is instantaneous: promote the previous deployment back to production. In-flight requests to the previous production deployment are not interrupted because Vercel keeps it running briefly during the transition. GCP Cloud Run performs zero-downtime rolling updates by default, launching new revision instances, routing traffic to them after health checks pass, and scaling down old revision instances with connection draining. AWS ECS with CodeDeploy blue-green deployment type automates the blue-green switch with configurable traffic shifting. For teams on these managed platforms, zero-downtime deployment is the default behaviour with no additional engineering. The configuration work described in this guide applies primarily to teams managing their own infrastructure on raw cloud compute.