What Playwright Tests Cover in an AI Product
Playwright E2E tests operate at the browser level, simulating real user interactions. For an AI SaaS product, the test suite SpeedMVPs builds covers: the authentication flow (sign up, email verification if applicable, sign in, password reset), the core AI feature workflow (submitting a query or document, waiting for the AI response, verifying the response is displayed), the billing flow (selecting a plan, completing checkout, verifying plan tier is reflected in the application), and the account management flow (updating profile, managing subscription, signing out). These flows are the backbone of the product, and a regression in any of them causes immediate user impact. Playwright tests catch these regressions in CI before they reach real users.
Handling Async AI Responses in Tests
Testing AI features in Playwright requires handling the asynchronous nature of LLM responses. A test that submits a query and immediately checks for a response will fail because the LLM call takes seconds. Playwright's auto-waiting mechanism handles most async scenarios: waitForSelector waits for an element to appear, waitForResponse can wait for a specific API response to complete. SpeedMVPs designs AI feature tests to wait for the response container to appear and contain content rather than checking immediately after submission. For streaming responses, the test waits for the streaming indicator to disappear before asserting the response content. In CI, LLM calls can be mocked using Playwright's route interception to avoid actual API costs and ensure test determinism, with a small number of integration tests that call the real API to verify the integration.
Test Data and Environment Isolation
Playwright tests need a reliable test environment with known data. SpeedMVPs sets up a dedicated test environment (typically the CI database seeded with known test data) and uses Playwright's setup project feature to create a test user and authenticate before the tests run. The authenticated session is saved to a cookie file and reused across tests, so each test does not need to go through the login flow, which significantly reduces test run time. Database state is reset between test runs using a seed script, ensuring tests run against a known baseline. For billing tests, Stripe or Lemon Squeezy test mode is used so no real payments are processed.
Parallelisation and CI Integration
Playwright runs tests in parallel by default, distributing tests across multiple worker processes. For a test suite of 30-50 test cases, Playwright can typically complete in under 3 minutes with appropriate parallelisation. In GitHub Actions, SpeedMVPs uses the Playwright sharding feature to split tests across multiple CI runners, further reducing wall-clock time. The Playwright GitHub Actions integration provides inline test failure reports directly in the pull request check, showing screenshots and videos of failed tests without needing to download the CI artifacts. This makes diagnosing test failures fast for developers who are not familiar with the test infrastructure.
Accessibility Testing with Playwright
Playwright integrates with the axe-core accessibility testing library to check pages for accessibility violations automatically as part of the E2E test suite. SpeedMVPs adds an axe check to the key pages in the test suite (home, login, main product page, settings) so accessibility regressions are caught in CI alongside functional regressions. For AI products where inclusivity is important, such as tools intended for use across a range of user abilities or tools deployed in the NHS context where WCAG 2.1 AA compliance is often required, this automated accessibility testing provides a baseline check. It does not replace manual accessibility review but catches common issues like missing alt text, incorrect ARIA attributes, and keyboard navigation failures.
What SpeedMVPs Delivers
Playwright test suite delivered as part of an AI MVP build includes: Playwright configuration for Chromium, Firefox, and WebKit (or Chromium only for faster CI), test setup with authenticated user state, tests covering the core user journeys (authentication, core AI feature, billing, account management), CI integration in GitHub Actions with sharding for parallel execution, test data seeding scripts, mocked LLM responses for deterministic AI feature tests, and documentation covering how to run tests locally, how to add new tests, and how to debug failures. The test suite is version-controlled alongside the application code and full ownership is transferred on delivery.