Business Context: The Opportunity and the Responsibility
AI-assisted mental health tools occupy a specific and important position in the care landscape. They are not a replacement for clinical therapy. They are not appropriate for people in acute mental health crisis. They are, however, valuable for a significant population: people experiencing mild to moderate anxiety or low mood who are on a waiting list, people who have completed a course of therapy and want to maintain the skills they developed, and people who want accessible, stigma-free support for everyday stress and wellbeing. Getting the product positioning right from the start is as important as getting the technology right. The MHRA regulates software as a medical device (SaMD) if it is intended to diagnose, treat, or mitigate a medical condition. A mental health support tool that claims to treat depression is almost certainly a medical device. A tool that provides CBT-based exercises, mindfulness practices, and psychoeducational content to support general wellbeing, with appropriate disclaimers, may sit outside the medical device boundary depending on its specific claims and functionality. SpeedMVPs works with clients during scoping to establish the correct regulatory position and advises when MHRA pre-submission enquiry is warranted. CQC registration may be required if the service constitutes regulated activity under the Health and Social Care Act 2008.
Architecture: Conversation Safety, Escalation, and Therapist Oversight
The system architecture has four layers, with safety embedded throughout. The conversation layer uses Claude with a carefully designed system prompt that positions the AI as a supportive companion and skills coach rather than a therapist. It can listen, reflect, offer evidence-based techniques (breathing exercises, cognitive reframing prompts, grounding techniques), and provide psychoeducational information. It is explicitly constrained not to provide diagnoses, not to interpret symptoms as specific conditions, and not to provide crisis intervention. The safety monitoring layer runs in parallel with every conversation: scanning messages for crisis signals (expressions of suicidal ideation, self-harm, severe distress) using both keyword matching and Claude's own assessment. When crisis signals are detected, the conversation is immediately directed to the crisis resource layer: prominent display of Samaritans (116 123), Crisis Text Line (text SHOUT to 85258), and local NHS urgent mental health services, with a clear message that the AI cannot provide crisis support. The therapist oversight layer, for products deployed through clinical partnerships, provides therapists with summary views of user conversations (with explicit user consent) for supervision and risk monitoring. The data layer handles all conversation content as special category health data under GDPR Article 9, with explicit consent, encryption at rest and in transit, and strict access controls.
AI Components: Claude as a Safety-Constrained Companion
Anthropic Claude is the model of choice for mental health support applications because its safety training and instruction following are among the strongest in the industry. The system prompt is the most critical engineering asset in this product. It encodes the product's clinical positioning (wellbeing support, not therapy), the conversation style (warm, non-judgmental, reflective), the technique repertoire (CBT-based exercises, mindfulness, behavioural activation, grounding), and the hard limits (no diagnosis, no crisis intervention, always signpost to professional help for clinical concerns). The prompt is developed in collaboration with a clinical psychologist or therapist during the build, not written by engineers alone. Claude is also used for safety classification: each user message is assessed for crisis indicators before the supportive response is generated. This dual-role architecture means Claude is both generating the conversation and monitoring it, which requires careful prompt design to prevent the safety monitoring from being visible to the user (which would undermine the natural conversation feel) while ensuring it runs reliably.
Challenges: Clinical Safety Boundaries, Scope Creep, and User Trust
The primary challenge in AI mental health products is maintaining clinical safety boundaries over time. Users of mental health support tools are vulnerable and may push the AI beyond its intended scope: asking for diagnosis, asking it to interpret their medications, or disclosing crisis-level distress. The system must handle all of these scenarios safely, consistently, and without making the user feel dismissed. SpeedMVPs tests the system against a comprehensive set of adversarial conversation scenarios before launch, including all the edge cases that clinical advisers identify as most likely to occur. Crisis escalation must be reliable without being hair-trigger. A system that interrupts every conversation mentioning sadness with a crisis line number is unhelpful and will damage user trust. A system that misses genuine crisis signals is dangerous. Calibrating this threshold requires clinical expertise and ongoing monitoring after launch. GDPR handling of mental health conversation data is the most sensitive data processing in any consumer health application. The product includes explicit consent (what is being recorded, how it is used, who can access it), a clear right to deletion (conversation history deleted on request), and a data handling policy that meets ICO expectations for special category health data.
Outcomes: Accessibility, Engagement, and Safety Record
The primary success metric for AI-assisted mental health tools is safe accessibility: providing support to people who would not otherwise seek it, without causing harm. Secondary metrics include user-reported wellbeing improvement (measured via validated scales such as PHQ-2, GAD-2, or WEMWBS at onboarding and follow-up), session engagement and return rate, and crisis escalation rate (the percentage of sessions where crisis resources are signposted). The safety record is as important as the engagement metrics. A mental health product with strong engagement but undetected crisis scenarios is not a success. SpeedMVPs builds in monthly clinical safety review as a recommended operational practice: a therapist or clinical lead reviewing a sample of conversations for safety signal quality, appropriate boundary maintenance, and technique quality.
Lessons: Clinical Partnership Is Not Optional
AI mental health products built without clinical oversight from the start consistently encounter the same problems: the system prompt drifts into unsafe territory, crisis signals are missed, and the product positions itself incorrectly in the regulatory landscape. A clinical psychologist or therapist involved in the product design from day one prevents all of these problems. They write the conversation guidelines, review the system prompt, test the crisis scenarios, and provide the clinical credibility that allows the product to be taken seriously by the NHS, commissioners, and potential clinical partners. The second lesson is to be explicit with users about what the product is and is not. Users who understand that the AI is a wellbeing companion, not a therapist, have appropriate expectations and are more likely to have positive experiences. Users who arrive expecting therapy and receive something different feel misled. Transparency about the product's nature and limits is both an ethical requirement and a practical driver of user satisfaction.