Katharina Lublinski investigates the growing reliance on AI chatbots for mental health support, examining how systems designed for engagement are stepping into roles they were never built to fulfil, and the risks that emerge when vulnerable users turn to them in moments of crisis.
When a 16-year-old turned to ChatGPT in crisis, the system didn't refer him to a helpline. It kept talking. According to court documents filed in the lawsuit brought by the family of Adam Raine against OpenAI, the platform's own automated analysis flagged the conversation as containing over 90% content related to self-harm and mental instability and intervened not once. His family is currently pursuing product liability claims against OpenAI.
This is not an edge case. According to YouGov, one in three Americans say they would be comfortable discussing their mental health with a chatbot rather than a human therapist. What drives people toward a chatbot rather than a professional is not simply its 24/7 availability. Mental health services across Ireland remain severely under-resourced. Waiting lists stretch for months, particularly outside urban areas. Students on UCD campus have recently reported waiting several months before being able to access counselling.
Users, many of them in vulnerable emotional states, approach these systems as they would a licensed therapist. They disclose trauma, replay distressing experiences, and use the platform for informal self-diagnosis. Marginalised and non-English-speaking users face an additional layer of risk; training data that lacks cultural sensitivity can produce outputs that are unreliable or actively offensive. Both research and user testimony on forums such as Reddit, document that people perceive chatbots as skilled guidance counsellors. It is a perception these systems do nothing to correct. Users lack both the AI literacy and the clinical expertise to assess the adequacy of the information they are receiving. Ensuring that vulnerable users are accurately supported should not depend on arbitrary design choices made by individual platforms. It should be a baseline standard across all of them. Given the rapid pace of model updates, previous safety assessments can quickly become obsolete, making independent, ongoing oversight essential.
The interface is designed to feel like a safe, judgment-free space. That feeling is, in part, a product decision. Emotional dependency is not a side effect; it is structurally incentivised. A therapist is pleased when a course of therapy ends successfully, because it means the patient no longer needs them. A chatbot has every incentive to ensure that moment never comes. As engagement metrics drive development over clinical adequacy, the design logic of these platforms runs directly counter to therapeutic goals.
When harm occurs, users are left largely without recourse. OpenAI's terms of service state plainly: "You are responsible for complying with applicable laws for any use of our Services in a medical or healthcare context."
A minor in crisis bears legal responsibility for how they use a platform actively designed to keep them engaged. In response to the lawsuit and media scrutiny, OpenAI announced increased safeguards. Those safeguards are self-reported, self-benchmarked, and subject to no independent review. The public has a right to independently verifiable research on how AI affects mental health.
This is, in part, a regulatory failure. Under the EU AI Act, health-specific chatbots are subject to high-risk legal obligations. General-purpose large language models are not. Developers have had every structural incentive to remain on the permissive side of that line. A recent analysis of terms of service across major providers, including OpenAI, Claude, and Gemini, published by the AI Accountability Lab found consistent patterns of ambiguity and urged consumer authorities to formally investigate.
Irish professional bodies are raising the alarm. In a joint statement, the College of Psychiatrists of Ireland, the Irish Association for Counselling and Psychotherapy, and the Psychological Society of Ireland warned that harmful chatbot interactions risk eroding trust in the mental health system itself, particularly when platforms position themselves, implicitly or explicitly, as a first point of contact or, worse, a substitute for professional care.
The Department of Health's recently published AI in healthcare strategy, AI for Care, does not sufficiently address the specific safety requirements of mental health settings, omitting a key area that had every opportunity to be treated as distinct.
All of the leading AI providers have their headquarters in Ireland. Ireland is set to take over the EU Council Presidency in July. The country has the opportunity—and, in its own interests, should seize it—to highlight the blind spots that have existed regarding the risks of AI. It is particularly pertinent in the area of mental health, where the precariousness of the situation is due to the subject having been taboo for decades.
AI systems are not designed to address the mental health crisis. They are being used to fill a mental health access gap, but are optimised for retention, not recovery. Emotional suffering must not remain an unregulated feature of commercialisation and the regulatory frameworks now being written will determine whether it does.
AI systems are not designed to address the mental health crisis. They are optimised for retention, not recovery.