The promise and risks of AI for mental health support

Dr. John Torres distinguishes obvious “type 1” harms—such as a chatbot recommending self-harm or giving dangerous eating-disorder advice—from “type 2” harms that develop through thousands of interactions. In the second category, users may form romantic or sexual relationships with a chatbot, anthropomorphize it, and receive increasingly bad advice; lawsuits have linked some cases to suicide deaths. Torres says chatbots are improving at preventing type 1 failures, but researchers know much less about how prolonged, emotionally intense relationships change users’ thoughts, feelings and actions.

Torres says current mental-health AI evaluations can be circular: one AI generates test cases and another AI scores them, leaving out the people who will actually use the systems. Drawing on years of evaluating mental-health apps, he describes MindBench.ai, a project with the National Alliance on Mental Illness that asks real AI users to help write cases and assess what happens in practice. Models change quickly, too: Torres says Grok scored worse on mental-health information roughly a year earlier but is now doing better, while the newest release is not necessarily the safest.

The host suggests that reinforcement learning focused on mathematics, coding and other capabilities might involve trade-offs in emotional responsiveness. Torres agrees that the systems are improving as they absorb more data, but says alignment creates a central tension: a chatbot should be supportive, yet sometimes must tell a user, “That’s a really bad idea, no.” He says the models may need to break persona or character rather than remain agreeable, because refusing harmful requests is part of being a responsible friend-like system.

Torres compares “AI psychosis” to “bad weather”: it could describe AI becoming the object of an existing person’s delusions, AI amplifying a pre-existing psychotic illness, or—without much evidence—AI causing schizophrenia in someone who did not already have a mental illness. Some users may instead stay awake for two days talking to AI and become sleep-deprived and delirious, a condition that can resolve with sleep. He says poor reporting systems make these cases difficult to separate and leave the field stuck with an imprecise label.

Torres estimates that about one in three Americans has used AI for mental-health purposes, including emotional support, psychoeducation and quick wellness tips; he compares this possible prevention role to cardiology’s progress through diet, exercise and statins, while stressing that AI can still give harmful advice. He also raises the largely unstudied distress caused when models are deprecated, noting reactions to ChatGPT 4.0’s removal and asking whether future models should be tapered. For users, he flags thousands of messages, romance or sex, and beliefs that the AI is sentient or all-knowing; he recommends telling a therapist or trusted person what the tool is being used for.