AI in therapy settings: what may be supported and what must stay human

'Using AI in therapy' does not describe one task. Organizing appointment text, finding a particular document among authorized records, and preparing material for professional review are different from interpreting a person's experience, developing a formulation, or making a clinical decision. The first step in evaluating a tool is to name exactly which task is being opened to automation.

WHO guidance on AI for health emphasizes that humans should remain in control of health decisions and that clinical experience and knowledge of the person remain essential. WHO's guidance on large multi-modal models notes that these systems may assist with administrative documentation while also producing inaccurate, incomplete, or false responses. The task boundary therefore matters before convenience or novelty.

This comparison is an evaluation frame, not a feature promise. Saying that a task may be supportable does not mean every product is safe or suitable for that task.

• May support — administrative organization: presenting appointment or task information in a form an authorized user can verify.
• May support — documentation workflow: locating existing records, showing sources, and preparing material for professional review.
• May support — organization and retrieval: returning relevant records with time and provenance inside a clinician-defined scope.
• Must stay human — clinical interpretation and formulation: understanding a person's meaning, patterns, and clinical context.
• Must stay human — diagnosis and treatment selection: diagnosing, choosing a therapeutic approach, or making medication and prescribing decisions.
• Must stay human — risk and emergency assessment: scoring risk, determining a crisis, or classifying an emergency.
• Must stay human — the therapeutic relationship: working with trust, timing, silence, culture, and the human meaning of the encounter.

A generative system can produce an answer that looks coherent, confident, and plausible. That appearance does not establish that the answer is factual, complete, or appropriate to the context. WHO's large-model guidance discusses plausible responses that may displace human knowledge, while NIST addresses confidently stated errors and the risk of excessive deference to automated systems.

In a mental-health context, an omitted detail is more than a technical accuracy problem. Information may be time-bound, come from a different source, represent an opinion rather than a fact, or depend on something that was not said. Collapsing those distinctions into a single fluent sentence can remove precisely the context a professional needs. AI output should therefore not be presented as the authority on clinical meaning.

This scenario is not a client case. A small clinical team evaluates a tool that may support documentation workflows. The team first limits the purpose to helping an authorized user locate a relevant document and see its source link. Diagnosis, formulation, treatment selection, and risk assessment are written down as excluded uses.

Using synthetic, non-identifying test records, the team looks for cases where the tool omits a source, combines context incorrectly, or speaks with unwarranted confidence. They check whether output can be rejected, use can be suspended, problems can be reported, and users understand that the tool is not a decision-maker.

The result is not a simple pass based on whether the tool produced text. The team weighs the benefit of the narrow support task against privacy, error, over-reliance, training, and workflow costs. Suitability belongs to the specific product, task, users, and service context being assessed.

Client, patient, or sensitive clinical information should not be pasted into arbitrary public AI tools when retention, access, model training, subprocessors, deletion, and incident boundaries are unclear. A chat box being easy to reach does not make it an appropriate clinical-data workspace.

An organization also needs to understand where input is processed, who may see output, and whether the service uses data for another purpose. Legal and professional obligations depend on the real context of use. An 'AI-enabled' label does not remove them.

A professional or clinical team can seek concrete answers to the following questions:

1. What exact task is supported, and which clinical tasks are explicitly out of scope?
2. How are users told that output may be inaccurate, incomplete, or inappropriate to context?
3. Are sources and uncertainty visible, and can the user reject output without friction?
4. What data enters the system, where is it retained, who can access it, and is it used for another purpose?
5. How was the tool tested with synthetic data and tasks that resemble the intended operating conditions?
6. Are there training, feedback, incident-reporting, and stop-use processes for automation-bias concerns?
7. Does the tool return time to the therapeutic relationship, or move professional attention into a new verification burden?

Therapeutic work is not made only of recorded sentences. The history of the relationship, nuance of language, silence, timing, cultural context, and a professional's ethical responsibility cannot be reduced to model output. These are different from the organizational support a tool may provide.

Eunora's direction is not client-facing AI therapy. The product aims to organize records and sources under professional control. Interpretation, clinical responsibility, and the relationship remain with the clinician.