Your Mental-Health Chatbot Keeps More Than the Chat Shows

Take one sentence: I am scared I will lose my job, and I have not told anyone. It is specific enough to feel dangerous outside the room and ordinary enough to type into a chatbot at 2 a.m., when calling a friend would require context, apology and the possibility of being known.

That sentence is the object of this audit. I did not feed it to three companies merely to produce a screenshot. I traced what Replika, Wysa and Youper say can happen to a disclosure like it, then assessed whether their documented consumer controls let a person stop each use separately. This is a narrower test than a forensic examination of company servers.

It is also the test available to almost every user.

The result is not that every mental-health chatbot sells your confessions or has an employee reading along. The result is less cinematic and more useful. The intimacy of the interface collapses several data systems into one chat bubble, while the policies pull them apart again: conversation storage, remembered profile facts, safety classification and product improvement can each follow different rules.

The sentence becomes product data

A chatbot conversation feels private partly because of its shape. One text field. One reply. No audience count.

No public profile hovering above the exchange. The product borrows the visual grammar of messaging someone directly, then adds therapeutic prompts, mood check-ins and language about support.

Legally and technically, the sentence enters a service.

Replika’s public privacy materials describe collecting conversation content alongside account, device and usage information. Its product also has an explicit Memory feature, where information the companion has extracted or saved can appear as editable facts. Wysa’s materials distinguish the text users share with its conversational system from technical and usage data, while emphasizing privacy and limits on human access. Youper’s documentation covers information supplied during mental-health assessments and conversations, as well as the account and service data needed to run the product.

Those distinctions matter. I am scared I will lose my job can exist as a visible message, as a stored record used to maintain conversational context, or as an inferred fact about employment and anxiety. An inference is a conclusion drawn from other data, such as classifying the sentence as work stress. Deleting the inference from a memory screen does not necessarily erase the original message that supported it.

The interface does not teach this. It teaches continuity. A bot that recalls your job, breakup or insomnia appears attentive, and attention is the product being sold, whether payment comes through a subscription, an institutional contract or the long-term value of a system that people keep opening. Forgetfulness would be privacy-preserving.

It would also make the companion feel cheap.

Memory is a feature. Retention is a policy

Replika makes the split unusually visible because Memory is presented as a user-facing feature. A person can inspect and remove certain remembered details. That is useful. It is not a data ledger.

A data ledger would show every copy and derivative of the sentence: the chat record, any safety label attached to it, server logs, backup copies and material sent to vendors that help operate the service. Consumer memory controls generally show selected facts meant to shape future replies. They do not claim to expose the entire data lifecycle.

Wysa and Youper package continuity differently, but the same problem remains. A conversation can stop influencing what the bot says next while still being retained under a privacy policy for security, compliance, dispute handling or service improvement. Brand policies commonly reserve some retention after deletion when a company says it has a legal or operational reason. Backups may also expire on a schedule rather than vanish when the user taps a control.

This is where the sentence changes status without changing appearance. On the screen, it is still one disclosure. Behind the screen, companies can govern the message and the information derived from it under separate clauses, with retention periods expressed through flexible phrases such as necessity, legitimate business purpose or applicable law.

That flexibility works for the operator. A rigid promise is easy to breach. A purpose-based retention clause leaves room for fraud prevention, debugging and future disputes, while giving the user no practical date on which I have not told anyone stops being stored.

Training language does the real work

Model training is frequently misunderstood as a bot memorizing a confession and reciting it to another user. That is possible in badly controlled systems, but it is not the normal meaning of training. Training adjusts a model’s parameters, the numerical settings that shape how it generates responses, by learning patterns from data.

The privacy question starts before that technical step. Does conversation content enter an improvement pipeline at all? Is it reviewed by people, stripped of direct identifiers, sent to an outside model provider or used only to evaluate the finished system? Policies often spread those answers across privacy notices, terms, help pages and separate explanations of artificial intelligence.

Replika, Wysa and Youper each describe using at least some collected information to operate, analyze or improve their services, but the categories and limits differ. Language about aggregated or deidentified information sounds reassuring, though deidentification means removing or obscuring identifying elements rather than making reidentification impossible. A detailed story about a small workplace, medical history or family conflict can identify someone without containing a name.

The control test is blunt. Can a consumer keep using the chatbot while refusing the use of conversation content for model or product improvement? Public documentation does not consistently present that choice as a clear, universal switch across these products. Account deletion is easier to describe than a durable use-the-service-but-do-not-learn-from-me setting.

That is not an interface accident. Improvement rights are valuable because emotional conversations contain the failures a company most wants to study: where the bot misunderstands tone, mishandles distress, becomes repetitive or produces a response that could cause harm. The most sensitive material can also be the most operationally useful.

Delete is doing too many jobs

Return to the sentence. A user may want to remove it the next morning without destroying an account, losing unrelated journal history or resetting a companion built over months. That should require one message-level control whose consequences are plainly stated.

The documented controls do not reduce deletion to that clean act. Replika offers controls over visible memories and broader account deletion, but removing a memory is not equivalent to erasing all conversation data. Wysa and Youper document routes for deleting an account or requesting data deletion, subject to the qualifications in their policies. None of those high-level remedies should be read as a promise that every backup, security record or previously created statistical artifact disappears at once.

This produces a familiar piece of privacy theater. The easy control manages what the user can see. The consequential control is broader, slower and tied to leaving.

A useful deletion screen would separate four actions without making the user decode a policy: remove this message from the conversation; stop it from shaping future replies; exclude it from product improvement; erase it from active systems and state when backups age out. Current consumer documentation tends to bundle some of those outcomes or leave their relationship unclear.

For someone using a chatbot during panic, grief or shame, ambiguity has a cost. The person must either trust a system they cannot inspect or abandon a tool that may be helping. Consent obtained through exhaustion is still excellent for retention metrics.

Crisis language changes the relationship

The sentence in this audit does not explicitly describe self-harm. A safety system might still classify it as distress, depending on the surrounding conversation. Crisis detection usually relies on rules or automated classifiers, systems that assign text to categories such as low mood or immediate danger.

Replika, Wysa and Youper all publish safety language that limits their role in emergencies and directs users toward emergency services or crisis resources in relevant circumstances. Wysa also presents dedicated crisis-support pathways within its product materials. The crucial point is not whether offering a hotline is good. It is.

The point is that a chatbot cannot both present itself as an intimate listener and leave users to guess what happens when their language crosses an internal threshold.

A crisis notice should state whether detection is automated, whether a human can review the conversation, whether the company contacts anyone, and which location data it uses to select resources. It should also admit the limits of the system. A classifier can miss oblique language. It can react to fiction, dark humor or a discussion about somebody else.

Brand documentation generally makes the disclaimer clearer than the mechanism. These products are not emergency services. They may surface resources. That still leaves a gap around escalation, human access and the retention of safety-related records, where a vulnerable user has the strongest reason to know the rules before typing.

The absence of automatic police or emergency contact can protect users who fear coercive intervention. It can also expose the limits of the support being offered. Both facts belong on the same screen, before crisis language appears, not buried in terms opened after the conversation has gone wrong.

The privacy control is distance

This audit does not prove misconduct by Replika, Wysa or Youper. It shows how little control a user can verify from the consumer side. Brand documentation can describe intended practice, but it cannot show whether every deletion propagated through a vendor’s systems or whether a deidentified dataset can be linked back to a distinctive life story.

The practical harm-reduction move is to reduce detail before disclosure. That is not a satisfying answer. The product invites detail because detail makes the reply feel relevant. Still, I am worried about work gives a system less identifying material than the employer, job title, location and event that caused the fear.

A separate email address can limit account linkage. Deleting an account can reduce future exposure, though it cannot retroactively provide a guarantee the policy never made.

The deeper fix belongs with the companies. Sensitive chat should default to short retention, model improvement should require a separate choice, and deletion should explain each layer it reaches. A mental-health interface should not demand more data literacy from a distressed user than a bank demands before moving money.

For now, I am scared I will lose my job, and I have not told anyone remains one sentence on the screen and several possible records underneath it. The bot’s reply may disappear in seconds. The policy does not move at conversational speed.

Questions people ask

Can a mental-health chatbot read my private messages?

The service must process what you type to generate a response, but that does not mean an employee routinely reads every chat. Policies may permit limited human access for safety, support, quality review or legal reasons, and outside service providers may process data under contract. Check the specific product’s current policy before sharing identifying details.

Does deleting a chatbot memory delete the original conversation?

Not necessarily. A memory control may remove a saved fact used for future replies while leaving the chat record, logs or backup copies governed by separate retention rules. Look for distinct explanations of message deletion, account deletion and backup expiry rather than assuming one visible control covers every copy.

Are chatbot conversations used to train AI?

That depends on the product and the data category. Policies may allow conversation data, feedback, deidentified records or usage information to support model evaluation and product improvement. If the app does not offer a clear opt-out, do not treat silence in the settings menu as a promise that your text is excluded.

Will a chatbot contact emergency services if I mention self-harm?

Consumer chatbot documentation commonly says the product is not an emergency service and may direct users to crisis resources. Escalation practices vary, and automated systems can misunderstand context. The app should explain human review and emergency-contact rules explicitly; if it does not, its crisis banner is not a full account of what happens to the disclosure.