Problems with AI counseling: what can go wrong and how to use it well

CareOps editorial team · November 7, 2026 · 8 min read

Mental wellness information and service guidance published by CareOps.

Have you ever told ChatGPT, Gemini or Claude about something that was weighing on you? The replies are fast and often comforting. But as a conversation goes on, you may wonder whether you can trust what it says. Here are the problems research and Korea's national guidance have documented, and how to use AI while keeping them small.

What problems come up in AI counseling? #

Many people talk to AI about their worries. In a survey behind Korea's national guidance, 68.8% of generative AI users also used it for emotional purposes such as everyday conversation, comfort or counseling. As use has grown, specific problems have been documented. The table summarizes six of them from research and the guidance.

Why does AI seem to agree with everything I say? #

Large language models sometimes lean toward agreeing with the person they're talking to. In a 2025 study, US researchers tested current large language models, including gpt-4o, on common counseling scenarios. The models at times expressed stigma toward mental health conditions or responded inappropriately to the scene, including encouraging delusional thinking. The researchers pointed to sycophancy as a likely reason and found the pattern in larger and newer models too.

Korea's guidance also advises not to treat an AI's agreement as evidence that you're right. Take the comfort, then ask it to separate what you actually said from what it inferred. Ready-to-use questions are in separating empathy from evidence.

Can I rely on AI counseling in a crisis? #

In a 2025 study, researchers asked ChatGPT, Claude and Gemini 30 hypothetical suicide-related questions, each rated for risk by 13 clinical experts, 100 times apiece: 9,000 responses in all. For very-low-risk and very-high-risk questions, all three matched expert judgment. For questions in between, the responses were inconsistent, sometimes answering and sometimes not.

When things feel very heavy, don't wait on an AI's reply; talk to someone close to you or a mental health service first. The guidance likewise advises stopping and telling a person if a reply feels risky or inappropriate. Crisis contacts are listed at the bottom of this page.

Does AI remember my past conversations? #

General-purpose AI has memory features too. ChatGPT can keep your preferred style or refer to earlier context through custom instructions and memory. What gets remembered, and when it's used, varies by service and setting. So two things can happen: you explain the same worry from scratch each time, or something you'd rather leave behind shows up in a later chat.

Check the settings of the AI you use to see what's stored, and end each conversation you want to continue with a short summary. How to do that is in keeping an AI counseling conversation going.

Is my personal information safe when I talk to AI? #

The guidance advises checking how your input is stored and used before you start, because what you share is hard to take back. Leave out names, contact details and addresses that could identify you or others, and ask about a medical document's contents rather than uploading it. It also helps to read a service's privacy policy or similar page for storage, outside transfer and deletion.

Should I believe it when AI sounds like a diagnosis? #

In the same survey, 14.6% had diagnosed themselves based on an AI's answer or taken it as a medical diagnosis, while 11.9% had checked the advice with a professional or official source. AI answers from what you type. Even a convincing reply is better treated as a question to look into than a conclusion about you. Diagnosis and medication are decided in medical care; if you want a first look at how you're doing, try a free self-check.

What if I can't stop talking to the AI? #

Because AI answers instantly at any hour, conversations can run on. In the survey, 16.3% kept a conversation going even when trying to focus on real-life tasks, rising to 23.4% among those using AI for counseling. 10.6% had made an important decision on AI advice alone without talking to anyone, 17.8% among counseling users. Before you begin, decide what you'll talk about and when you'll stop, and take big decisions to people you trust as well.

How to use AI counseling while keeping the risks small #

1. Set a purpose and an end point: pick one thing to sort out today and decide when you'll stop.
2. Leave out identifying details: no names, contacts, addresses, school or workplace names, and no documents.
3. Don't treat agreement as proof: ask it to separate what you said, what it guessed and what's still unknown.
4. Turn diagnosis-like claims into questions: check them with a professional or an official source.
5. Keep a summary if you want to continue: write a few lines on what you covered and what to look at next.
6. In hard moments, talk to people too: reach someone close to you, a counselor or a mental health service.

This checklist draws on the recommendations from Korea's Ministry of Health and Welfare and the Korean Neuropsychiatric Association. All 21 recommendations and a checklist for choosing an app are in the mental health AI guideline overview. How AI and human counselors differ, and how to use both together, is covered in AI counselor or human counselor?.

What does an AI app built for counseling do differently? #

General-purpose AI is built to answer anything; an AI counseling app is built around the flow of counseling from the start. When choosing one, check how it's designed for each problem above. The right-hand column describes CareOps Maeumi.

CareOps Maeumi is an AI counseling app built around sessions, established approaches and memory of past conversations. You can see how it compares with general-purpose AI in ChatGPT vs. AI counseling apps. Get the app

This guide shares mental wellness information. If you are in immediate danger, contact emergency services where you are. Support and contact information.