Using AI chatbots: privacy, accuracy and scams explained

Is it safe to use AI chatbots? Like any online service, they come with trade-offs, mainly around what happens to what you type, whether the answers are right, and scams that use the same technology. AI chatbots can answer questions, draft emails, explain a confusing letter and summarise long documents in plain language. Many people now use them for everyday personal tasks. For anyone asking whether it’s safe to use AI chatbots, this guide explains three things: what happens to information you type into a chatbot, why the answers can be wrong, and how scammers are using the same technology. It also shows how chat history settings work, using one provider as an example.

This guide is about personal use. Using AI tools at work raises extra questions about business information, covered in AI assistants at work and confidential data.

Why it matters

Chatbots feel like a private conversation, but they’re an online service run by a company. What you type, paste or upload is sent to that company’s computers. And because chatbots write fluent, confident text, it’s easy to trust an answer that turns out to be wrong.

How a chatbot works, in brief

A generative AI chatbot is built on a large language model: software trained on huge amounts of text to predict which words are likely to come next. It doesn’t look facts up the way a search engine does unless it has been connected to a search tool, and it doesn’t “know” whether what it writes is true. That’s why it can produce answers that read well but contain mistakes, invented quotes or references that don’t exist.

The Australian Signals Directorate’s Australian Cyber Security Centre (ASD’s ACSC) describes this in its Engaging with artificial intelligence publication, written for organisations: generative AI systems are “known to hallucinate information that is not factually correct”. “Hallucinate” is the term used when an AI makes something up.

What happens to what you type

The Office of the Australian Information Commissioner (OAIC) is Australia’s privacy regulator. Its guidance on commercially available AI products is written for businesses, but it describes risks that apply to anyone:

• Once personal information is entered into an AI system, it “will be very difficult to track or control how it is used”.
• Some products’ terms or settings “allow the product owner to collect the data input by customers for further training and development of AI technologies.”
• The OAIC warns this can lead to personal information entered by one user appearing in a response to another.

The ACSC makes a similar point, saying entered information “may be incorporated into the system’s training data”.

What the OAIC says not to share

The OAIC’s guidance says that, as a matter of best practice, organisations should “not enter personal information, and particularly sensitive information, into publicly available AI chatbots”.

Personal information is information about an identifiable person. The OAIC describes sensitive information as “a subset of personal information that is generally afforded a higher level of privacy protection”. Its examples include a person’s “political opinions or religious or philosophical beliefs”, and it notes that photos can reveal sensitive information such as race or health details.

In everyday terms, this can include:

• full names alongside addresses, dates of birth or phone numbers
• identity documents, account numbers, passwords or one-time codes
• medical results or details of someone’s health
• other people’s information, photos or recordings.

The OAIC’s page for individuals, from its Privacy Awareness Week campaign, describes being careful about “what you tell generative AI” and being “choosy about who you give your information to, and what you consent to”. It notes that default settings on apps and services “may discourage users from making privacy-protective choices”, and describes choosing more protective options where they’re offered.

A fictional example

Priya wants help writing a complaint letter about a medical bill. Her first draft prompt includes her full name, address, Medicare number and test results. Before sending it, she remembers the OAIC’s point that information entered is hard to control. She replaces the details with placeholders (“[my name]”, “[the test]”), gets a draft letter, and fills in her own details in her word processor afterwards.

Chat history and training settings

Most chatbots keep a history of your conversations, and many have settings that control whether conversations are saved to your account or used to improve the service. These settings differ by provider and change often, so each provider’s privacy policy and help pages are the place to check.

How to review or turn off chat activity (Google Gemini example)

Based on Google’s instructions (checked October 2026). Menus can change; Google’s guide has the latest steps. Other chatbots have similar settings, described in their own help pages.

1. Go to gemini.google.com and sign in.
2. Click Settings & help, then Activity. (Google also gives the direct address myactivity.google.com/product/gemini.)
3. Review saved conversations. To remove some, use Delete above the list and choose a time range (such as the last hour, last day, all time or a custom range), or the delete icon next to a single item.
4. To stop saving future chats, click On near the top, then choose Turn off or Turn off and delete activity.

Google says: “If you’re 18 or over, Keep Activity is on by default.” It also says that even with it turned off, “your conversations will be saved with your account for up to 72 hours” to provide the service and process feedback. In other words, turning off history doesn’t mean nothing is stored at all.

Checking AI answers

AI chatbots can sound confident and still be wrong. The OAIC puts it this way: “AI outputs can often appear credible even when they produce errors or false information.”

The OAIC’s guidance, written for businesses, describes people being able to verify information and “critically assess outputs”. The ACSC similarly describes understanding “the extent to which the system’s outputs can be relied upon”. In practice, that tends to mean:

• checking important facts against the original source, such as the government agency or organisation concerned
• treating quotes, references and links in an answer as unconfirmed until they’ve been found and opened
• being especially careful with health, money, legal and government-payment questions, where the “Where to get official advice” box on each guide on this site lists the official services.

How scammers use AI

The same tools are available to scammers. Scamwatch’s page on how scammers use technology and AI describes several methods:

• Deepfakes. “Scammers use AI to create fake videos and audio recordings that look and sound real.” Scamwatch says these can show public figures appearing to endorse investments.
• Voice cloning. “Scammers can use a few seconds of a voice recording to create AI clones”, which are then used in calls pretending to be someone the person knows.
• Chatbots run by scammers. Scamwatch says “AI chatbots can have conversations with multiple people at the same time”, and can keep fake personalities going for months.
• Fake websites. “Scammers use AI tools to create perfect copies of real websites”, including working contact forms.

The deepfake and voice-cloning scams guide explains these in more detail.

What Scamwatch says to do

Scamwatch describes these steps for AI-assisted scams:

1. “Always call the official phone number of the organisation to check”, or call the person back on a number you already know is theirs, rather than using contact details in the message.
2. “Always type web addresses directly into your browser” instead of clicking links in emails, messages or social media ads.
3. “Never send money to someone you’ve only met online.”
4. Check investment offers by searching for scam warnings and checking ASIC’s website to see whether the person is licensed. See how investment scams work.

Things to watch for

• Prompts that include identity, account or health details. The OAIC describes not entering personal and sensitive information into public chatbots.
• Fake chatbot apps and websites. Scamwatch says scammers make convincing copies of real websites.
• Answers that include links. A link in an AI answer may be wrong or lead somewhere unexpected; typing the address of a known site is what Scamwatch describes for organisations.
• Celebrity videos promoting investments. Scamwatch says deepfakes can show public figures appearing to endorse investments.
• A familiar voice asking for money urgently. Scamwatch describes calling the person back on a number you already know.

Common questions

Is it safe to use AI chatbots?

They come with trade-offs, like any online service. The OAIC says that once information is entered into an AI system, it “will be very difficult to track or control how it is used”, and that AI outputs “can often appear credible even when they produce errors or false information.” Scamwatch also describes scammers using AI for deepfakes, voice cloning and fake websites. This guide covers what happens to what you type and checking AI answers.

Is it safe to share personal details with an AI chatbot?

The OAIC describes keeping personal information, and particularly sensitive information, out of publicly available AI chatbots as best practice, in guidance written for organisations. In everyday terms, that can include identity documents, account numbers, passwords and one-time codes, health details, and other people’s information or photos. The OAIC notes photos can reveal sensitive information too. See what the OAIC says not to share, including a fictional example of using placeholders.

Are my AI chatbot conversations private?

Not in the way a private conversation is. What you type, paste or upload is sent to the provider’s computers and handled under its privacy policy and settings. Google, for example, says: “If you’re 18 or over, Keep Activity is on by default”, and that even with it turned off, “your conversations will be saved with your account for up to 72 hours”. Each provider’s help pages explain its own settings. See chat history and training settings.

Do AI chatbots use my chats for training?

Some can. The OAIC says some products’ terms or settings “allow the product owner to collect the data input by customers for further training and development of AI technologies”, and warns this can lead to personal information entered by one user appearing in a response to another. The ACSC says entered information “may be incorporated into the system’s training data”. Each provider’s privacy policy explains whether and how it does this. See what happens to what you type.

Why do AI chatbots make things up?

Because a chatbot predicts which words are likely to come next, rather than checking whether what it writes is true. The ACSC says generative AI systems are “known to hallucinate information that is not factually correct”, which can include invented quotes and references that don’t exist. The OAIC and the ACSC both describe checking outputs, and important facts can be checked against the original source. See how a chatbot works, in brief.

What do I do if I’ve been scammed with a deepfake or AI chatbot?

Scamwatch describes contacting your bank or card provider immediately, changing passwords, getting free help from IDCARE on 1800 595 160, and reporting the scam to Scamwatch. For AI-assisted scams, it describes calling the official phone number of the organisation, or calling the person back on a number you already know is theirs. The guide to what to do first after a scam brings these steps together.

Where to get official help

• The OAIC’s Privacy Awareness Week page for individuals explains privacy rights and protections.
• If money or details have already been handed over, Scamwatch describes contacting your bank or card provider immediately, changing passwords, getting free help from IDCARE on 1800 595 160 (also listed on IDCARE’s contact page), and reporting the scam to Scamwatch. The guide to what to do first after a scam brings these steps together.
• Related: impersonation scams, what prompt injection is and securing your online accounts.