Checking AI-generated content: facts, sources, images and voices

Knowing how to check if something is AI-generated, and whether an AI answer is right, is now an everyday skill, and no single tool can reliably tell either way. The checks official and maker sources describe work regardless: trace each claim back to its source, and confirm a familiar voice or face through a separate, trusted channel. AI chatbots can produce confident answers that are wrong, and scammers use AI to make fake images, videos and voices. This guide explains why AI makes mistakes, how official and maker sources describe checking facts and sources, how AI shows up in scams, and what tools such as content credentials can and can’t tell you.

Why AI gets things wrong

Generative AI tools write by predicting likely words, based on patterns in the data they were trained on. That makes them fluent, but fluency isn’t the same as accuracy. ASD’s guidance Engaging with artificial intelligence says outputs generated by an AI system “may not always be accurate or factually correct.”

When an AI tool confidently produces something false or made up, it is often called a hallucination. The NSW Government’s generative AI guidance describes this as “confidently producing nonsensical or off-topic information”. It also notes that AI output “can be biased or taken out of context”, because the technology “learns from extensive internet data”.

Microsoft’s guide to checking Copilot output describes several ways answers go wrong. It says a response “can sound polished and still be wrong, incomplete, or risky to use”, and that the AI may have “blended ideas, overstated certainty, or filled gaps with assumptions.” It adds that “Many AI mistakes are really omissions”: the answer leaves out a caveat, a risk or an exception.

Common examples include:

• Made-up facts, dates, numbers or quotes.
• References, links or reports that don’t exist, or that don’t say what the AI claims.
• Out-of-date information presented as current.
• Rules or laws from another country presented as if they apply in Australia.

How to check AI answers

Microsoft’s guidance frames checking around a few questions. Adapted for everyday use:

1. Does it match the source? Microsoft asks: “Do the claims match the original files, notes, messages, or data?” If a statement “can’t be traced back to a source”, Microsoft says to treat it as unconfirmed until verified.
2. Are the key details confirmed? Microsoft lists “facts, names, dates, numbers, and recommendations” as the details to confirm.
3. What’s missing? Microsoft suggests checking for missing caveats, assumptions or risks.
4. Would it hold up elsewhere? Microsoft asks: “Would this response hold up in different scenarios?”

The NSW Government’s guidance puts it simply: “Fact-check all information against a verifiable source.”

In practice, checking often means:

• Opening the cited source and finding the exact sentence, rather than trusting the summary.
• Going to the official source directly, such as a government agency’s own website, for anything about money, health, law or government services.
• Checking the date of the information, especially for prices, rules and product features.
• Being extra careful with numbers, which AI tools can get wrong even when the rest of the answer is right.

Microsoft also notes that Copilot “can help you validate its output—but it can’t certify its own correctness.” Asking the same AI “are you sure?” isn’t a substitute for checking an independent source.

For what to share with chatbots in the first place, see our guide to using AI chatbots safely.

How AI-generated images, video and voices are used in scams

Scamwatch says “Scammers use AI to create fake videos and audio recordings that look and sound real.” It describes several patterns:

• Fake endorsements. Scamwatch says deepfakes can make public figures appear to back investments, products or requests for money.
• Voice clones. Scamwatch says “Scammers can use a few seconds of a voice recording to create AI clones”, which are used to call people pretending to be a friend or family member in distress.
• Fake people. Scamwatch says scammers use AI to “modify existing images and create photo albums of someone who doesn’t exist”, which can support fake profiles.

Our guides to deepfake and voice cloning scams and how scammers use AI in phishing and scams cover these in more depth.

What Scamwatch says to do

Scamwatch’s approach doesn’t rely on spotting the fake. It focuses on checking independently:

• “Always take the time to call your friend or family member back on their real number to check.”
• “Don’t reply to a message from a new number claiming it’s your friend or family member.”
• “A personalised message doesn’t make it legitimate.”
• “Always call the official phone number of the organisation to check.”

This matters because Scamwatch describes AI fakes that “look and sound real.” Visual or audio clues people sometimes look for, such as odd details or unnatural movement, may simply not be there, and their absence doesn’t mean a video or voice is genuine. Checking through a separate, trusted channel works regardless of how good the fake is.

A fictional example

Daniel gets a voice message that sounds exactly like his manager, saying she’s stuck at an airport and needs gift card codes sent to a new number. The voice is convincing. Instead of replying, Daniel calls his manager on the number already saved in his phone. She’s at her desk and knows nothing about it. (Fictional; any resemblance is coincidental.)

What content credentials and watermarks can show

Some tools and organisations are adding information to images, video and audio to show where they came from. Two approaches are common.

Content Credentials (C2PA)

Content Credentials are based on an open standard developed by the Coalition for Content Provenance and Authenticity (C2PA). The C2PA describes them as “a cryptographically bound structure that records an asset’s provenance”, meaning facts about the history of a piece of content, such as which tool created or edited it and whether generative AI was used.

The C2PA is clear about the limits:

• Provenance information “cannot tell you whether the digital content is true, accurate or factual”.
• “Provenance is not always complete”, because edits made with tools that don’t support the standard may go unrecorded.
• The information can be removed from a file, although the C2PA describes techniques such as watermarking to help recover it.
• Many files have no credentials at all. The C2PA says adding them is optional, and that it doesn’t intend for unlabelled content to be treated as automatically less trustworthy.

So credentials can add useful context when present. Their absence proves nothing either way.

How to inspect Content Credentials with Adobe’s tool

Based on Adobe’s instructions (checked October 2026). Adobe labels this tool a beta. Adobe’s Inspect Content Credentials help page has the latest steps.

1. Go to the Adobe Content Authenticity website (contentauthenticity.adobe.com).
2. In the top navigation bar, select the Inspect tab.
3. Select Browse files, then choose the file on your device.

Adobe says the tool shows details such as who created the content, how it was made, and whether generative AI was used, and that uploaded content “isn’t stored anywhere.”

Google SynthID watermarks

Google describes SynthID as “a tool that embeds digital watermarks directly into content generated by Google’s AI models.”

How to check for a SynthID watermark in Gemini

Based on Google’s instructions (checked October 2026). Google’s SynthID help page has the latest steps and file limits.

1. Sign in to Gemini in a web browser on a computer.
2. Attach one image, video or audio file.
3. Ask a question such as “Is this AI-generated?”

Google says Gemini “can currently only recognize content created by Google AI tools.” If no watermark is found, Google says the content “could have been created by other AI systems.” Google also lists cases where it can’t give a clear answer, such as content too simple or abstract to carry a watermark.

Things to watch for

• Confident answers with no source, or sources that don’t say what the AI claims.
• Citations that don’t exist or lead to unrelated pages.
• Urgency combined with a familiar voice or face, especially requests for money, codes or gift cards.
• Messages from new numbers claiming to be someone known.
• Treating a detection tool as a verdict. Both C2PA and Google describe limits to what their tools show.
• Sharing before checking. Passing on an AI-generated claim or image can spread it further.

Common questions

How can I tell if something is AI-generated?

Often you can’t tell for certain, because no single tool reliably detects all AI-generated content. The checks official and maker sources describe work either way: trace each important claim back to its source, confirm facts, names, dates and numbers, and check a familiar voice or face through a separate channel. Content credentials and watermarks can add context but aren’t a verdict. See how to check AI answers and what content credentials can show.

What are AI hallucinations?

A hallucination is when an AI tool confidently produces information that is false or made up. The NSW Government’s generative AI guidance describes it as “confidently producing nonsensical or off-topic information,” and ASD says AI outputs “may not always be accurate or factually correct.” Common examples include made-up facts or quotes, references that don’t exist, and rules from another country presented as if they apply in Australia. See why AI gets things wrong.

How do I fact-check an AI answer?

Microsoft’s guidance describes checking that claims match the original source and confirming the key facts, names, dates, numbers and recommendations. The NSW Government’s guidance says to “Fact-check all information against a verifiable source.” Going directly to the official source, such as an agency’s own website, is often the quickest check. Microsoft notes Copilot can’t certify its own correctness, so asking the AI “are you sure?” isn’t an independent check. See how to check AI answers.

How can I tell if an image is AI-generated?

There’s no reliable way in every case. Content Credentials or a Google SynthID watermark can show that some content was made or edited with AI, but the C2PA and Google both say that missing credentials or no watermark doesn’t prove content is real. Google says Gemini can currently only recognise content created by Google AI tools. What content credentials and watermarks can show has the steps for Adobe’s Inspect tool and Gemini.

How can I tell if a voice call is an AI voice clone?

It can be very hard to tell by listening. Scamwatch says scammers can use a few seconds of a voice recording to create AI clones, so a convincing voice isn’t evidence it’s genuine. Scamwatch says to call the person back on their real number to check, and not to reply to messages from new numbers claiming to be a friend or family member. See what Scamwatch says to do.

What do I do if I think I’ve been scammed with an AI voice or deepfake?

Scams can be reported to Scamwatch at scamwatch.gov.au/report-a-scam. If money has been sent, our guide to first steps after a scam explains what official services say to do. If personal information may have been misused, IDCARE says it offers free expert support, and its website lists 1800 595 160 for individuals. Our guide to deepfake and voice cloning scams covers these scams in more depth.

Where to get official help

• Scamwatch’s page on how scammers use technology and AI has current warnings, and scams can be reported at scamwatch.gov.au/report-a-scam.
• If personal information may have been misused, IDCARE says it offers free expert support; its website lists 1800 595 160 for individuals.
• If money has been sent to a scammer, our guide to first steps after a scam explains what official services say to do.