AI Hallucinations in Financial Reports, How to Stop Them

Last reviewed: September 2026

Why does an AI make up a number that looks right?

Because it is not reading your ledger, it is producing plausible text. A language model predicts what comes next. When that is a supplier's ABN, a September GST figure or a reference to an accounting standard, it produces something with the correct shape: right number of digits, right order of magnitude, right formatting. Shape is what it optimises for. Truth is a by-product, and not guaranteed.

Why it guesses rather than abstains is better understood than most people assume. The paper Why Language Models Hallucinate, published 4 September 2025 by Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, opens: "Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty." Its argument is that models hallucinate because training and evaluation procedures reward guessing over acknowledging uncertainty, and that the problem persists because models are optimised to be good test-takers. A model rewarded its whole life for answering will answer.

The finance version is obvious. Ask a general-purpose assistant what a client's June quarter GST liability was, give it no access to the file, and it will not refuse. It will produce a figure: correctly formatted, believable, entirely invented. That is a different problem from the coding errors in why QuickBooks and Xero AI gets things wrong, where the tool really is reading the file and applying a bad rule. Here there is no rule and no file, only fluent text.

Has this actually happened in real professional reports?

Yes, in Australia and overseas, at firms with far more review process than a small practice.

Two details matter more than the embarrassment. The Deloitte Australia corrections ran deep enough that the Department of Employment and Workplace Relations has republished the document twice: its final report page records a version updated 3 February 2026, replacing one dated 26 September 2025. And Sullivan and Cromwell's office manual already told lawyers to trust nothing and verify everything, yet citation checking still missed the invented authorities. A policy telling people to check is not a control. It is a hope.

For scale, the legal profession keeps a public count. Damien Charlotin's AI Hallucination Cases database, last updated 5 September 2026, lists 2,023 court and tribunal decisions worldwide where a party relied on hallucinated material, 110 in Australia. Accounting has no equivalent register, which is not the same as having no equivalent problem.

What is the difference between an AI that retrieves and an AI that recalls?

It is the whole question, and vendor marketing blurs it. An AI that recalls answers from patterns absorbed during training, with no live connection to your file and no way to tell a fact it learned from one it is inventing. An AI that retrieves is handed the records first: the bank line, the invoice, the ledger extract. It then reports what it was given and cites where each figure came from.

Both produce fluent, confident prose. Only one gives a figure you can check in nine seconds. So the rule for Australian bookkeepers and BAS agents is simple: a number that cannot be traced to a transaction ID is not a number, it is a guess. Retrieval does not make an AI correct. It makes its errors visible, the only property that has ever made a control work.

Retrieval has limits worth naming. A retrieval-based tool can still summarise the right ledger badly, use the wrong date range, or blend two similarly named accounts. It can be handed a source and quietly wander off it, which is how the EY study ended up with footnotes that did not support the claims above them. The gap between "cited" and "correct" is where a human reviewer earns their fee, as we argued in our piece on trusting AI with transaction categorisation.

How do I test whether a finance AI is retrieving or guessing?

Four tests, in order, run against a real client file rather than a vendor sandbox. Any tool failing test three should not touch a ledger.

1. Source-linked figures. Ask for a number, then ask where it came from. A retrieval-based tool returns an invoice number, a date, an account code and ideally a link into Xero or MYOB. A recall-based tool restates the number in different words, or gives a reference that does not resolve. Try three figures, because a tool can get lucky once.
2. Reconciliation to a control total. Ask for a breakdown that must add to something you already know: the bank balance, the GST clearing account, the debtors ledger total. Fabricated components almost never reconcile, so this catches most invented detail without verifying a single line.
3. Refusal behaviour when data is missing. Remove a document from the file, then ask a question that needs it. The correct answer names the missing record. The dangerous answer is a plausible figure. Vendors never volunteer this test, and it separates finance products from chat interfaces pointed at finance.
4. Variance check against the prior period. Compare against the same period last quarter and last year, and make the tool explain any movement above a threshold you set. Hallucinated figures are plausible in isolation and implausible in a time series. This also catches genuine data errors, the more common problem.

Worked example, labelled as such rather than dressed up as a client story. A practice asks for the subcontractor spend going into a board pack and gets $284,610. Test one: twelve supplier bills cited by document number. Test two: those twelve add to the trial balance. Test three: one supplier's June invoice was never uploaded, and the tool says so instead of estimating. Test four: the figure sits 41% above last quarter, and the tool names the two bills causing it. That number is defensible in a way it never was before.

Which finance outputs are safe to automate, and which are not?

The dividing line is not task difficulty, it is whether a wrong answer would be visible. Work where an error announces itself can be automated. Work where an error hides inside a plausible figure cannot.

Who carries the wrong number in Australia?

The practitioner does, in terms that leave no room for a vendor-blaming defence. TPB(GS) 55/2026, issued 22 July 2026, states that tax practitioners remain accountable for the accuracy of information and advice they provide to their clients. It addresses hallucination directly: AI models may hallucinate or provide inaccurate information and as such cannot be relied on as a replacement for tax knowledge, experience or expertise. It then asks practitioners to verify and review AI generated content for accuracy throughout each step of the workflow, and to document each step. The consent side is covered in our companion post on what tax and BAS agents must disclose.

Record keeping is no more helpful. The ATO's record-keeping rules for business require that records be kept, not be changed, and be stored in a way that protects them from change, and note the ATO may ask you to show appropriate safeguards are in place. A hallucinated figure that has flowed into a lodgement is not fixed by deleting the chat log. You need evidence the figure had a source when it was used, the architecture set out in the sibling post on AI agents connecting directly into Xero and QuickBooks.

The confidence gap is worth naming. Workiva's 2026 Midyear Executive Benchmark Survey, reported by Accounting Today on 11 August 2026, found 84% of executives in finance, audit and sustainability somewhat or very confident in AI accuracy, 26% saying internal audits had detected AI errors reaching external audiences or the board, and only 11% feeling their own data was AI-ready. That is a global survey, not an Australian one. UK research commissioned by ICAS, published 17 March 2026, found 72% of accounting professionals surveyed were concerned generative AI could produce errors or reach incorrect decisions. Executive confidence and practitioner worry point in opposite directions, and the incidents above suggest the practitioners are right.

What does an AI built not to hallucinate actually look like?

It looks like a system with permission to fail loudly. That is the design rule Agentive builds to, and it follows from the research: an AI allowed to say "I do not have this" hallucinates less than one that is not. If the only acceptable output is an answer, the system will manufacture answers. If "the June supplier invoice is missing, I cannot complete this reconciliation" is an expected result rather than a failure, the pressure to invent disappears.

Three properties follow. Agentive's finance capability works against the source records in Xero and MYOB rather than from recall, so every figure ties to a transaction. Every action is written to a per-action audit log, so a figure's provenance survives the conversation. When a document is missing or ambiguous, the configured behaviour is to escalate rather than estimate. It runs single-tenant on AWS Sydney with inference inside Australia, and client data is never used to train a model, for a separate reason set out in our data governance write-up.

Our honest position, having built this for Australian practices: the escalation path is harder to design than the automation, and it decides whether the tool is usable in a regulated practice. Anyone can build something that answers. Building something that reliably declines, names what it is missing, and routes it to the right person is the real engineering problem. The accountants use-case page maps where those escalation points sit.

What should a practice do about this in the next fortnight?

Run the four tests on whatever AI tool is already in the practice, including the ones nobody approved. Most firms find a staff member pasting client figures into a general assistant and reading the answer back out: no retrieval, no citation, no audit trail, and a confidentiality problem on top. Then add one line to the review checklist: no figure enters a client deliverable unless someone can name the transaction or document it came from. That line costs nothing and would have caught every incident above.

The reassuring close would be that the models are improving and the problem is receding. The models are improving. The problem is not, because the incentive causing it sits in how models are trained and graded, not in how large they are. What has changed is that the defence is understood and buildable: retrieve rather than recall, cite the source record, reconcile to a control total, and let the system refuse. Australian bookkeepers and accountants do not need an AI that is never wrong. They need one that cannot be wrong invisibly. That is the standard the AI Operation Engine is built to, and the one to demand of every vendor asking for a client file.