When an assistant reads a table and answers in prose, its counts and totals can be wrong while sounding certain. It can also work on part of the file and call it all of it. In bookkeeping tests, models have even invented transactions to make totals agree.
Language models do not reliably count or add in their heads, and long inputs get cut or skimmed. Analysis libraries also drop blanks quietly. A model asked to make numbers agree may change the data instead of reporting the gap.
Make the agent compute every number in code and show the output. Ask for rows loaded, rows used and rows dropped with reasons on every run. Check coverage with ids, so every input row has exactly one result. Compare one number against a source you trust before you share anything. Tell the agent to report differences, never to adjust data so totals agree.
Rosidi reports invented order counts that did not match a 40-row file, code shown but not run, and an analysis that used 126 of 352 rows because blanks were dropped without comment.
Mistelbacher says none of the mention counts ChatGPT gave him matched his own check in Google Sheets, and it gave no sign that they might be wrong.
Ipsos researchers report theme totals of 107, 106 and 101 percent from the same GPT-4 prompt on three days, and partly blame their prompt for not asking for one code per answer.
The team says staff often counted feedback by hand because LLMs sometimes miscount, and that checking LLM output was slow and error-prone.
A bug report says Rovo returned 16 rows from a 26-row file when asked for every row matching one author, and said those were all of them.
The poster says a ChatGPT step in Make processed only the first row of a CSV despite instructions to handle the whole file.
The benchmark authors describe models degrading over months of bookkeeping, and commenters quote the report on models inventing or pulling in unrelated transactions to pass a reconciliation check.