Artificial intelligence (AI) is making waves in almost every industry, and accounting is no exception. The promise of automating tedious tasks, improving efficiency and gaining deeper insights from financial data is an exciting prospect for many businesses.
While it can be a powerful tool for many applications, a recent study highlights the importance of proceeding with caution before entrusting it with critical accounting functions.
The Penrose "AccountingBench" Study
A study published summer 2025 by Penrose Labs, called "AccountingBench" put leading AI models to the test to see if they could perform one of the most fundamental accounting tasks; closing the books for a real business. This involves ensuring a company's internal financial records accurately reflect its external reality — what the bank statements say, what customers owe and what the company owes to vendors. It's a meticulous process where small errors can compound over time with serious consequences.
The study used a year's worth of financial data from a real SaaS company and compared the AI's performance to that of a human CPA.
What Did the Study Find?
The results were a mixed bag, offering a valuable reality check on the current state of AI in accounting.
- Initial Promise, Long-Term Problems
While some of the most advanced AI models performed well in the initial months, they struggled to maintain accuracy over a longer period. Over time, they accumulated significant errors, leading to "incoherent results." - Some Models Stumbled Early
Not all AI models were up to the task. Some of the models tested were unable to even close one month's worth of books. - The "Butterfly Effect" of Errors
The study highlighted the “butterfly task,” where a small mistake in one period can cascade into much larger problems later. AI models particularly struggled with this long-term consequence of initial errors. - AI Doesn’t Understand Accounting Rules
The nuances of accrual accounting, deferred revenue and asset depreciation proved challenging for the models, which tended to revert to a simpler cash-basis approach.
Our Takeaway: Don't Replace Your Accountant With an Algorithm Just Yet
So, what does this mean for your business? The Penrose study reinforces a message we believe is crucial for our clients: while AI has the potential to be a fantastic assistant, it is not yet a replacement for the expertise, judgment and critical thinking of a human accountant.
The study shows that for complex, high-stakes tasks like closing the books, the current generation of AI can fall short. Blindly jumping into AI to replace core accounting functions without extensive testing and human oversight is a risky proposition.
Need Help?
It’s important to be discerning and to rely on trusted professionals for the critical work of ensuring your financial records are accurate and reliable. Contact us online or call 800.899.4623 to start a conversation on how we can help.
