Humans Reign Supreme: ChatGPT Falls Short on Accounting Exams

The massive study was crowd-sourced with 327 co-authors from 186 institutions in 14 different countries.

Last month, OpenAI introduced its latest AI chatbot product, GPT-4. The company claims that the bot, which employs machine learning to produce text in a natural language style, performed exceptionally well on various exams. Specifically, it achieved a score in the 90th percentile on the bar exam, passed 13 out of 15 AP exams, and received a near-perfect score on the GRE Verbal test.

Academicians from Brigham Young University (BYU) and 186 other institutions were curious about the performance of OpenAI’s technology on accounting exams. Hence, they subjected the original ChatGPT model to testing. The researchers have stated that although ChatGPT still requires improvement in the field of accounting, it has the potential to revolutionize the way people teach and learn, for the better.

“When this technology first came out, everyone was worried that students could now use it to cheat,” said lead study author David Wood, a BYU professor of accounting. “But opportunities to cheat have always existed. So for us, we’re trying to focus on what we can do with this technology now that we couldn’t do before to improve the teaching process for faculty and the learning process for students. Testing it out was eye-opening.”

Since its debut in November 2022, ChatGPT has become the fastest-growing technology platform ever, reaching 100 million users in under two months. In response to intense debate about how models like ChatGPT should factor into education, Wood decided to recruit as many professors as possible to see how the AI fared against actual university accounting students.

His co-author recruiting pitch on social media exploded: 327 co-authors from 186 educational institutions in 14 countries participated in the research, contributing 25,181 classroom accounting exam questions. They also recruited undergrad BYU students (including Wood’s daughter, Jessica) to feed another 2,268 textbook test bank questions to ChatGPT. The questions covered accounting information systems (AIS), auditing, financial accounting, managerial accounting, and tax, and varied in difficulty and type (true/false, multiple choice, short answer, etc.).

ChatGPT’s Performance: Strengths and Weaknesses

Although ChatGPT’s performance was impressive, the students performed better. Students scored an overall average of 76.7%, compared to ChatGPT’s score of 47.4%. On 11.3% of questions, ChatGPT scored higher than the student average, doing particularly well on AIS and auditing. But the AI bot did worse on tax, financial, and managerial assessments, possibly because ChatGPT struggled with the mathematical processes required for the latter type.

When it came to question type, ChatGPT did better on true/false questions (68.7% correct) and multiple-choice questions (59.5%), but struggled with short-answer questions (between 28.7% and 39.1%). In general, higher-order questions were harder for ChatGPT to answer. In fact, sometimes ChatGPT would provide authoritative written descriptions for incorrect answers, or answer the same question in different ways.

“It’s not perfect; you’re not going to be using it for everything,” said Jessica Wood, currently a freshman at BYU. “Trying to learn solely by using ChatGPT is a fool’s errand.”

The researchers also uncovered some other fascinating trends through the study, including:

• ChatGPT doesn’t always recognize when it is doing math and makes nonsensical errors such as adding two numbers in a subtraction problem, or dividing numbers incorrectly.
• ChatGPT often provides explanations for its answers, even if they are incorrect. Other times, ChatGPT’s descriptions are accurate, but it will then proceed to select the wrong multiple-choice answer.
• ChatGPT sometimes makes up facts. For example, when providing a reference, it generates a real-looking reference that is completely fabricated. The work and sometimes the authors do not even exist.

ChatGPT’s Potential in Teaching and Learning

That said, authors fully expect GPT-4 to improve exponentially on the accounting questions posed in their study, and the issues mentioned above. What they find most promising is how the chatbot can help improve teaching and learning, including the ability to design and test assignments, or perhaps be used for drafting portions of a project.

“It’s an opportunity to reflect on whether we are teaching value-added information or not,” said study coauthor and fellow BYU accounting professor Melissa Larson. “This is a disruption, and we need to assess where we go from here. Of course, I’m still going to have TAs, but this is going to force us to use them in different ways.”

Reference: “The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?” by David A. Wood, Muskan P. Achhpilia, Mollie T. Adams, Sanaz Aghazadeh, Kazeem Akinyele, Mfon Akpan, Kristian D. Allee, Abigail M. Allen, Elizabeth D. Almer, Daniel Ames, Viktor Arity, Dereck Barr-Pulliam, K. Asli Basoglu, Andrew Belnap, Jeremiah W. Bentley, Terje Berg, Nathan R. Berglund, Erica Berry, Avishek Bhandari, Md Nazmul Hasan Bhuyan, Paul W. Black, Eva Blondeel, David Bond, Annika Bonrath, A. Faye Borthick, Erik S. Boyle, Marianne Bradford, Duane M. Brandon, Joseph F. Brazel, Bryan G. Brockbank, Marcus Burger, Dmitri Byzalov, James N. Cannon, Cecil Caro, Abraham H. Carr, Jack Cathey, Ryan Cating, Kimberly Charron, Stacy Chavez, Jason Chen, Jennifer C. Chen, Jennifer W. Chen, Christine Cheng, Xu Cheng, Brant E. Christensen, Kimberly Swanson Church, N. J. Cicone, Patience Constance, Lauren A. Cooper, Candice L. Correia, Joshua Coyne, W. Alec Cram, Asher Curtis, Ronald J. Daigle, Steve Dannemiller, Stephan A. Davenport, Gregory S. Dawson, Karen J. L. De Meyst, Scott Dell, Sebahattin Demirkan, Christine A. Denison, Hrishikesh Desai, Steven DeSimone, Leah M. Diehl, Ruth Dimes, Bei Dong, Amy Donnelly, Adam du Pon, Huijue Kelly Duan, Ada Duffey, Ryan T. Dunn, Mary P. Durkin, Ann C. Dzuranin, Rachel M. Eberle, Matthew S. Ege, Dina El Mahdy, Adam Esplin, Marc Eulerich, Patricia Everaert, Nusrat Farah, Lauren Farish, Michael Favere-Marchesi, Dutch Fayard, Jessica R. Filosa, Melinda Ford, Diana R. Franz, Bachman P. Fulmer, Sarah Fulmer, Zhan Z. Furner, Sonia Gantman, Steve Garner, Jace Garrett, Xin Geng, Joanna Golden, William Goldman, Josue Gomez, Mark Gooley, Shawn P. Granitto, Karen Y. Green, Cindy L. Greenman, Gaurav Gupta, Ronald N. Guymon, Kevin Hale, Christopher J. Harper, S. Allen Hartt, Holly Hawk, Steven R. Hawkins, Erin M. Hawkins, David C. Hay, Rafael Heinzelmann, Cassy D. Henderson, Bradley E. Hendricks, William G. Heninger, Mary S. Hill, Nicole Holden, D. Kip Holderness, Jr., Travis P. Holt, Jeffrey L. Hoopes, Sheng-Feng Hsieh, Feiqi Huang, Hua-Wei Huang, Ting-Chiao Huang, Brian W. Huels, Kara Hunter, Patrick J. Hurley, Kerry Inger, Sharif Islam, Isaac Ison, Hussein Issa, Andrew B. Jackson, Scott C. Jackson, Diane J. Janvrin, Peggy D. Jimenez, Daniel Johanson, J. Scott Judd, Brett S. Kawada, Andrea Seaton Kelton, Sara Kern, Jon N. Kerr, Marsha B. Keune, Mindy Kim (Hyo Jung), Brian D. Knox, Gregory Kogan, Amr Kotb, Ronja Krane, Joleen Kremin, Kimberly S. Krieg, Jonathan Kugel, Ellen M. Kulset, Chamara Kuruppu, Garrison LaDuca, Barbara A. Lamberton, Melvin A. Lamboy-Ruiz, Bradley Lang, Stephannie A. Larocque, Melissa P. Larson, Bradley P. Lawson, James G. Lawson, Lorraine Lee, Margarita M. Lenk, Michelle Li-Kuehne, Jonathan Liljegren, Yi-Hung Lin, Wu-Po Liu, Zishang Liu, Brandon Lock, James H. Long, Tina Loraas, Suzanne Lowensohn, Thomas R. Loy, Hakim Lyngstadaas, Wim Maas, Jason E. MacGregor, Dag Øivind Madsen, Carissa L. Malone, Maximilian Margolin, Mary E. Marshall, Rachel M. Martin, Colleen McClain Mpofu, Chris McCoy, Nicholas C. McGuigan, Dwayne N. McSwain, Michele D. Meckfessel, Mark J. Mellon, Olivia S. Melton, Julie M. Mercado, Steven Mitsuda, Kennedy Modugu, Stephen Moehrle, Amirali Moeini Chaghervand, Kevin Moffitt, Joon Seok Moon, Brigitte Muehlmann, Johnna Murray, Emmanuel S. Mwaungulu, Noah Myers, J. Conrad Naegle, Jr., Martin J. Ndicu, Aaron S. Nelson, Anh L. Nguyen, Thomas Niederkofler, Ehsan Nikbakht, Ann D. O’Brien, Kehinde M. Ogunade, Daniel O’Leary, Mitchell J. Oler, Derek K. Oler, Kari Joseph Olsen, John I. Otalor, Kyle W. Outlaw, Michael. E. Ozlanski, Jenny Parlier, Jeffrey S. Paterson, Christopher A. Pearson, Michael J. Petersen, Steven T. Petra, Matthew D. Pickard, Jeffrey Pickerd, Robert Pinsker, Catherine Plante, James M. Plečnik, Richard A. Price, III, Linda A. Quick, Jana Raedy, Robyn Raschke, Julie Ravenscraft, Vernon Richardson, Brett A. Rixom, John F. Robertson, Iyad Rock, Miles A. Romney, Andrea Rozario, Michael F. Ruff, Kathleen Rupley, Ali Saeedi, Aaron Saiewitz, Leigh W. Salzsieder, Sayan Sarkar, Michael Saulls, Tialei A. Scanlan, Tammie J. Schaefer, Daniel Schaupp, Gary P. Schneider, Andreas Seebeck, R. Drew Sellers, Samantha C. Seto, Romi-Lee Sevel, Yuxin Shan, Matthew G. Sherwood, Maggie Singorahardjo, Hanna Kristin Skaftadottir, Justyna Skomra, Jason L. Smith, Dallin O. Smith, James Smith, Mason C. Snow, Ryan D. Sommerfeldt, Kate B. Sorensen, Trevor L. Sorensen, Andrew C. Spieler, Matthew A. Stallings, Lesya Stallings, Alan Stancill, Jonathan D. Stanley, Chad M. Stefaniak, Nathaniel M. Stephens, Bryan W. Stewart, Theophanis C. Stratopoulos, Daniel A. Street, Meena Subedi, Scott L. Summers, Charlotte H. Sundkvist, Christina Synn, Amanuel Tadesse, Gregory P. Tapis, Kerri Tassin, Samantha Taylor, Mary Teal, Ryan Teeter, Meredith Tharapos, Jochen C. Theis, Jack Thomas, Kristen S. Thompson, Todd A. Thornock, Wendy Tietz, Anthony M. Travalent, Brad S. Trinkle, J. Mike Truelson, Michael C. Turner, Brandon Vagner, Hamid Vakilzadeh, Jesse van der Geest, Victor van Pelt, Scott D. Vandervelde, Jose Vega, Sandra Vera-Muñoz, Brigham Villanueva, Nishani Edirisinghe Vincent, Martin Wagener, Stephanie Walton, Rick C. Warne, Olena V. Watanabe, David Watson, Marcia Weidenmier Watson, Jill Weber, Thomas Weirich, Ashley N. West, Amanda L. Wilford, Aaron B. Wilson, Brian Winrow, Timothy Winrow, Tasia S. Winrow, Denise Wiseman, Annie L. Witte, Bryan D. Wood, Jessica Wood, Darryl Woolley, Nicole S. Wright, Juan Wu, Xiaomei Xiong (Grazia), Dimitri Yatsenko, Courtney E. Yazzie, Glen M. Young, Chanyuan Zhang (Abigail), Aleksandra B. Zimmerman and Emily Zoet, 18 April 2023, Issues in Accounting Education.
DOI: 10.2308/ISSUES-2023-013

Never miss a breakthrough: Join the SciTechDaily newsletter.
Follow us on Google and Google News.