In the same way large language models and AI agents are changing how people conduct business and build software; they’re also changing the ways academics conduct research and publish papers.
With AI offering so many ways to streamline study construction, analyze data, and draft papers, the pace of publishing has been supercharged — but researchers are still developing the rules of the road.
Elena Karahanna, C. Herman and Mary Virginia Terry Distinguished Chair of Business Administration and Regents’ Professor, together with Carolina Alves de Lima Salge and Weifeng Li, associate professors of Management Information Systems, convened one of the first conferences focused on determining the best practices for using Generative AI in scholarly research.
They welcomed 110 researchers from across the United States and Canada to the inaugural Conference on Generative AI in Research Methods at the Terry Executive Education Center on Oct. 2.
“What we’re trying to do is start a conversation around what is being done, what are the risks, and what are some best practices that we can share, so we can do this more responsibly and with better outcomes”, Karahanna said.
“Any tool that makes it easy to do something will be used, but we have to make sure it’s used responsibly.”
During the conference, a panel of journal editors, moderated by MIS Department Head Jerry Kane, expressed concerns about the increase of papers they’ve received since generative AI swept through scholarly communities.
The goal was to convene researchers and explore new methods to use AI for conducting better research and publishing more impactful papers — not just more papers at a faster clip.
“We’ve been doing research using different methods for decades,” Karahanna said. “Whether it is experiments, whether it is qualitative work, whether it is surveys, econometrics or data science — the question is, how can we use generative AI to change what we’re doing in novel ways?
“We want to know what it enables us to do that was extremely difficult or almost impossible before.”
As part of the conference, Aaron Schecter, associate professor of MIS, presented a framework for incorporating AI agents into the research process to keep human researchers involved and accountable.
Other researchers presented on using agents to code vast amounts of text to find trends, patterns, and topics, or how to develop and ethically use “silicon subjects” — randomized armies of personas who could be treated as a survey population.
One topic that came up repeatedly was transparency on behalf of the researchers and the models they use.
Journal editors now expect the inclusion of a transparency statement with the submission of a paper detailing how AI was used and, if needed, copies of the prompts and agent’s chain-of-thought log in the paper’s appendices.
“And researchers need to access and understand their models’ complete train-of-thought logs for each part of the process and data analysis. They also need to know how to interact with their models to understand actions they may have taken without the researchers’ knowledge”, Salge said.
“If you’re delegating full tasks, and this agent is just autonomously doing things without your knowledge — there can be mistakes,” Salge said. “It’s fascinating, but there are a lot of risks too, and we’re just learning about them. That’s what this conference is about: what are the risks of different methods and how do we mitigate them? We’re starting that conversation.”