In this article
- Learn about the benefits and pitfalls of leveraging large language models (LLMs) like Microsoft Copilot or ChatGPT to perform research.
- Find out why prospective students should be cautious when comparing online MBA programs using ChatGPT.
As the adoption of AI becomes increasingly widespread, tools like Microsoft Copilot and ChatGPT are aiding a variety of business and academic tasks in new and exciting ways—generating ideas, summarizing data, comparing information, outlining content, and performing preliminary research, to name a few.
“For many of us, large language models (LLMs) like Microsoft Copilot or ChatGPT have replaced search engines for more complex tasks,” says Will Geoghegan, clinical professor and chair of the Kelley Direct Online MBA Program. These AI-powered tools can quickly compile data from multiple online sources and help users compare information.
For example, Geoghegan says prospective students are increasingly using ChatGPT to triangulate which online MBAs are consistently ranked among top schools. While this can be a helpful first step, he recommends exercising caution—particularly when it comes to making important personal decisions based on AI-generated information.
“AI is amazing as a starting point—but the information should always be validated,” says Geoghegan.
What are the challenges of using AI for research?
The decision-making process that an AI model uses to provide information is not always transparent, even to experts. This can become a challenge for applications where a prospective student needs to understand the rationale behind a conclusion or recommendation—such as ranking online graduate programs.
LLMs are built to always provide an answer—and that answer may be a predictive guess if the model doesn’t find the information you’re looking for. This tendency is referred to as a hallucination, when the technology provides an inaccurate response instead of saying, “I don’t know.”
“There can be plenty of issues with hallucinations and questionable rigor,” says Geoghegan.
Another challenge inherent to the technology comes from the fact that an AI model learns from the data it is trained on. If the information that feeds the model includes bias or inaccuracies, those inaccuracies or prejudices can be perpetuated.
And AI models can produce results that are difficult to replicate for a variety of reasons—the proprietary nature of the algorithm, slight changes to the data processing, or differences in hardware, for example. Reproducibility is critical for accurate research, and these issues can quickly become a challenge in business and academic contexts.
“While AI is a powerful tool, it is not infallible,” says Geoghegan. “Overreliance on AI can have real-world impacts. AI should augment human expertise, not replace it.”
Putting ChatGPT to the test: What is the best online MBA program?
To illustrate Geoghegan’s point, we asked ChatGPT about the best online MBA program. “Best” depends on several subjective and objective factors, which it is quick to point out, but the tool ultimately doesn’t provide the same 2024 rankings as the most reputable rankings sources—nor is it transparent about where the information is coming from or how decisions were made to rank schools.
In this ChatGPT example, the Kelley Direct Online MBA is ranked third by ChatGPT—when in reality, the program is currently ranked #1 by U.S. News & World Report, The Princeton Review, QS, Poets&Quants, and Fortune.
How can you verify AI-generated information?
When it comes to making real-life decisions that can have personal impacts—like which MBA programs to consider—Geoghegan recommends verifying data from an AI tool against those respected rankings sources.
These rankings bodies each have a complex methodology to evaluate online MBA programs each year. For example, U.S. News & World Report compares and ranks online MBA programs based on:
- Student engagement
- Opportunities to interact with peers and professors both online and in person
- The responsiveness of faculty and staff
- The credentials of the faculty and instructors
- The aptitudes and accomplishments of students and alumni
- The support services such as career coaching and financial aid resources
- Technology capabilities that enhance distance learning
Poets&Quants’ annual rankings compare three equally-weighted categories—admission standards, academic experience, and career outcomes—while The Princeton Review considers 18 different categories. Fortune, in contrast, exclusively looks at Fortune 1000 data to weigh where the leaders of the biggest publicly traded companies in the United States earned their MBA.
These rankings often incorporate alumni feedback—and can offer a much more reliable perspective to help prospective students make decisions.
Many online MBA programs themselves offer additional tools to help students compare programs. Kelley Direct offers an MBA comparison tool, which is designed to provide students with guidance on how to research programs based on their needs and interests.
Do prompts really make a difference?
Prompt engineering, or the instructions that you use to ask an AI model to perform a task, can make a difference in getting more accurate output, says Geoghegan. Whenever possible, use prompts that are specific and clear.
“Asking for ‘online MBA rankings’ versus something more complex and specific like ‘create a table of online MBA rankings using The Princeton Review, U.S. News & World Report, Poets&Quants, Fortune, and QS’ might yield different results, even though the annual rankings are objective,” says Geoghegan.
Geoghegan also points out that the tool matters—not just the prompt. The instructions “can you create a table of online MBA rankings” with some specific sources returns incorrect information from ChatGPT.
For this reason, Geoghegan encourages students to consider Microsoft Copilot over other AI tools because it tends to be more reliable.
“Copilot will link to the source for the information (like the rankings bodies you’re asking it to pull from), so you have the opportunity to do the more in-depth, rigorous validation of the information yourself,” says Geoghegan.
The bottom line
AI can serve as a powerful tool for prospective MBA students who want to quickly aggregate and organize the information that’s available online. But the best way to make confident decisions is to conduct your own additional personal research to validate AI-generated content.