Daniel Mercer
Artificial intelligence is increasingly integrated into hiring processes, raising concerns among critics about the potential for inherent biases embedded within their algorithms. Recent research indicates that even in scenarios devoid of pre-existing biases, AI models can inadvertently develop new social biases.
A groundbreaking study conducted by teams from Princeton University and the University of Chicago investigated how large language models (LLMs) performed in a hiring simulation previously tested with human participants. In this simulation, participants were tasked with assigning candidates to specific roles and subsequently provided feedback on the success of their hiring decisions. All candidates were equally qualified, yet each belonged to one of four fictitious ethnic groups: Tufa, Aima, Reku, or Weki. Human participants tended to develop biases based on the feedback they received. For instance, if a participant hired a Tufa as a doctor and received unfavorable feedback, they were unlikely to consider another Tufa for the same position in the future. These biases persisted long after the simulation concluded. When LLMs were tasked with the same simulation, the results indicated a significantly higher rate of bias.
“LLMs can spontaneously develop novel social biases about artificial demographic groups even when no inherent differences exist,” the researchers stated in their findings. “These results demonstrate that LLMs are not just passive reflections of human biases but can actively generate new ones through experience, raising pressing concerns about their societal impact over time.”
This issue centers around a decision-making concept known as explore-exploit tradeoffs. This principle reflects the daily decision-making process where individuals must choose between trying something new—an exploration that can provide valuable insights but carries a risk of failure—or sticking with familiar choices that have proven successful in the past. When the stakes are high, people typically prefer to rely on established options (exploitation) rather than experiment with the unknown (exploration). The researchers note that artificial intelligence systems, driven by reward-maximizing behaviors, are less inclined to explore, creating conditions conducive to the emergence of stereotypes.
The research examined 15 different models from leading AI providers, including OpenAI, Anthropic, DeepSeek, Meta, Google, and Alibaba. Among these, OpenAI’s o3 reasoning model demonstrated the most pronounced stratification of the fictitious applicants. The study found that newer, larger models equipped with advanced reasoning capabilities yielded more biased outcomes.
“One straightforward explanation is that superior models make more precise inferences based on past results: instead of making random selections, a stronger LLM may favor candidates from a demographic if previous assignments of similar roles were successful,” the researchers explained. “However, this seemingly logical inclination can be counterproductive, as it risks diminishing exploration and inadvertently marginalizing certain social groups.”
A recent survey by ManPower Group reveals that over 90% of companies now utilize AI in their talent acquisition processes. As AI-driven hiring software increasingly automates recruitment, job seekers are expressing concerns about the unintended consequences that may deprive them of genuine opportunities. Workday, a prominent provider of human capital management software, is currently facing a class-action lawsuit asserting that its AI-powered hiring tools are discriminatory. The tendency of AI to prioritize past performance has also been linked to claims of discrimination in other workplace scenarios, such as a lawsuit against Meta, where employees alleged that AI-based layoff decisions were biased against individuals with disabilities or those requiring protected medical or family leave.
The ramifications of these findings extend beyond the workplace. AI systems have been previously criticized for producing biased outcomes in various applications, affecting real individuals, from healthcare decisions to tenant screening processes in housing.
The researchers emphasize that an LLM’s ability to quickly identify patterns and its inclination to generalize are crucial for learning new tasks without extensive data. these same characteristics also pose risks in practical applications.
“The challenge lies in designing strategies that selectively deter harmful pattern-matching while maintaining the beneficial forms of abstraction that enhance the power of LLMs,” the researchers concluded. “Achieving this equilibrium may be complex, but it is essential for fostering equitable and socially responsible AI systems.”
