Biases in the use of Artificial Intelligence
Behavioral New World
November 15, 2023
Mid-month bonus
My first mid-month bonus (January 15 of this year) looked at what behavioral economics can contribute to thinking about leadership. One challenge facing today’s leaders is deciding how to best deploy Artificial Intelligence (AI) tools in their organization. AI is arguably more complex than previous technologies and there is remarkably little known about the consequences of using it. This mid-month missive looks at what behavioral economics can contribute to thinking about the best use of some of the newest AI tools. My collaborator here is J Scott Christianson, a friend and colleague, who writes about Generative AI/ChatGPT in his Prof C Substack.
Although the idea of machine or artificial intelligence (AI) has been around since before the transistor, it has recently burst into broad public awareness because of the public availability of large language models such as ChatGPT, Bard, and Claude. (“Burst” might be an understatement—ChatGPT reached 1 million users in five days.[1]) Part of the awareness is that, because of the way AI is trained, it can display biases.
A related issue that has received little attention is the biases that we, as users, bring when we encounter AI systems. Are our biases causing us not to use AI to its full potential? Might our (perhaps unconscious) biases in the use of AI lead to poor or misleading outcomes? A recent study of how radiologists used AI-generated diagnoses provides some insights.
In a National Bureau of Economic Research discussion paper,[2] four researchers ran an experiment with over 180 radiologists to see if giving them AI assistance in diagnosing chest X-rays improved their accuracy. Before proceeding with the experiment, the researchers established that the AI performed as well as board-certified radiologists (that is, at a very high level).
Given the efficacy of the AI system, you would think that providing AI diagnoses would help the radiologists’ accuracy. After all, the AI diagnoses were more accurate than nearly two-thirds of the participants. But it did not! Why? The researchers suggest that biases in the radiologists’ use of AI explain this counter-intuitive outcome. They describe two causes.
First, on average, the radiologists gave too little weight to the AI diagnosis, less than would have been optimal. Put differently, they gave too much weight to their own opinions. The researchers call this “automation neglect.” Why does automation neglect occur? The researchers suggest something they call “egocentric bias.” We are inclined to call this hubris or overconfidence (see John’s newsletter of September 2020).
Further, algorithms are viewed as less trustworthy than human predictions, likely because we don't understand how they work and view AI as a “black box.” The black box problem has become a significant issue as AI systems have grown in complexity and strategies for training AI have been developed that don’t require much human guidance, so-called “unsupervised learning.”[3]
Second, radiologists seem not to have recognized that their diagnoses were correlated with AI’s predictions. They incorrectly acted as if the two diagnoses (theirs and AI’s) were independent, but they were not—the radiologists and the AI were looking at the same data, so their diagnoses were naturally related. This oversight meant that the radiologists did not use the AI information to maximum benefit. The researchers show theoretically that this “signal dependence neglect” is a more important influence than egocentric bias.
Because of these biases, the radiologists didn’t achieve the potential benefits of combining human expertise and AI capabilities. The researchers conclude that the best system is to choose either the human or the AI to diagnose each case, but not have them work together.
The study is an excellent reminder that humans have predictable biases when using algorithms and their output. We need to be aware of these cognitive limitations when designing human-AI collaboration systems in medicine and elsewhere. The research is a cautionary tale for those looking to deploy AI in their organizations.
The image at the top was generated by the prompt: “A comic-book style image showcasing a modern coffee shop scene. At the center of the image, two professors of European descent are seated at a table, deeply engaged in a conversation with a humanoid robot. They are surrounded by high-tech educational tools indicative of a futuristic learning environment. The coffee shop is styled to reflect a contemporary ambiance with sleek furniture and advanced gadgets that could be used for academic purposes. Bright, vivid colors and sharp contrasts are used throughout to convey the vibrancy and dynamic atmosphere typical of a bustling business class setting.”
[1] https://explodingtopics.com/blog/ai-statistics
[2] https://www.nber.org/papers/w31422
[3] An entire field of research known as Explainable AI (XAI) has developed in recent years to make AI systems more understandable and interpretable by humans. XAI is being used in a variety of fields where the methods of determining an outcome are critical to know (such as healthcare, finance, and law).
 | A guest post by
|