Machines & Medicine - EP1: Clinical Decision Making

The Machines and Medicine podcast discusses ethical dilemmas in technology and healthcare, focusing on the role of doctors, patients, and programmers in clinical decision-making. An example of IBM Watson for Oncology's failure shows the complexity of assigning blame for AI errors. The discussion includes ethical considerations, automation bias, and AI black boxes. Guests Ahmed and Saqib discuss the importance of explainability in AI tools for healthcare and suggest solutions like stricter regulations and educating doctors on AI. The podcast raises concerns about patient consent, accountability, and the impact of AI on healthcare. Hi there, welcome to the Machines and Medicine podcast, where we discuss the biggest current ethical dilemmas of technology and healthcare. I'm your host, Nazmul Haque, and today we will be exploring the role of doctor, patient and programmer in clinical decision making. Back in 2012, IBM released IBM Watson for Oncology, an AI tool built to provide evidence-based treatment options for cancer patients. Although it did show some promise initially, it received a ton of backlash for generating incorrect treatment options and being trained on hypothetical rather than real patient data. It was eventually scrapped in 2021. This leads us to the crux of today's issue. If an AI model does make an error in clinical judgment, for example, by missing a clinical diagnosis, who is actually to blame? Is it the doctor, the hospital, the software engineer, or perhaps even the government? To help answer this question, I'm joined by Ahmed Abdullah, a UCL medical student and founder of Claro, an AI-based startup. Ahmed, how are you doing today? Thank you for having me, Nazmul. I'm doing well, how are you? I'm doing well, thanks for asking. So to jump straight in, who do you feel should be to blame if an AI tool makes a mistake which is acted upon? I believe that in this question, there is no single party responsible for making a mistake. I believe the software designer and their team are responsible in ensuring an AI-based tool follows correct clinical guidelines. I believe they should employ a doctor to do penetration tests and to stress test the AI-based tool to see if it makes a mistake, or if the NMN is a mistake, for example. I believe wider scale tests and pilots should be employed by the hospital and the government before the AI-based model is deployed at scale. Thank you for that, Ahmed. If I did have to push you for an answer, who would you say is the most liable? Ultimately, I think the responsibility lies with the software developer, which was their decision to begin and start creating this AI-based tool. It's only the doctor and the hospital who joined later in the pipeline after the software was already developed. You might be interested to hear that a study was done in the US to answer this very question, and on average, more patients thought that the physician should be to blame, whereas physicians thought it is the company making the software or the healthcare organisation who should be held liable. Both groups felt that the government should be held equally liable. This issue becomes more nuanced if we break down the ethics behind it. Let us take the following hypothetical example. Option one is to have an AI-diagnostic tool, which is twice as efficient as a doctor, but for every 99 correct diagnoses, one is incorrect. Option two is to have a doctor who, despite working slower, has 100% success rate. So who do you pick? If we apply deontological ethics, this idea that certain actions are inherently wrong based on duties and rules, and also the medical pillar of non-manifestance, that doctors should do no harm, the obvious choice becomes the doctor, as the harm from the AI-diagnostic tool is inevitable, despite only being a small chance. However, if we then apply the medical pillar of beneficence, this idea that doctors should always do good, and utilitarian ethics, that this good should be for the greatest number of people, the obvious choice now switches to the AI tool. The question now becomes, does the suffering of a few justify the benefit of many? How about if this was in the context of a life-threatening disease like cancer, where misdiagnosis could be the difference between life and death? Interestingly though, while doctors and hospitals are held to such high ethical standards, the same cannot be said for the programmers building the very tools on which the hospitals run. This leads me on to another issue with AI-assisted decision making, automation bias, which relates to the human tendency to overly rely on automated systems and favour their suggestions, even if they contradict common sense or evidence. There is a growing concern that this is reducing the agency of our doctors, i.e. their ability to act freely and make independent decisions. This reduced agency could sway their clinical judgments, even beyond what they know to be true. Off the back of this, Ahmed, are you worried about the effect that AI could be having on the problem-solving skills and empathy of our future doctors? With regards to empathy, I'm not worried. I believe empathy is still built upon personal connections with in-person patients, for example, but I feel a lot of medical students outsource their thinking to AI-based tools. I've never seen this effect on myself and it's something that I'm constantly trying to battle and ensure that I can maintain my normal cognitive skill. Automation bias also poses another risk. There is a common understanding with LLMs that the quality of data in equals the quality of data out. Therefore, biased data will produce biased results. This happened with Optum, a US health service company, which developed an AI tool to predict high-risk patients for care management programmes. Unfortunately, it was found that the tool had underestimated the needs of Black patients as it used healthcare costs as a proxy for medical need. However, historically, less money is spent on Black patients due to poorer access to healthcare rather than lower healthcare need. Luckily, in this case, the problem was identified by understanding the AI support processes, but what if these processes are less clear? This brings us to our next segment, which is about AI black boxes. At this point, I want to introduce my second guest, Saqib, a UCL computer science student and AI enthusiast. Saqib, how are you doing today? I'm doing great. Thanks for having me on. Thank you for being here. Saqib, let me start by asking you this. What is your current understanding of AI black boxes, and do you feel they can be at all problematic? Yeah, it's an interesting question. Black boxes are actually quite common in computer science, and they occur when we don't understand the inner implementation of a technology, and they can definitely be problematic. AI is a statistical machine, and the outputs are not easily predictable. Can they be problematic? Yes, especially when used in very critical conditions like medicine and law. Precisely, you've hit the nail on the head. In healthcare, this can really complicate matters. As if neither patient nor doctor fully understands how an AI algorithm has reached its conclusion, is it actually possible for a patient to get fully informed consent? When developing these AI tools, programmers may focus on explainability, which outlines how well an algorithm's processes can be understood, or they may opt for performance by creating a highly complex tool which is very effective but harder to explain. Off the back of this, Saqib, given your background in computer science, do you feel programmers should prioritize explainability or performance when developing AI healthcare tools? So yeah, there's definitely a trade-off between these two aspects of AI. You tend to see that more explainability does lead to less performance, as some of the effort from the AI is going towards having to explain itself. For myself, at least, I feel as if explainability, especially in places like healthcare, does tend to be more important because there is a responsibility for the doctor to at least be able to observe the AI in its state. However, I also do tend to see explainability not always taking away from performance because explainability from the programming side allows them to iterate and improve on the outputs by understanding how the outputs are reached. This topic also ties into patient autonomy, i.e. a patient's right to make well-informed and voluntary decisions. Informed consent relates to three points of understanding, the diagnosis and what it entails, the pros and cons of treatment, and why a recommendation has been made. If an AI advisory tool exists in a black box, it becomes harder to justify why this clinical decision has been taken. Also, if the patient is not informed about the use of AI, we must consider how this shifts accountability for errors and whether this omission of information can be considered as medical negligence. In the past, it has always been the doctor and their team responsible for clinical decisions, making the issue of accountability simple. However, the increased integration of AI has blurred this line of accountability. Moreover, automation bias has revealed concerns about agency, clinical competence, and healthcare inequalities, while AI black boxes have complicated matters of patient consent. Off the back of this and to conclude, I wanted to ask both my guests if you have any viable solutions in mind to tackle any of these challenges, starting with Ahmed. Going back to my previous point about putting the responsibility of doctor-developers for income diagnosing, I believe stricter penetration testing and regulations should be in place for deploying new AI tools into the market. The NHS is already implementing this through its MHRA guidelines and classified AI tools with medical devices, and I believe this is a good step forward in determining whether AI tools should be deployed at scale in healthcare. Thank you for that, Ahmed. Now, Fakid, same question for you. So, in order to tackle explainability and observability with AI in medical use cases, I'd say that a lot can be done just by educating doctors by, in fact, giving them a course in how AI and LMMs work, for example, because these are complex topics, and if a doctor is going to use them in such critical cases, it is important that they understand how it works and how the AI reaches its conclusions. All in all, with the way that AI and its research is moving forward, I feel quite optimistic that patients will be able to feel a sense of peace of mind with how their diagnosis is reached, and that can do great things for the industry. Thank you so much to both my guests and to you for tuning in to this week's episode of Machines and Medicine. Join us next week where we will discuss advancement in facial recognition and what this means for the healthcare industry. Till next time, stay curious.