Ethics of AI Podcast: The Opt Out Problem: Who Shapes the AI Systems Shaping Us?

The podcast discusses the opt-out problem with generative AI. Women and those in creative fields show lower adoption rates. Trust and agency are key reasons for opting out. Patricia, a tech leader, works to increase adoption, while Bumi, a literature student, avoids AI to maintain personal interpretation. Trust and agency issues impact AI adoption and system evolution. Addressing biases and rebuilding trust is crucial for inclusive AI development. The opt-out problem raises questions about who shapes AI's future. Welcome to this Ethics of AI podcast. My name is Mayher and today we are exploring the opt-out problem. Why certain groups are choosing not to use generative AI and why that matters more than you might think. Chats QBT reached 100 million users in just two months, faster than any technology in history. Today generative AI tools like Chats QBT and Copilot are being used by hundreds of millions of people in workplaces, universities and homes around the world. But not everyone is using them and the pattern of who opts out is not random. Women and people in the creative industries show less AI adoption. Research from Harvard Business School shows that women adopt generative AI tools at a 25% lower rate than men, even when access is completely equal. In one study of over 17,000 entrepreneurs given identical tools and instructions, women were still 13% less likely to engage with Chats QBT. This suggests that this issue isn't simply just access, it's something deeper. But why does opting out even matter? It matters because generative AI systems are shaped not only by the data that they're trained on, but also by the people that interact with them. Patterns of use, feedback and engagement influence how these systems evolve. In other words, who participates helps shape what AI becomes and these systems are increasingly shaping us in return. To explore the opt-out problem, I spoke to two people with very different perspectives. Patricia, a senior leader in technology who runs AI workshops designed to increase adoption of these tools, and Bumi, an English literature student at UCL who has consciously chose not to use generative AI in her work. Using three ethical concepts, algorithmic bias, agency and trust, this podcast explores why some people are choosing to step away from generative AI and what that decision might mean for technology shaping our future. Generative AI systems are trained on huge amounts of existing data. Because that data reflects the inequalities of the world that produced it, these outputs can reproduce those same patterns. This is known as algorithmic bias. Systematic errors that create unfair outcomes, often privileging some groups over others. Bias doesn't require malicious intent and it can be implicit. As psychologists Banaji and Greenwald argue, bias is often absorbed rather than deliberately designed. Researchers have focused on addressing bias training data, but far less attention has been paid to the other issue, the opt-out problem. If certain groups choose not to engage with generative AI at all, their perspectives are also missing from the systems that are shaping our world. It is important to understand why people are opting out and what we can do to address this. To better understand who isn't using AI and why, I'm honored to welcome Patricia Sousa to the podcast. After attending the AI for Good Summit in Geneva, Patricia began running AI workshops exclusively for women to build confidence. She has also spoken on AI inclusion at major organizations, including J.P. Morgan. Hi Mayra, thank you for having me in your podcast. What inspired you to start running these workshops? Back last year, I was invited to speak at an event of the Women in Treasury group here in Luxembourg, and the topic was AI. And I was a bit, not a bit, a lot surprised with the feedback from the women. Many women were not aware of the differences between AI and gen-AI, they thought they had never used AI before. Some of them say that they feel that using AI is like cheating, and that shows me there is a huge gap. So I started discussing with some organizations to have workshops only for women to have this first experiment. Have you noticed differences in the way men and women approach AI tools? And why do you think this is? You mentioned before that some women feel that using AI is cheating. Why do you think this perception exists? This comes also again from the societal biases. Women are taking way harder in terms of how they should behave, in terms of societal expectations. So women have to go by the book, follow all the rules, and that's exactly the experience most women have when they are using AI. Do you personally think AI is cheating? I would rationally say I don't think using AI is cheating, because the safety, for example, would say that using Google Maps is cheating. What do you think happens if certain groups don't adopt AI at the same rate as others? That's a very good question. AI has several ethical ones to be discussed. One is where the data is coming from. Whether this data is represented in a lower worldwide population. That's one of the aspects. We have also the usage aspect. For me to understand whether AI is representing me or not, I need to see the biases. I need to be aware of the biases, and I need to use massively, so that I also train the data. That's one of the things. We are also training the data. So if a certain group adopts AI more than others, for example, males adopt AI more, it will be made for them. Correct. Thank you so much, Patricia, for joining this podcast. Patricia describes some women opting out of AI as they view it as a risk or cheating. This demonstrates a lack of trust. Trust defined by Mayor Davis and Shulman requires ability, benevolence, and integrity. AI's ability is weakest for underrepresented groups. Its benevolence is shaped by commercial motives rather than in visual interest, and its integrity reflects training data that does not represent everyone equally. Opting out is not a coincidence. It's a rational outcome of a system that has not earned trust. And according to Davis's technology acceptance model, when trust and perceived usefulness decline, adoption declines too. Let's see this in practice. Patricia compared this to Google Maps. If everyone else uses navigation and you don't because you feel like it's cheating or you don't trust it, you will still reach the destination just slower. But there is another consequence. The system learns from the people using it. If only certain drivers use it, the routes get optimized for them and not for you. But Patricia's perspective is not universal. To explore the other side of the opt-out problem, I spoke to Bumi, a UCL English literature student who has consciously chosen not to use her end of AI. Hi, Bumi. Hello. So you study English literature, a subject that relies on interpretation and analysis of text. You've consciously chosen not to use AI tools like chat you can see in your work. Why is that? For me, literature is about interpretation and how the things I've experienced shape the way that I read a text. So I feel like if I'm using AI to generate ideas, then it feels like I'm outsourcing that thinking. Some people would argue that AI is just another tool, similar to having an editor look over your work. Why does AI feel different to you? With a human editor, they have expertise and their own understanding. And I think it's a much more of a collaborative experience. With AI, it's just acting off of patterns and data. It doesn't really feel credible to me. So to you, it's not really about accuracy. It's more about ownership and control over your work. Yeah, exactly. I think literature is so based on individual interpretation that if AI is giving me these ideas, then the interpretation doesn't really feel like mine anymore. Do you ever worry that opting out of AI may put you at a disadvantage in the future? Yeah, I do worry about that sometimes. I think at the speed that it's grown in the past couple of years, in the future, it'll be completely unavoidable. And I think it will be important to know how it works. Boomi describes the loss of agency. Anscomb defines agency as intentional action carried out for a reason. Her Boomi literary interpretation is exactly that. Personal act shaped by her experiences, judgment, and reasoning. Negative agency is a feeling of having little control over one's actions or environment. Boomi experiences a sense of negative agency when using generative AI. Interpretations it produces feels detached from the authentic human experience and interpretation that her work depends on. This links to autonomy. Christmas argues autonomy requires both competency and authenticity. A human editor may have competency through expertise and authenticity through a shared goal of improving the work. But Boomi does not see generative AI as possessing either. Decision to opt out therefore protects agency. But it also raises a broader ethical question. If people like Boomi disengage from these systems entirely, whose perspectives will shape the next generation of AI? As we approach the end of this podcast, we can see Patricia and Boomi reveal that the opt out problem is not just about whether people use AI, but by identifying who isn't and why. For Patricia, the challenge is trust. If generative AI appears biased, unreliable, or built without certain groups in mind, choosing it becomes a rational response. For Boomi, the issue is agency. Literary interpretation is an intentional act rooted in personal judgment and experience. If AI is used to generate those interpretations, the work risks losing authenticity that gives it meaning. Together, these responses reveal a cycle. Algorithmic bias weakens trust. Broken trust reduces adoption and reduced adoption limits training generative AI models. This reduces relevance and inclusions of models even further. Addressing this requires more than better data. It requires rebuilding trust through transparency and explainability. AI that shows users how decisions are made and potential limitations. It requires designing for agency, giving users genuine control over outputs instead of overriding personal identity. Systems that learn from underrepresented groups can help to account for the bias data that these models are trained on. So the opt out problem leaves us to question, if the future of AI is shaped by those who use it the most, what happens when the people who would challenge it most choose not to use it at all?