Below is the full transcript of Episode 6 of My Robot Teacher (lightly edited for clarity and concision; filler words and false starts have been removed).
Guest:
Safiya Umoja Noble: David O. Sears Presidential Endowed Chair of Social Sciences and Professor at the University of California, Los Angeles (UCLA); Founder of the UCLA Center on Resilience and Digital Justice
Also available on: Apple / Spotify
CHAPTER 1: INTRODUCTION [00:00 - 1:57]
[00:00] SAFIYA NOBLE: We have so many guardrails that I think people don’t even think about in terms of how we come to trust different kinds of human knowledge. And those things are not available to me, and they’re certainly not legible within a large language model.
[00:09] TAIYO INOUE: Welcome back to My Robot Teacher. I’m Taiyo Inoue.
[00:23] SARAH SENK: And I’m Sarah Senk. Today we’re thrilled to be talking with Safiya Noble. She’s the David O. Sears Presidential Endowed Chair of Social Science at UCLA, the Director of UCLA’s Center on Resilience and Digital Justice, and she joins us in this episode to unpack how ostensibly neutral AI systems encode harm; why interdisciplinarity is hard but essential; and what a public interest technology ecosystem could look like.
[00:46] TAIYO: As we all know, generative AI continues to enter our classrooms and our campuses. So we’ve been thinking a lot about Safiya’s seminal work, Algorithms of Oppression, which destroyed the idea that algorithms have a kind of objectivity or neutrality.
[1:01] SARAH: Yeah, it was a highly influential book for challenging Google search as a sort of authoritative index of human knowledge on the internet. And it’s really got us thinking lately about what the implications are now that AI systems are shaping what students find online and potentially how they even imagine knowledge.
[1:16] TAIYO: But before we get into it, we want to take a moment to ask you for a quick favor. Help us grow this community and continue these conversations. Subscribe, leave a review. Send this episode to your friends and colleagues. Get the algorithm to notice us, please.
[1:36] SARAH: I can’t believe we’re about to trash talk algorithms for like an hour, but it’s like, please help us, right! That algorithm!
[1:44] TAIYO: No, no, no, no. We’re not trash talking algorithms! I love algorithms!
[1:48] SARAH: Sure you do, Taiyo; we all do.
[1:49] TAIYO: Anyway, subscribe! Leave a comment.
[1:53] SARAH: Now, onto the conversation.
CHAPTER 2: [1:58-29:34]
[1:58] SARAH: Can you start by telling us a bit about how you came to found the Center on Resilience and Digital Justice?
[2:04] SAFIYA NOBLE: In 2020, after the uprisings for George Floyd, I just felt like one of the things that I could offer is a sharper analysis and research and support to people who were working in a variety of different kinds of organizations - from governments and policymakers to community organizers and people kind of on the front lines of dealing with where the rubber meets the road in terms of a variety of different kinds of dangerous technologies in our society. We co-teach each other about what’s happening, and we’re also able to bring to bear a lot of expertise and refined thinking because of so many scholars around the world in Global North, the Global South, and share ideas about how to be resilient in the face of harmful, dangerous technologies in our society. So that’s a great organization and I really care about it deeply.
[3:07] SARAH: It’s, I think, an excellent model for something Taiyo and I have talked about before for the latent super intelligence of higher education and having people from a variety of different fields come together with a core problem that they’re working on from a number of different perspectives and then engaging… to actually have an influence in policy. And, um, so it seems like it, it really is bridging the academic - I hate to say real world, but the academic real world divide.
[3:30] SAFIYA NOBLE: I don’t know, nothing’s more real than the things we feel in academia, especially now with colleges and universities under attack in the United States, for sure. But you’re right, I mean, we want evidence-based policy. We want the best knowledge that we can bring to bear from scientists and social scientists to make its way into improving society for everyone. This work is really about that. And I feel, you know, very proud also to make a space for people who want to do that kind of work and really don’t know how to do it. I mean, I think we have to build the centers of expertise and also honor and respect the expertise of people outside of academia, too. And so we really do that in practice as well. And that is powerfully instructive and I think a great model for higher ed, too.
[4:24] TAIYO: Yeah. I feel like Sarah and I really have come to value interdisciplinarity andsee the power of it to solve some of the more pernicious problems in the world. I mean, I’m a math person, and Sarah is a humanist, a lit professor. And so our paths don’t normally cross, right? But for whatever reason, we’re here now. And I think we’re realizing that these kinds of collaborations need to happen way more often than they are, but that there are structures within higher education and academia, which keep us within our disciplinary silos. But it seems like the work that you’re doing is really trailblazing like this kind of interdisciplinary path, particularly with these research institutes, so it sounds fantastic to me.
[5:07] SAFIYA NOBLE: Yeah, thank you, I will say that interdisciplinarity is very difficult, you know, that kind of work is difficult. I have always told my students, and this was told to me, that when you want to be interdisciplinary, you have to be conversant with a lot of fields who may not be conversant back with you. And so I learned that in my own graduate study where I was trying to bring black feminism and, you know, sociological theories about society and power, race, gender, class, all of these things to bear in looking at technical systems in an information school where I went to grad school at the University of Illinois Urbana-Champaign and that was, you know, anomalous. I mean, people, I can remember saying things like, “Well, you know, I want to use black feminism to help us understand technology better.” And people saying to me, “What is black feminism? Who’s ever even heard of that?” And I was like, “oh, like a whole bunch of people across campus.”
[6:06] TAIYO: Yeah.
[6:07] SAFIYA NOBLE: You know, it’s like a way of grounding ourselves, I think, in a lot of approaches. And I love that you say, you know, even math - math has traditionally been a liberal art. It’s been perceived that way. It’s a newer phenomena, I think, to have math be so discreetly pivoting more toward engineering, let’s say, or data science or these kinds of other fields, and not as much the liberal arts, but of course, how beautiful our conversations can be when we understand all of these disciplines as arts, that we are refining, [that] there are subjectivities that we can press on and investigate and we can stay curious and these things are not fixed and known forever.
[6:53] SARAH: Totally.
[6:54] TAIYO: It’s been actually incredibly rewarding to get to know Sarah and talk with her so deeply about all of the different, like, really deep ideas that are in both of our fields and finding, like, there’s way more commonality, it turns out, than you would initially expect. Like, we just, there’s just never a really good opportunity, it feels like, it’s just not structurally built in to have these kinds of conversations. But when they do happen, not always, but like sometimes really amazing things can come out of it. So, yeah, that’s, I’m just singing the praises of interdisciplinary.
[7:30] SAFIYA NOBLE: Yes. And I love that you’re having this podcast to model it.
[7:32] TAIYO: Exactly. That’s what we’re trying to do. Absolutely.
[7:36] SARAH: Just following the flow of this particular line of thought, I think when we chatted, I don’t know if Taiyo and I made it super clear that we use LLMs a lot. So while we’re really interested in critiques, we - like at a personal level - love using them. And one of the reasons that I think we both got so excited about them is because we found that they were kind of translation tools. So you mentioned having to be, at least have some kind of fluency in another discipline in order to have interdisciplinarity. And we were talking about how we found this - you know, I even caught myself doing it when you were saying, “Oh, when somebody asked, what’s black feminism? Is that a thing?” And I kind of caught myself rolling my eyes, like, “How could somebody not know that’s a thing?” And one of the things we were thinking was: the things that are important to us, right - like, I can’t imagine other people who had no idea what that is. And then I probably have a little bit of disdain or disrespect, like “How do you not know what that is?” Right? And then that creeps into a human relationship in a way that it doesn’t with an LLM. And so what we would find is that Taiyo would ask me something and maybe if it was, you know, just me and him, we didn’t know each other. I’d be like, “Ugh, this math guy asking me about critical theory. Right?” Like, can’t he look it up? But instead I can say, “Hey, how do you explain this concept of the conditions of possibility? Dah, dah, dah da to a mathematician.” And then I’d be like, Taiyo, what’s your area of expertise? And he’d say, topographic algebra something…
[9:00] TAIYO: Oh my God. Did you just say “topographic?” Okay that’s very upsetting, Sarah.
[9:07] SARAH: [laughs] Clearly, I need our robot teacher to teach me that.
[9:10] SAFIYA NOBLE: That’s okay. My husband does that to me too. He’s like, she’s, like, in information, some-endology-some-something. I don’t know.
[9:17] TAIYO: [laughs]
[9:18] SARAH [laughs] Exactly. So we used it to, to, like, diffuse those moments and instead be like, “No, this is what I do. And here’s how I can tailor it immediately to exactly this audience.” And it’s like, “Ohhhh!” Clearly we need to do it again, Taiyo, with your field for me. But I feel like you’ve really gotten a lot of concepts that were new.
[9:36] TAIYO: Oh, Absolutely!
[9:37] SARAH: And I didn’t have to do any emotional labor at all to explain to you because I outsourced that emotional labor to ChatGPT to explain it to somebody who is totally new to it. And so that was the thing that we saw as having massive potential for collaboration in higher education. But I am aware that we are, I think, a little bit potentially, like, too utopian about it. And so that’s why we were especially excited to talk to you about all the ways that we need to be thinking much more critically about this.
[10:03] SAFIYA NOBLE: Yeah. You know, I remember when ChatGPT 3 came out and my colleagues and I immediately started, you know, kicking the tires on it, like within a day. I would type in questions - because I feel like even asking it is a little too anthropomorphic. And I would ask about things that I know a lot about so that I could see, like, is this reliable? And of course, you know, getting back made up citations. I especially love when the citations will be like, you know, multiple authors and they’re kind of like those people’s names that are in the field, but oh, those people don’t actually like each other. Like, that’s just how hilarious those kinds of things were. And you know, I would use that as an example with my own students to say, well, how would you know if you didn’t have a PhD in this area? If you weren’t asking very, very specific questions in a domain you truly know, how would you know that what you get back is reliable or trustworthy?
[11:03] SARAH: Yeah.
[11:04] SAFIYA NOBLE: You know, we have different kinds of models or standards for supporting the veracity of knowledge. We have, you know, we have scholarly publishing houses. We, you know, we have peer review. We check each other’s work again and again and again at a field level, right? We have librarians who check the legitimacy of a publisher or a press. You know, we have so many guardrails that I think people don’t even think about in terms of how we come to trust different kinds of human knowledge, and those things are not available to me, and they’re certainly not legible within a large language model. So I personally don’t want to have all the subreddits stacked up against all the evidence-based research, peer-reviewed research, training a model, and then you know, like you get what you get. And if you don’t know, you don’t know, and that’s one of the things that I think I feel the most cautious about in the domain of education - for using these and I will just say, you know, they were, these chatbots were built for Fortune 500 companies, right? Those were like the imagined user of a chatbot like ChatGPT and already, I mean, there was just a study that came out a couple weeks ago that, um, showed that corporate America is rejecting large language models because it’s creating so much more work for them because so many errors and incorrect information are coming back that they’re having to increase like their workforce by 30%, just to check the mistakes of the LLM. And so I think, well, wow, if these things were designed for them, I don’t know, American Airlines or whatever, Amex, I don’t know who it was designed for, but like these kinds of companies, if they’re not good enough for them, why is education the market? Then like, next up, let’s try it on the kids!
[13:00] TAIYO & SARAH: [laughter]
[13:00] SAFIYA NOBLE: And the professors and the teachers, and we know the answer to that, but you know that those are the things that I can’t not know and think about.
[13:10] SARAH: Mmmm. Mmmhmm. What is the answer to that? You know the answer to that, but maybe explain the answer to that through what Algorithms of Oppression is about.
[13:17] SAFIYA NOBLE: Yeah so back in 2011, I was thinking about the politics of large digital media platforms.
[00:13:32] Safiya: Yeah, so back in 2011, I was thinking about, kind of the politics of large digital media platforms, and I was thinking about a lot of different kinds of systems that were embedded with values, but at the time it was very difficult to talk about that because there were only a handful of scholars who were writing about like the politics of technology. Uh, I mean, there’s been a long history in science and technology studies about different kinds of, let’s say, analog technologies. But you know, now we had this kind of boom in internet based platforms and even the internet itself was starting to become like a series of walled gardens where people might, in some part of the world get on Facebook and that was the internet. That was their experience. Their entire experience was kinda like inside that ecosystem, and they had no way to kind of move me beyond the walls of Facebook. And at that time as that was happening and people were starting to write about that, this incredible book came out by Siva Vaidhyanathan called The Globalization of Everything and Why We Should Worry. And I loved that book and I thought he was such an amazing scholar - media studies scholar. And I just really look - still do - look up to him. And he was writing about what does it mean when we put all of our eggs, so to speak, around knowledge into these private ad tech companies, like, what, what might the stakes of that be? And it was a great book. And of course for anyone who is like a sociologist, which I was, you know, that it’s going to be differential for different kinds of people too beyond that. So I thought, oh, well, I’m just going to kind of do a series of studies and look and see, you know, well, if it’s bad for the public, how much worse is it for people of color? How much worse is it for women and girls of color? Right? How much, you know, what, what other things can we learn about how to nuance, let’s say, along these kind of a race, gender, ethnicity, class based issues, and sure enough, I mean, the first study that I did was - I took all of the US census categories, uh, racial and ethnic categories, and I just did, I did, I combined them, like I pluralized them, then I combined them with like boys, girls, men, women, really just to see like, what do we get when we look for these different kinds of communities online. And much like you would imagine, you know, when you looked for white girls, it was like “white girls dresses,” white, like the color white, you know, white didn’t really represent an ethnicity online and in search engines. And of course I was mostly interested in search because one, it’s just a super boring technology nobody even thinks about it twice. I loved what Siva had written and I felt like, here I was in information school with all the librarians. And the librarians were starting to massively adopt search technology, and they were bringing Google into the library, both from the book digitization project, which we of course know was training what has come to be Gemini. We knew they were trying to replicate library search to look like Google. So you’d go like starting to go to a big academic library webpages, and it was just like “Nothing to see, just a box, you know, put anything you want.” And so they were moving away from this more, um, sophisticated and nuanced and detailed way of digging through the troves of knowledge, of human knowledge that they were collecting. So I was super interested in how these big investments in human knowledge were shifting from the pressure of companies like Google and Microsoft and Yahoo, and when you searched in these ad-tech companies, you got things like pornography being the primary representation of girls and women of color. So “black girls” was almost completely, 80% of the first page was porn. Latina girls, Asian girls. Now you didn’t have to add the word sex or porn; our identities just became synonymous with porn in the ad-tech environment and of course, which was quite different than in a library. And so that really kicked off, you know, many years of studying in what ways are search engines steering us towards stereotypes, subtly remaking our reliance or our trust in ad tech versus other kinds of institutional knowledge - like a library, like a school, like a university, like your parents, I don’t know, right, your grandparents. And that that book Algorithms of Oppression basically became, a very close reading of the impact of search engines on society in harmful ways that I will tell you, um, at the time and even through till today, teachers and parents read this book and they’re just like, “I’m ill that for a decade I pushed students and I didn’t ask any critical questions. I didn’t even kind of think twice about whether we could trust what was happening in there or not.” You know, I think that book contributed ultimately to kind of normalizing a conversation that algorithms can discriminate, that these technologies are not just math. They’re not just computer science; it’s not just programming that values are embedded into programming choices and these things are consequential.
[18:57] SARAH: Absolutely.
[18:58] TAIYO: One of the things that I really admired about it was in one of the late chapters, how you talked about how things that can have the appearance of neutrality, because they’re a product of so many human decisions, can oftentimes be biased. And the example that will stick with me for the rest of my life that I took from your book was about the Library of Congress subject headings. I found that part of your book just really, really fascinating and the historical, the way that these things have evolved through time. And so there’s this idea of bias - and bias is a word that is technical jargon in statistics, but it has a certain connotation also for the general public, and there’s this idea that bias is bad, and I wonder how you would respond to that.
[19:45] SAFIYA NOBLE: Yeah, it’s so interesting because when I was writing that search engines could be biased, I was trying to be explicit about bias toward what - not with the assumption that any technology or any human endeavor could ever be unbiased, but what is it kind of pointing toward? Because obviously we can, we can influence or bias all kinds of things toward increased civil rights, increased human rights, and increased sense of justice, right? Fairness. One of the things that I, if I could go back and rewrite parts of Algorithms, I would be much more explicit with this idea that we cannot de-bias technology. In fact, that is probably not a worthwhile endeavor because so many decisions that are of a nature of, like, kind of impacting humanity or impacting the environment or kind of impacting the world are contextual. They are - how we see a situation or how we frame something in the United States is quite different than how it might be in other parts of the world. So there is no kind of null state where there’s a completely unbiased technology. I do not believe so because I think these are all products of human creation. The question that is more interesting to me is what are we biased toward? What problems of the past do we continue to bring into these systems and predict into the future? How does the worst of us persist in terms of the worst, like most atrocious, inhumane values we’ve ever had - how does that linger and become so normalized that in fact it’s even hard to detect? And that’s one of the things that Algorithms of Oppression does is it shows like, oh, well the fact that we get mostly pornography when we look for women of color or black women in particular, is because of the long history of kind of sexist and racist ideology about black women that was actually used politically in service of things like denying black women human and civil rights, not affording them what other women were afforded through public policy and other kinds of choices. You know, you have to make up a stereotype about black women being hypersexual at the kind of historical moment when the transatlantic slave trade is outlawed, because the only way in which the enslaved labor force can be reproduced to the African labor force is by black women and girls having children who are born into bondage because it, the trafficking becomes illegal. So you need stereotypes, and stereotypes get invented that black women love to have a lot of babies, that black women are more sexual. Right? And to normalize the practice of human trafficking and enslavement. So people don’t know these histories and therefore they’ll just be like, “I don’t know, like black women are like kind of like sexier or do more sexy things or I don’t know, like,” you know, the kinds of stereotypes. And we in fact have a huge entertainment industry that has profited in the billions off of those kinds of images and stories. Most people don’t even know where they come from.
[22:56] SARAH: Mm-hmm.
[22:57] SAFIYA NOBLE: So it’s kind of like that. It’s like these histories and legacies are with us. They get normalized. We don’t ask questions. We don’t know. And I, I was hoping that we could peel back some of these stories so that then we could say, “Hmm, okay. That’s why no one would ever check or do a safety check on the product to see how it misrepresents people of color, or women and girls of color because we don’t even think about them as a normal part of everyday practice, business practice in Silicon Valley. So no one’s checking for that. But you can be guaranteed that they are checking to make sure that Proctor and Gamble is represented fairly. Because that’s the actual customer for a search engine, right? Or for any of these technologies or these big, you know, these big companies that pay for the ad tech.
[23:48] SARAH: Mmmhmm. I think the two things that are so nefarious about that, too, it’s, like you said, that people aren’t even aware that this is happening, that there may be people who are, who - a racist trope may have become so normalized over centuries of historical documents, pop culture, just reifying these narratives. And then even somebody on like the reinforcement, the human like reinforcement learning side is like, “Yeah, that looks fine. That looks familiar with what I’m used to.” So yeah, we just spoke to somebody who works on red-teaming and was telling us about the different strategies they used to try and make the models safer and to make them perpetuate fewer harms and how it’s an incredibly complex process because of all of the ways that you have these biases creeping in through the data that it’s trained on.
[23:30] SAFIYA NOBLE: I mean, one of the things that I call for as an intervention in the book is we should be hiring people into Silicon Valley, into tech companies who have graduate level degrees in ethnic studies, in gender studies, in social sciences, in the humanities, people who are very well trained at recognizing what they’re seeing and who just quite frankly are going to ask different questions and they’re going to test products differently and they’re going to have a deep knowledge - and this is the thing that, you know, I often say to my computer science students, I say this in the book: “You have no business designing technology for society [if] you know nothing about society.” And you think about students who go into computer science majors at all the kind of major universities from which Silicon Valley hires from. They typically have AP tested out of most of their humanities courses, right? So they don’t have any college level humanities and maybe very little social science training. The schedule is packed with the kinds of courses that they need to take. There’s not a lot of room for electives beyond the major. And of course, this means that many times it’s like, well, how in the world could you be designing products or projects for people? And all you have is your own anecdotal experiences from your own life. And you’re 19. I mean, I’m just going to tell you that’s not going to cut it. And this to me is one of the reasons why we are where we are with the kinds of dangerous technologies we have in society. No fault of their own. It’s a systematic problem.
[26:09] SARAH: You know, our recent interview with Madison Van Doren, this came up when she was talking about picturing an engineer who’s like 22 years old and fresh out of college and who’s just monomaniacally focused on whether or not they can even get their model to work, and likely they’re gonna access text that is freely available to them, and so if they don’t wanna scrape Twitter, they might be looking at books that are out of copyright, and those are going to be texts that are over a hundred years old and likely disproportionately representing white male authors. I just think it’s interesting that, you know, if you’re trying to build something and you’re focused solely on whether the product will work, maybe that focus on the implications of your biased data set might be lost on you.
[26:54] SAFIYA NOBLE: It’s true. And listen, I mean, I really, I’ve talked to so many data scientists because I work, you know, data science adjacent. And one of the things that I would say - I’m more of a data studies person than data science. But, you know, I will say that many data scientists and data analysts are like, look, I just get the data anywhere I can get it, wherever I can find free data sets or affordable data sets, then I’m going to go with them. And they trust that the social scientists who have prepared those data sets have done that accurately. And now here we are over with, you know, our friends in math and statistics who can help us understand how we, you know, sometimes we are arbitrarily looking for breaks in the data to make the categories. These categories often are an abstraction of a particular phenomena. They’re not specific all the time to people, although, you know, there’s great research from people like Professor Latanya Sweeney at Harvard who talks about huge health data sets that are aggregated and supposed to be, you know, depersonalized, can very easily be kind of disaggregated right back to the person. So you really have no privacy, even in these big data sets. But if you’re the data scientist, you’re just trying to take whatever data you can get, the more you can get. And you may not even understand the implications of what that data represents, who it represents, whether it’s a fair representation of people or not. We’ve seen this, like, let’s say, during the mortgage crisis, you know, when hedge funds were betting against Americans, hoping that they would default on their loans, right, so that they could make a lot of money off of the demise of American homeowners. And, you know, this kind of like predatory practices of looking for the profile of a person you could sell a subprime mortgage to so that you could then be sure that they default, so that then you could make a lot of money. I mean, this is like such a sick and amoral way of using data. Certainly it was profitable, but you know, I would guess that many analysts who were working on Wall Street in data may not have even known what the project was that they were working on, right? Um, certainly some did. Many did. You know, there’s also like many steps in the process of being data, making data using data and exploiting data, right, with different people in different parts of that, those equations.
CHAPTER 3: [29:35-50:56]
[29:25] TAIYO: One thing that I’ve heard said about these large language models, for example, is that they are grown and not built. Maybe you’ve heard such things before - that what these AI labs kind of do is set up the environment in which the mathematical algorithms that determine how this thing learns from the text that it ingests, how that ultimately results in the outputs that we see. There’s a contrast, I think, then, that we can make between the decisions that are made in the large language model or in the artificial intelligence space versus those that might have been made with the Google search algorithm. How does this complicate issues around algorithmic justice and around the kinds of biases that we might want to see in these AI systems?
[30:23] SAFIYA NOBLE: Yes. Well, certainly that book was written before the era of large language models and I find it really interesting, even the kinds of language and words that we use around different kinds of technologies. I think we typically think of something growing as something organic in matter, right? And, we generally see that as inevitable, healthy, something that’s positive. So that’s one way that I think the industry could talk about kind of what’s happening, because obviously there’s a ton of structured and unstructured data and companies are writing these kinds of machine learning algorithms to parse that data. And even in the era of search, when search engineers, computer scientists were asked how does the search algorithm work? You know, this would be hundreds of people who worked on search. They’d say like, we don’t really know, we’re not exactly sure. You know, they kind of had a sense of how it had grown, but, uh, there was a point at which the data analysis was kind of more difficult to compute and explain than human beings could do. Um, that’s really what the promise, I think, of large language models was - was that there’s so much information that the human brain cannot fully compute it. And so we use this kind of big compute, right, to help us look for patterns, look for anomalies, help us like, make sense of, of whatever training data we’re using to train that model. I think one of the things that is probably more accurate is. We could describe that rather than saying like, the AI kind of grows, you know, we could say it a different way, I think, you know, there are people who would, who might say, “We steal all the information, all the copy written information, all the intellectual property of anybody that we can find, and we do whatever we want with it, and we reinterpret it the way we choose, and, um, what are you gonna do about it?” I mean, there’s like a kind of a crass dimension of how the industry talks about what it’s doing and, and what it’s making versus what others who are not benefiting from that point of view would do. I mean, I find it really troublesome that large language models are talked about as inevitable as without consequence. We don’t ever talk really in the mainstream about the incredible environmental damage that comes from these models. Um, the last study I saw, uh, said that one prompt was the equivalent of pouring eight glasses of water on the ground. One prompt. We know that there are, like now probably we’re in the billions now of prompts globally, so the water that’s that that is required, the huge data centers that are required that we know are polluting the environments, these are going to into low income neighborhoods like in Memphis and Austin, and places where people are pushing back hard and saying, why is our asthma getting worse? Why does the environment smell so polluted? What is this off-gassing? People becoming very ill, huge land acquisitions happening in the global south, in South America in particular: Karen Hao writes about this beautifully in her new book, Empire of AI, that I cannot recommend enough, just like getting inside the weeds and the details of what these projects are like in the world versus like the marketing hype and speak that comes out of the company salespeople, I think is really important. It’s only fair that we look at these projects in their totality rather than just in our, you know, I don’t know, white collar, computerized occupations where these technologies are gamified and they seem fun and like, you know, clever, but what it costs us to have the fun little clever thing may not be worth it at all. And I think that’s the part that is so much more interesting to talk about and learn about, quite frankly.
[34:57] TAIYO: That’s really interesting. Yeah, I think it’s really important to talk about the costs of these kinds of technologies and actually like, I’ve heaped a lot of praise on this book, but what I feel like is one of the amazing things about this book that you published in 2018 was how it sort of, as you say, normalized the conversation around some of the negative aspects of something that might have previously been viewed as neutral or objective or something like that. I think that’s really important and it’s like amazing that an academic such as yourself was able to have such a large impact on the world through a scholarly work like this. And I think that is incredibly important to have continue having that kind of criticism, uh, leveled against these AI labs who are amassing tons of power, amassing tons of resources, and that sort of thing. I think I would push back a little bit against some of the, um, environmental impacts. Like I’m seeing lots of different estimates, wildly varying on the ecological impacts of prompting these LLMs. So I, I’m definitely in a state of uncertainty about what the true and honest accounting of the ecological impacts really are. And beyond that though, I guess I would want to. Not just take account of the risks, but ask what are the potential benefits that humanity might see as a result of the bringing about - the promulgation of this technology? Which is it safe to say, the genie’s out of the bottle right now, and so we kind of have to live with it? What I’m sort of hoping is that with criticism, uh, properly calibrated to the technology that we, academics, for example, can push the industry into outcomes that we like - better futures for humanity. What do you think about that?
[36:41] SAFIYA NOBLE: Well, I think that, uh, you know, success of these projects is in the eye of the narrator, and you know, there are a lot of ways that I am kind of attuned to, I mean, let’s say, oriented to thinking about these things. First of all, you know, I was born and raised in California. I grew up in the San Joaquin Valley. I lived in the Bay Area during the first dot com boom and bust. I live in Los Angeles now. I’m very attuned to environmental catastrophes. I just am, and so there’s no way that I can put that to the side. Let’s say, as I think about the impact of something like Silicon Valley on this state, I think about things like, what does it mean that the richest industry on planet Earth, which is right here in our home state, has for the most part, not paid taxes, offshored most of its profits and extracted, most of its, you know, opportunities from public institutions in this state as well, including the California State University system as OpenAI is one of its latest acquisitions, right? The transfer of wealth of public taxpayer dollars into the hands of Silicon Valley companies. And then I look and say like, well, and then what did we get in return? To me, it’s like I’m very grounded in the material truth of what do we really get from it. I am not influenced in any way, I don’t think, by the promises of what we’re gonna get one day, because we we’re already like a solid 50 years into Silicon Valley. So I’m gonna say that at least the last 20, the state should have been transformed by the incredible wealth that comes out of this corridor. And I guess we could throw Hollywood in there now because of, um, Hollywood mostly being tech companies. And so the question to me is, and of course in fact what we see here in LA is restaurants and caterers going out of business, costumers, all of the craft work, you know, including what happens to actors and writers and directors, right? With the kind of advent of these technologies, these AI art tools, projects. I’m looking for the evidence of how this is making us better and it’s hard to come by. That’s not even going outside of the United States and going into other places in the world where we’re talking about, you know, the Kenyan data cleaners who have to look at the most horrific kinds of deeply, psychologically, emotionally traumatizing material in order to clean data, to train a large language model. I mean, we can go around the world and look at people who do the worst, most damaging kinds of work. But even if you don’t empathize with people who aren’t in your front yard, you should ask yourselves, well, are these technologies going to be so great that they raise the wages of teachers? Are we gonna hire more professors and teaching assistants? Are we gonna educate more people with these things because we have more resources? Because, look what’s here. I mean, I don’t know. I’m not seeing the evidence of it. So I guess it’s kind of like what is then who is positively benefiting, and that feels like a much smaller group of people relative to the whole,
[40:20] TAIYO: Hmm. Interesting. Yeah.
[40:22] SAFIYA NOBLE: Um, I knew it’s not the answer you’re…
[40:23] TAIYO: No! This is the answer we want, uh, for sure!
[40:29] SARAH: Taiyo wants you to roast him.
[40:31] TAIYO: Absolutely. I, I’m always telling, I’m telling everybody: I, I want to be, I wanna be shot down. I want to be burned. Yeah, tell me I roast me, please. Um, I will say like, I think one of the really most conspicuous examples of like a great success, uh, that’s come out of the AI labs recently is the Alpha Fold project, which won Google and in particular like Demis Hassabis and some other related, uh, scientists, the um, Nobel Prize. And they were able to do this work of folding 200 million amino acid sequences to figure out the structure of the catalog of proteins that we know exist in the biological world. And this to me is a mind blowing, um, achievement by these AI labs, which I think is representative of the kind of potential, not fully realized - I grant you that - and not fully realize materially in the world yet that I’m hoping because I am trying to be optimistic about all of this will impact education. And that’s really what this podcast is about, is trying to think about how AI will interact with education. One thing that I want to note though that’s really great is that Sarah and I are CSU professors and you’re a CSU graduate, which is amazing.
[41:52] SAFIYA NOBLE: Absolutely. Yes. You know, I got my bachelor’s degree in sociology from Fresno State, and I got a master’s in sociology from San Jose State. And I love the CSU so much. I mean, I would not be the person that I am without the incredible faculty and students who come through the California State University system. Absolutely world class. So much love and care in that system for students and for the public and, um, great teachers. I mean, I got all my critical thinking skills, truly from the CSU I think. I hear what you’re saying. Listen, I’m gonna, I’m gonna concede that there might be some ways to think about the application of these technologies, but the problem that I just, I still struggle with is that: how will those things map then to healthcare for everyone? Because it’s one thing to map a disease state, right, and 3D model proteins. But if the incredible investment that has come from the federal government vis-a-vis NSF and NIH, and all of the federal money that we know has gone into these companies and these researchers, if it doesn’t translate to extending, you know, our lives, giving us healthcare, giving us better quality food, air, shelter. Like, it’s just hard to understand. And I, I relate to these things like these, you know, like Manhattan project kinds of, um, endeavors where it’s like, well, we do this incredible science for the love of science. And we don’t take any responsibility for the translation to the public good. And, so now, yeah, we have, um, nuclear weapons and we have to be stressed out all the time about, um, how we answer to our children about the proliferation of nuclear weapons, which are actually things that kids worry about and ask about often - at least in my house. I guess I wonder again, you know, doing science for the sake of science is phenomenal. And we need universities and labs to be able to do basic science, and I really believe in that of all types. But we also need to have that work have a positive relationship to society and definitely that’s why I struggle sometimes with these projects.
[44:15] SARAH: Yeah. I guess I wonder if the idea of large scale technological projects that really make the world a better place a world a better place without deleterious effects on invisible labor forces is really possible and part of it is if it’s being driven by organizations whose primary objective is maximizing profit, that just seems to me potentially at odds with public interest, and I’m wondering what would it look like to have public interest be the main focus. Taiyo and I were talking about this recently, thinking a bit about counterfactual history, like what would a public interest social media company have looked like 20 years ago? Like, “Use this for 30 minutes now go turn it off, and like walk outside or talk with another human face to face?”
[45:00] SAFIYA NOBLE: I love that. I’ll take that. I’ll have what she’s having. Well, you know, at the end of the book I do talk about public interest search as just, again, a way to experiment with interfaces and who gets to kind of organize knowledge. I mean, I saw that as fundamentally an extraordinary project for librarians who are already trained with a kind of a professional standards and an ethos of care about information and knowledge, availability for the public. And that, I believe, would look very different just the same way that, you know, public media looks very different than, you know, corporate media. Right. So we would have different logics. If the primary logic wasn’t that we have to make money at all costs, then we would have different kinds of projects. And there’s really a burgeoning movement that is, is kind of called this public interest technology movement. And there are many campuses around the country who are part of a university network of public interest technologists. And I actually just was a co-researcher with my colleague Jessica Peak, from the UCLA law school who did a great study for the Public Interest Technology Network and our center and that report is gonna be released very soon here at the end of 2025. And one of the things we discovered was that when you, when we surveyed undergraduates, they were very interested in public interest technology careers, but they couldn’t figure out the pathway. They didn’t know what kinds of courses, they couldn’t get degrees, they didn’t know how to marry these kinds of conversations about like the social, political, ethical impact of technology on society. They were very interested in working in nonprofits, in government, in schools, in all kinds of places that weren’t just Silicon Valley. And they really didn’t want to do that at like one-tenth of what you would make if you went to go work for Meta. I think that signals to us that we have a generation of undergraduates for sure right now who are very interested in these conversations. They’re very interested in pathways. I think we have an enormous opportunity in higher education to make those pathways and people will innovate different kinds of products and projects. If you marry that with increasing pressure on philanthropy and funders and investors to not just invest in what’s available to them, which is the Palantirs of the world, right, like the ultimate surveillance tech, but that students are able to launch different kinds of pro-social, pro-rights, pro-privacy kinds of technologies, and there are companies they can go work in and they’re profitable because guess what? The public also wants those kinds of products. Um, so it’s like an ecosystem that I strongly believe exists. The research shows and we need to have a shift in investment and, um, educational priorities to make it happen.
[48:07] SARAH: I love this so much as we’re wrapping up because I was thinking what kinds of things can we do? You know, we have the, it was very controversial to have this CSU-OpenAI partnership, though the majority of faculty are against it and are not using it, and we’ve been really interested in thinking about how can we use access to the tools we have now to maybe start thinking about those, building those kinds of centers and going back to the collaborative and interdisciplinary potential to translate between our disciplines and figure out new ways with that expert in the room. Um, and one of the things that we always tell faculty is when you’re using it for the first time, grill it in your discipline. So just echoing what you said earlier that the best way to see what the strengths and weaknesses are is to engage with something that you really, really know about and that you can push back on. And the idea, I, I think it’s, I’m optimistic about the possibility of having… It seemed now that there’s such a, a widespread interest, maybe I’m just in a bubble, but a lot of people are concerned about the impacts of AI on people’s brains, people’s souls, right, people’s attention spans. They’ve seen what happened with social media, and I think what we’re missing is a really concrete kind of formula for like, what can you do if you really care about, uh, about the public interest? So maybe that’s what we need is like CSU Centers for Public Interest Technologies.
[49:26] SAFIYA NOBLE: We do.
[49:28] SARAH: I know there are some developing, right? Every campus should have one.
[49:30] SAFIYA NOBLE: Every campus should have one. We should have degrees in data and society, or kind of critical internet studies. We should normalize these kinds of interdisciplinary approaches. We’re so much stronger and smarter together. That’s what the LLM is promising. But I guarantee you where it’s really happening is with people. And that is like everywhere that we can work to break down barriers, um, as educators and help meet the students in their curiosities and, you know, I mean, the one thing we know about a large language model is it can only be trained on what has already happened. So think of like the thing that makes us human is our creativity, and all the things that are yet to come, and who knows what we will create, what we will be inspired to do, how we will change. And you know, we all are changing like at a cellular level and beyond on a daily basis. And that is like our superpower. These models will never be able to do that, I don’t believe. And so we need to keep ourselves grounded in what is still possible. That’s the thing that to me, is gonna be the, um, the thing that reminds us that the machines are not smarter than us.
CHAPTER 4 [50:57-55:42]
[50:57] SARAH: I love that note we ended on the, you know, deep humanity of our creativity and change as our superpower and adaptation. And I think that is why I fall on more of the optimistic side of the AI space in education. But I find that in my own teaching, that I’m using LLMs in ways that actually do break down barriers between people that I work with and the students that I teach. And at the same time, you know, I’m reflecting a lot on what Safiya talked about during our interview. And even as I’m celebrating the breaking down barriers part, I recognize it’s super important to focus, too, on the way that these systems are built upon a system of invisible labor and extractive supply chains. And so, you know, she mentioned the people labeling toxic content, who are deeply traumatized, data annotators who are underpaid. And you can think about like the cobalt and lithium miners: people have died so that we can have these iPhones on which we are recording this podcast. And we should take that really, really seriously. None of this, I want to say, is new with AI, though, that I think it is the same political economy, perhaps, but it’s, I think that is why it belongs in the classroom. Like the lesson for me here was not, we shouldn’t be using technologies, but rather it’s: “We have to be developing theories and responses to their deleterious impacts.” And so I think that means humanists with engineers, labor scholars with computer scientists, you know, people who make policy working and informing product teams, and that advocating for things, like advocacy for transparent supply chains and worker protections, energy and water standards. Like there is no reason that that data center in Memphis has to be as polluting as it is. There are other models elsewhere in the world that are not that destructive and aren’t burping out a bunch of methane. I think that really what we should be focusing on right now is developing curricula that teach students how to build and how to care simultaneously so that we are very vigilant about this future that we’re innovating isn’t just a set of like organized harms. I don’t know, what do you think though?
[53:12] TAIYO: Oh, yeah, I also loved the interview, and it was such a privilege and honor to be able to talk with Safiya because her work in particular proves that academic critique can really reshape the world. So, for example, Algorithms of Oppression really came out during a formative period for search technology, and I think it genuinely changed the conversation. And not just in universities, by the way, but within the broader public. And that to me is what effective critique looks like. It names the problem, yes, but it also influences the people that have the power to do something about it. We’re in a formative moment right now with AI, and to me, AI is the most important technological development ever. And to someone who studies hyperbolic geometry…
[54:00] SARAH: Oh you’re being pretty hyperbolic right now!
[54:02] TAIYO: Okay, yeah, guilty. But I really believe it, sincerely.
[54:07] SARAH: I know you do.
[54:08] TAIYO: I think of the future as being hyperbolic, in the sense that small deflections early can compound into radically different futures. And we’re living in those early moments right now. The choices we make or refuse to make will determine trajectories that become exponentially harder to change once they’re set. So ignoring the problem, staying on the sidelines because we’re worried about corporate capture or we’re uncertain about some of the downside risks of the technology, that’s neglecting the students who need us most. In particular, I think we need to be working and we need to be building. We need to be creating a kind of research infrastructure for faculty. I think we need to think about public interest technology and take that really seriously within academia. We need our work to be public facing. Because critique is necessary, but it’s not sufficient. We have to follow Safiya’s lead and realize the power that we have, and not cow ourselves into inaction.
[55:11] SARAH: Thanks for listening. My Robot Teacher is hosted by me, Sarah Senk…
TAIYO: …and me, Taiyo Inoue, and it’s produced by Edit Audio.
SARAH: Special thanks to the California Education Learning Lab for sponsoring this podcast. If this episode got you thinking, please pass it on. Share it with a colleague, Dean, or that faculty listserv where people won’t stop talking about AI.
TAIYO: See you next time.