Satwik Mishra

Welcome to another episode of Trustworthy Tech Dialogues. I’m Satwik Mishra, The Executive Director for the Centre for Trustworthy Technology. As 2024 comes to a close, we find ourselves at a pivotal juncture a year where AI didn’t just advance but profoundly reshaped how we live, work, and connect. AI has redrawn the boundaries of trust, pushing us to rethink the relationship between technological progress and responsibility.

This year, AI didn’t just test its own limits but shattered societal expectations, unlock new possibilities, and inspired novel approaches to governance, collaboration and creativity. Amongst these changes one wavering call echoed from all stakeholders “Trust in Technology”, trust became the vital link connecting progress to purpose, innovation to impact, and technology to humanity. The choices we make today then determine if an AI-powered future is built not just on ambition, but on a foundation of trust that ensures lasting progress.

In this final episode of the year, we reflect on 2024’s milestones, explore the themes that defined the public discourse and discuss how trust can continue to guide AI’s evolution in the year ahead. In this episode, we’ll discuss global governance and multi-stakeholderism, the debate of Open-Source Models versus Proprietary Models, idea of Sovereign AI, the industry perspective on trustworthy AI.

The debate around deepfakes and misinformation and finally, whether we need a new social contract for trust and technology to explore these pivotal themes. I was honored to host two distinguished global voices at the forefront of trust in technology. First, we have Glen Weyl, who is the founder and research lead of the Microsoft Research Special Project, the Plural Technology Collaboratory.

He’s the founder of RadicalxChange Foundation, a leading think tank in Web3 space. The founder and chair of Plurality Institute, which coordinates a global academic network focused on developing technologies for cooperation across differences. Finally, he also serves as a Senior Advisor to GETTING-Plurality Research Network at Harvard University. Next, we have Audrey Tang, world renowed for pioneering contributions to digital freedom.

Audrey is a senior fellow at Project Liberty and recognized by Times as one of the 100 most influential people in AI. Together, Glenn and Audrey have co-authored the 2024 book “Plurality: Technology for Collaborative Diversity and Democracy”. This insightful work explores how collaborative technologies can bolster democratic values, foster meaningful recognition, and bridge social and cultural divides as we shape a shared future.

I hope you enjoy this conversation.

Satwik Mishra

Welcome Glen and Audrey for this end of year’s special Trustworthy Tech Dialogues. Thank you so much for joining us. I am very happy to be here. Good local time, everyone. So let me begin by starting off with a theme on global governance and multi-stakeholderism. This year we saw several initiatives come through, both at the national level and at the multilateral level.

We had the EU AI act come about, we had the executive order follow throughs, we had the UN launches Global Digital Compact, we had the Seoul Summit, the second summit after U.K. And finally, we also just had the inaugural convening of the network of the AI Safety Institutes.

Audrey Tang

Yeah, I was there in San Francisco.

 Satwik Mishra

Yeah. So, Audrey, let me begin with you.

What interventions in 2024 do you think best highlight mechanisms for aligning community aspirations with global AI policies right now?

Audrey Tang

Yeah, I think I’m really heartened by this idea of a safety commons. So, a set of shared open-source tools that portrays openness and safety, not as two camps, but in the same commons, so people can work on these things together.

So the tools for explainable AI, for AI for participation, the trust and safety tools in the open-source realm that makes trust and safety decentralized but actually more safe. So on and so forth each and every of those leading labs have some of those tools internally. But these are strictly non-dual-use. So open sourcing it is a generally positive thing to do and AI as convening power made it possible to have this kind of safety commons.

Satwik Mishra

Glen, let me ask you in the policy space over the year, we have always seen that there’s this policy perception that policy always lags technology development, right, like policy is always behind. What do you think is the biggest challenge in bridging this gap between policy intentions and where technology stands?

Glen Weyl

Well, I think policy being behind technology is a choice not inevitability and policy being negative and constrictive is also a choice rather than a necessity. We know that India pioneered through India Stack something that for all the hype about crypto has made much more of a difference in the way that people make payments, for more people than has anything coming out of the crypto space.

On the other hand, in, you know, the United States there’s a lot of challenges sorting out crypto policy and keeping up with core crypto innovation. So you see two different societies, one that chose to lead, one that chose to follow, and other societies like Europe that have maybe been faster in keeping up with the regulatory side. But haven’t led as much in the positive direction.

So, these are all options that we have and that at different times different countries have taken, you know, the internet was not done by a bunch of hackers. The internet was created by funding from the United States Department of Defence, given to universities in cooperation with a variety of civic and corporate initiatives. So, all of these are options, we’ve just chosen in certain places to adopt some of these options and not others.

Satwik Mishra

One of the things that we, our team discovered and through the last two weeks of preparing for this podcast is the fascinating idea of “Alignment Assemblies”, in public policy and aligning community aspirations.

Before we move on to the next section, I would love to know more. Like tell us, tell the audience about Alignment Assemblies and what is its potential to bridge that trust gap between community aspirations and what policy is trying to do.

Glen Weyl

One thing I’ll just mention is in the history of Alignment Assemblies, it’s actually quite a funny history because I was at Microsoft and a woman named Divya Siddharth was working for me, and at the same time that the AI explosion was about to happen, I was talking to Audrey about what she was going to do at the Summit for Democracies that year, and Divya was deciding to leave, Microsoft.

And it kind of all came together to create the Alignment Assemblies idea, which Audrey was deeply involved in and can and can tell you more about.

Audrey Tang

Yes. So, around this time of Summit for Democracy last year, or in 2023, we had this idea that maybe we can have AI and democracy reinforcing each other. So, using AI tools to quickly get the consensus around the boundary conditions of how the societies accept AI into the society but by domesticating it and have clear guardrails and guide rails and so on, not unconditionally.

And then, this alignment makes AI more trustworthy, which then we can use this aligned AI to generate more trust by having a facilitated more conversations and so on is a virtuous cycle. So, we used a tool called Polis, which shows the people in the AI communities like where they’re divided. In March 2023, they’re divided over this idea of whether saying AI is very risky.

Extinction level is a hype that paralyzed people or is necessary so that people sprang into action that was very divisive. But the bridging statement was that if we can just focus on information integrity, because that’s a short-term problem that’s going to affect everyone, then we have working experience, right. So, we send SMS messages to 200,000 random numbers and ask people how do you feel about the deep fakes online,

the crypto scams, some featuring celebrities and things like that. So, people shared how they felt, and many people volunteered to have a citizen assembly online and we using sortation Random Stratified Sampling chose 450 people randomly that represents the entire society as a microcosm and then went online to talk for hours and hours in 45 rooms of ten people each facilitated by an AI showing in real time the transcript of the points of agreement, encouraging people who are quiet to speak up more so that we, then in the plenary, read back of the sense making of all the 45 different rooms and their suggestions and basically set the rules for, for example,

Facebook or YouTube, or TikTok any advertisement platforms that shows anything to more than 5% of our people in those platforms they have to have digital signatures, KYC to proof that this Audrey Tang selling crypto is actually signed by Audrey Tang. And if they don’t and somebody gets scammed for $1 million then that platform if they do not have KYC will be liable for that $1 million.

And so this is a very quick way to show that actually democracy can work as quickly as emerging technologies.

Glen Weyl

Yeah, I think I think that’s a beautiful illustration, both of that and of the way that democracy can actually lead the development of technologies and approaches that then diffuse into other sectors.

Audrey Tang

Yeah so, it’s not just about, you know, putting, more on the gas or on the brakes but rather on the steering wheel so that everybody have their hands on the steering wheel so that society is not forcibly aligned by AI but the society can align together.

Satwik Mishra

That’s wonderfully put “Not more on the gas, but more on the steering wheel”.

Alignment Assemblies is absolutely fascinating. We will link it to our show notes for everyone, in the audience. Moving on to the next theme that I really wanted to get both your opinions on, and which is pretty much been in public debate all through this last year is this debate between open-source models and proprietary models and recently a report analysing the universe of open-source models and proprietary models came across the fact that the best of proprietary models and open-source models are only a year apart in terms of capabilities.

Right. So, we have open-source models really catching up. Let me start off with Glen. What do you think, what is this shrinking gap between open-source and proprietary models capabilities mean for the future of AI development and deployment?

Glen Weyl

Well, I think there’s a big question about whether the proprietary models are leading and the open-source models are following or whether they’re innovating together

and I think many of the assumptions that we have, about what open-source does for safety depend on this because, for example, if you think that the proprietary models are leading and the open-source models are following, then actually you might if you’re a safety-oriented person, want more open-source. Why is that? Because if you actually want to slow down the process, you want to make sure the proprietary models can’t capture a lot of profits

and therefore, there’s not an incentive to invest in creating them in the first place. So, I actually think a lot of the reasoning about this stuff maybe gets a little bit scrambled by not thinking about the economics clearly and that actually getting stuff quickly to open-source models is little bit like, you know, going very quickly to generics, it might actually limit the race rather than creating irresponsible deployment and on the other hand, allowing people to keep everything proprietary might actually be precisely what fuels the acceleration of the race.

Satwik Mishra

It’s very interesting, Audrey let me ask you, how can the international community balance the benefits of this open-source AI ecosystem and what has sometimes often been said about the misuse of open-source AI. Recently, the last couple of months, we’ve heard reports of open-source models being used for military applications.

Tell us and we had this discussion when we met in D.C. around orienting open-source models towards safety. How do you think that balance can come about? Well, it’s no, question that, for example, the Qwen models or Tongyi Qianwen models are, gaining ground in terms of capabilities and reasoning and things like that. So, and they’ve been doing a lot of open-source on hugging face.

Right and so I don’t think, I agree with Glen that I don’t think portraying this as somehow the proprietary models that are doing this and open-source models are doing this, follow up is helpful. Now, I think they’re having like different cultural norms, different norms, for, for example, how the training materials reflect the culture expectations and so on.

So, if, we say, okay, from the liberal, democratic order we stopped doing open-source models and from the more totalitarian, authoritarian order, they keep doing open-source models. What will happen is then for the deployment sides that needs on- premises deployments or customization or further fine tuning or remixing and so on will become much more authoritarian because that’s the material then they have to work with.

And so in a sense, I think having more open-source models that reflects a pluralistic value set and, the research like those in the Sakana AI team is doing that when you want a new deployment site instead of this huge model that you can then fine tune, you can automatically remix smaller specialty models together and then do a kind of cultural alignment, making sure that you have the local like seed bank of storytelling and morals and values and so on, and you mix those smaller models together and then you have a culturally aligned capable for that specific purpose aligned model.

Now, I think this is much better than the fully generalist model. An AI that can do kind of everything because for us, it is much easier to align to a specific social role and this is the same for AI especially agentic AIs, if you have very clear expectations. It doesn’t need actually to memorize all the different other uses on their fields and this lower models also makes explainability much easier.

So, this is again in the safety commons.

Glen Weyl

I also think, you know, a very important role that the open models play is even if, let’s say, it’s going to end up being the case that closed models dominate.

They put a lot of pressure because if the close models want to say, we’re safer, we’re reflecting social values better and that’s why you should limit the open models. Then they better build the legitimacy and the participation to ensure that that’s actually the case. The open models, without them, they, you know, the closed folks wouldn’t even be using those social arguments.

They would just be saying this is our property, right. And we’ve seen that happen before in software. Right. But once they’re with the open models, they’re competing with them. Then they need to actively ask for them to be shut down on the basis of some greater virtue that they have. And then they better prove that virtue. Right. And, you know, that hasn’t always been the case in terms of the ability of these companies to be true to promises.

They have to act in the public interest, those promises have a much better chance of being redeemed with the competition of open-source models being narrow alongside.

Audrey Tang

Right. So, while they initially compete, they can be seen as converging on this democratic socially input two models point.

Satwik Mishra

That’s wonderful, instead of competing they might converge.

So, like just before we move on to the next section, I’d love to like quick takeaways on looking ahead and doing some scenario mapping for 2025. What do you think would be, what do you think this debate would look like between open-source and proprietary models? The end of the 2025.

Audrey Tang

many, many frontier labs now have both, right. You have Gemini and then you have Gemma, and the Gemma and Gemma scope and the ecosystem of open trust and safety tools.

Safety commons, cybersecurity all this is useful for a Gemini as well, right. So, we can probably see in the new year more and more of this realization that the democracy commons, the safety commons is a place that is positive some for everyone. So, it’s worth investing in and then the proprietary models take some of those safety tools and so on so that they don’t have to reinvent the wheel internally, I think that’s bound to happen.

Satwik Mishra

Let’s move on to the next section, which was another debate through 2024 which was the idea of Sovereign AI by nations develop and govern AI systems tailored to the unique economic, cultural, social and political needs. Audrey, let’s start off on this idea of Sovereign AI.

What lessons do you think nations coming into their own with AI ecosystem right now can learn from the early adopters of AI to build competitive and responsibly AI ecosystems?

Audrey Tang

Yeah, certainly. So, when we did Alignment Assembly, we not just set the rules for information integrity. As part of that process, we also collected the people’s expectations on how AI should align to the local culture values if I use in public sector as facilitator, as mediator, and so on and so forth.

there’s a open research question of how to, in a multicultural setting to have this baseline culture and then to have culture specific uses and for a specific deployment size, the code of conduct that work within those cultures and this is a field called “Pluralistic Alignment”.

So I think this is very interesting because then we can then for humans instead of coding like Python or something, or a set of standard answers, you just tell stories of how you expect AI to interact with you, and then automagically, you can fine tune AI models so that it becomes your community model that builds from that infrastructure like the Sovereign AI.

So, I see Sovereign AI is kind of a translational layer between the base foundation models and the specific community uses and this pipeline by itself is very useful.

Glen Weyl

And I would say that for the base models, eventually those adapted models will be very useful because they’re actually data sources for learning about the different ways that the broad model needs to learn how to behave.

So, I think it’s a circular process where the adaptation fuels improvement in the underlying infrastructure and gives more people a chance to participate which of course is precisely what creating more data is.

Satwik Mishra

So, Glen on this exactly on Pluralistic Alignment and the pipeline that you mentioned. So, from the base level to having Sovereign AI, we also have a global, public policy ecosystem right now with the UN stepping in with the safety summits, which are happening, where do you think comes into alignment between the Sovereign AI and these global interventions at multilateral institutions?

How do countries right now stepping into this domain, balance what’s already happening at the multilateral level and developing their own Sovereign AI ecosystems?

Glen Weyl

Well, I think the real question is how do the frontier model developers incorporate the information from these fine tuned models and bring it back into their base model and how do we in fact, we ensure, especially given that a lot of these sovereign models are open to the extent that they’re incorporating unique cultural information from places with, you know, endangered languages and specific cultural attributes.

How do we ensure that those communities receive the support that they give to the underlying frameworks back and I think that’ll be fascinating, because it maybe that some element of that is kind of a share a like feature where the underlying models if they’re using open-source adaptations or if there’s become important, you know, there might be a legal doctrine that they have to be made open themselves, at least to some extent and that may actually advantage the open models and allow a more open ecosystem or in some cases those communities, you know, inputs are particularly important, there may be a need for public policy that directly channels some of the value that’s created by those, proprietary models back to those communities and I think that can be a critical role for global policy institutions to try to figure out what is a fairer framework so that sharing of value, or for these share like type policy.

Satwik Mishra

So, one of the things in this space around Sovereign AI often spoken about exactly to what you’re saying is, will the demand for interoperability across these models aligning to different countries become higher and higher as the year progresses?

Audrey Tang

Well, I think there is interoperability because almost by definition, a Sovereign AI is itself a adaptive machine learning model that you can put it in as I mentioned, a remix of it’s both open-source or a kind of chain reaction with a model from a foreign culture and so on. So, it can actually serve as a cultural translator, a adapter, a intermediary, a negotiator, right, both Sovereign AIs training something like a diplomat, and that can negotiate different values, because at the end of the day,

what is the most important for a Sovereign AI developer is this idea that it’s not a colonizing force that tech happens to our people but rather our people have expectations. How tech should operate, and the tech itself should be steered continuously about a changing also expectations from the people.

So, the Sovereign AI developer is in this faster iteration cycle with higher bit rate of communication with the local culture. So that if multiple, sovereign nations start doing that, then you can also then see them pooling together for a norm in a unilateral in the plural lateral setting and so on, so that the global compact far from being very abstract and to be implementing the top down way can actually negotiate with the community level, the middle layer of the sovereign AIs interacting by interpreting the Global Compact to the local people, but also have the global wishes and the frontier development on the socio technical parts of this system to reflect back, to the global repository and this is very exciting. This is a true cross-pollination of pluralistic alignment.

Satwik Mishra

The next section that I wanted stepping away from the public policy landscape. I want to discuss the industry perspective now through 2024. If there’s one thing, we know that industry definitely embraced AI all throughout its operations, it was seeking increased productivity, it was seeking operational effectiveness.

It was catering its services and products towards embracing AI, while at the same time there were several challenges which came out through the year, we had transparency concerns, we had data privacy issues, we had bias in the AI models, we had the skills gap in the workforce, and we had also legacy system integration, which required modernization to embrace AI.

Now, through our research across the year, we always saw industry adapt and come up with new pilots to approaches to governance. They were participatory governance models. They were decentralized governance model and accountability frameworks which were needed in this space and Glen this is something that you speak a lot about in your work. So, let me start off with you.

Do you think from the industry perspective, do you think decentralized governance models could help industries, address that trust mandates around services while continuing to foster innovation and build out fascinating services and products using AI?

Glen Weyl

Well, look, I, I think that, you know, one of the most common tropes about technology adoption is the hype cycle and I think the reason we get into the hype cycle is that there’s a series of social and organizational adaptations that are needed to actually use any technology effectively.

It has to fit with the patterns of how people work, and with it has to be, it has to change the way that they organize themselves. There was a classic example of this is when electricity came in, it did very little until you reorganize the workplace around a central, you know, generator that could then fuel all of the things within the workplace.

Just having a bunch of lights connected to the outside didn’t do much. You had to actually be able to organize the whole production process and the way that people interact with their work around the existence of electricity and the problem is we focus often because we want to go on the accelerator, not so much on the social side, but the social, the thing is actually that’s a mistake because you press on the accelerator when you’re pointed at a wall and you don’t go very far.

But if you focus on the steering wheel and then the accelerator is brought in as a secondary concern you actually go further faster because by empowering people to adapt the tools to their context you actually allow for more rapid adoption of the tools and, you know, obviously the privacy issues are, you know, one critical element of that and in fact, you’ve seen far less of the advance than you would hope to have seen in medical diagnosis because we can’t share the data across, different medical institutions because of the medical privacy, considerations.

So trying to center only the building of the tools and not the socio technical moonshots to overcome the data sharing issues and so forth actually slows down the path, the technical progress. This is why the debate over acceleration and deceleration is just missing the whole point. We need to get more acceleration. We need more focus on the social adaptation challenges that allow that acceleration to occur.

Satwik Mishra

On the social technical moonshots, Audrey, from the industry perspective, do you think participatory approaches like the Alignment Assemblies we spoke about could help align services and products to local contexts while maintaining their global operations as we know it?

Audrey Tang

Yeah. I mean, if local city councils and town halls and so on, like about 10,000 people in science, have already embraced the use of the tools like Polis or Remesh, Cortical and things like that in fostering conversations and having those conversations to have meaningful impact on decision making by sharing them in a macro mimic fashion so that, you know, that these come from this part of the region or these part of the stakeholder group, you don’t necessarily need to identify the real name, of the persons speaking it, because of power imbalances and things like that. So, for this kind of safe sharing, if it works on a polity of 10,000 people, it certainly works in the enterprise setting of 10,000 people.

It’s, essentially the same dynamic, right. When I was Digital Minister, I work with my staff so that we have this internal tool where anybody with 30 people counter signing a idea can trigger a brainstorm event where I response, every month in a livestream and if there’s more than 100 people, who agreed on some course of action, then I’m accountable on taking initiative and to outline how to implement it and so on.

So, the very low hanging fruits of where AI can be deployed in the frontline are discovered by people who are more in the frontline. We empower people closest to the pain using this way of macro mimic, participation. And so it’s obviously, has worked and nowadays the tools are very easy and they’re integrated that with most enterprise productivity tools.

So we need to think of productivity as social productivity. Also, pluralism in productivity is actually much more useful when aligning these new tools because otherwise you roll out new tools, it doesn’t fit and you’re slamming to a wall, and people have a negative valence, feel bad about it and they have to do two system, the old system and the new system and it just doesn’t work, right. Discovery process involving these tools is very useful in the enterprise setting too.

Satwik Mishra

On that, let’s split my question around enterprises before we step into the next session was like, let’s do a little bit of scenario mapping from both your ends and let me know, like what do you think in 2025, will we see more collaboration between enterprises and public policy, or would we go down the route of more conflicts.

Audrey Tang

Is that a Glen question.

Glen Weyl

Yeah. I mean, I, I think it will always be diverse, and we’ll see different models in different places. I’m very excited by what they’re doing in Japan in this regard. They have this idea that Audrey has been involved in at SusHi Tech, where it’s like sustainable high tech. Is that right?

Audrey Tang

Sus Hi Tech. SusHi Tech.

Glen Weyl

Yeah, exactly. I mean, the, where the idea is to avoid the hype cycle precisely by more closely socially integrating the development of the technology and, they have this incredible museum called Miraikan the Museum of Emerging Technology in the future, which simultaneously empowers citizens to help understand and shape the future, and in doing so, draws from them the insights that the industry needs to adapt their products better.

So, like that kind of co-creation, I think, is critical to productivity and augurs really well for what I think we’re seeing is an exciting model of participative socially grounded tech as a way to actually accelerate, the development of technology for industries.

Audrey Tang

Yeah, I call this people centric public private partnerships. So, 4 Ps,

Satwik Mishra

It’s wonderful SusHi Tech, that’s another link that we’d love linking to show notes for everyone to explore. All right. Moving on to our next section, through 2024 which was pretty much there throughout at the beginning of the year, which was really much in public discourse was the challenge of deepfakes and misinformation.

Now, I remember in January of 2024 all those articles around, this is going to be the year when democracy dies, this is going to be the year when deepfakes will take over, it’ll effect elections, and we survived 2024. We went through fine on the elections app and there weren’t any reports, as far as I know about deepfakes completely turning over any elections.

Glen Weyl

There’s one at the end of year, which was in Romania.

Satwik Mishra

It was Romania, Alright.

Glen Weyl

That was the first election to my knowledge that’s ever been invalidated because of this th

Audrey Tang

Information manipulations, supposedly.

Satwik Mishra

We saw some deepfakes which was around Taylor Swift endorsing Donald Trump, we saw that infamous robocall from Biden to New Hampshire voters not to vote. In this theme right now as we stand at the end of 2024. Audrey, what do you think, what was the role of governments and industry in 2025 in combating misinformation and protecting user rights and out here specifically because I’ve read Plurality and I would really encourage everyone in the audience to read it as well.

Audrey Tang

Yeah. So thank you, Plurality, available freely at plurality.net shows what we call prebunking because debunking sometimes results in people accusing each other of being, you know, foreign information manipulation agents and so on.

Of course, the foreign manipulation increases polarization, but this accusation also increases polarization. So, you end up is a very highly polarized space. And so, our approach is before the foreign interference happened, we predict what formula will look like.

And as public service announcements just tell the people that these kinds of attacks are coming. Right. So as early as 2022, I had an actor impersonating me, deep faking me, and showing exactly how the deepfake is done and that is like immunity, right. When people have seen it, the next time they saw a polarization attack using deepfakes, they were like, oh, a year ago, Audrey did that.

And that’s the attack, right and before the election, we also say you are likely getting messages that says, you know, this voting booth is unsafe and so on, which actually did happen and people was like, yeah, of course we knew it will happen and then we said, oh, there will be, fake videos and so on, accusing the tallying process as being rigged and people saw that message and they can go to any of their trusted, you know, influencers on social media.

We have all three parties, influencers just witnessing the paper ballot only tallying process in front of different angles keeping full documents. So again, that did happen after the election, but people was like, yeah, of course.

So so it was prebunked and so whenever you prebunk something you increase the democratic resilience because then people talk about this prebunking as a shared social object, no matter which part of the parties they are they seize to be polarized. They’re like oh we all deeply care about this attack scenario. And so that’s they see this polarization and so on from a different lens, right.

Oh, if we get polarized, then actually we’re victims of attack. We’re like scam victims or something. I feel like, oh, so of course we’re not getting polarized, over deepfakes and so on. So, this, this, prebunking process, I think is very important and we involve like middle schoolers especially, but all people in the education system by switching from a literacy, like data media literacy education into a competence education literacy where you consuming information, competence where you contribute to the collaborative fact finding and things like that, and people, again, who have gone through with this process have prebunking in their minds and become entirely inoculated against this kind of outrage provoked by deepfakes and so on, because they have thought like a journalist and this actually disbalancing, this bridging of sources and so on is a long time, antidote that takes, effects even a year or two years after this kind of deliberative competence practice.

Glen Weyl

I think we’ve also seen the way in which these, you know, quote, deepfakes are not all bad. I mean, they’ve been used to do satire and, in the future, I think it’s important to keep in mind that throughout most of human history there was nothing that you could give to someone that without any information about the social context would immediately tell you that that was valid, you know, we had writing, we had painting all these things, you know, could be used to make fiction, or they could be used to make truth, right.

They were not just immediately, accepted and it’s only in the last, you know, maximum 200 years that we’ve had media that you sort of expect that it’s always going to be truthful. Maybe it’s actually better now that we could go back to thinking always about the social context of the things that we see. Everywhere is something or something unless everything is a painting or something.

Yeah, exactly. And, and I think that that’s in many ways much better because, then we actually have better privacy because someone can’t just take a photo of you and people believe that that’s you, they assume that it’s fake.

Satwik Mishra

I ended my research paper at Harvard three years back on this topic, saying that let’s just assume everything is false unless proven otherwise.

And maybe that’s that’s the way forward. But prebunking comes in as a social intervention to deal with the information ecosystem and enhance societal trust which is fascinating. Another aspect from the technical lens, Glen which is being considered are decentralized systems from blockchains and in the content provenance space. Do you think there’s something out there going forward in terms of creating more provenance and authenticity in our information ecosystem?

Glen Weyl

I mean, blockchains can play a role but many other ways of doing cryptographic signing are relevant and there are many different substrates. Probably in those prominent, they say, the C2PA Coalition for content provenance and authentication and that can be very valuable. I think the interesting things that go beyond that which maybe blockchains can help facilitate are what Audrey would call macro mimic systems where various people with various credentials can attest to various things without revealing their full name and that or old contact information and also systems where you might be able to attest that someone else’s attestation was correct without that original person having to, you know, show that, or where a journalist could prove that a source had some relationship to the topic without it having to give that person his name. So, trying to get those more sophisticated versions I think it’s very exciting and that cryptography could play a really critical role in making that happen.

Satwik Mishra

Privacy preserving technologies and digital wallets and digital identities is a fascinating topic, but I’ll I want to veer into another area which we don’t want to discuss because that I could go on about with my experience in India. So finally, let’s move on to my final section which is overlooking through 2024 and looking at 2025 around a possibility of a new social contract which interfaces the need for trust in emerging technology.

And okay, let me start off with Glen. Glen, are there any radical steps from industry and government that you would want to be prioritized in 2024 to ensure that this emerging technological progress continues to foster innovation while upholding equity and trust?

Glen Weyl

I mean, I’d like to see all governments and companies center this kind of participation, which we’ve shown can work more centrally in their approach to, on both sides, coming to consensus and solutions rather than everyone wanting to put a stamp in a way that doesn’t cooperate so well. If they can use common participation and just infrastructures that can easily compose, then there’s the possibility of really laddering up to a interoperable global participation layer.

Another thing I really love to see is, you know, we’ve seen a lot of this litigation from content creators and, you know, kind of a defensive crouch from the model creators. And I’d love to see governments and civil society getting involved in trying to get away from a purely legalistic way to actually try to identify who is making the greatest contributions to making the models work and trying to ensure that those people are recognized and compensated in a way that is proportionate to the value that they’re inputting to the systems, rather than just based on whether they’re replicating an output.

Satwik Mishra

A radical focus on more participation and multi-stakeholderism, valuing those who are contributing to this entire ecosystem. Audrey, let me move onto you and ask you, what do you think would be the essential principles that should define this new social contract for emerging technology and trust in 2025, what do you hope for in the coming year?

Audrey Tang

Yeah, I think there three principles should be voice, choice, and stake.

That’s from Project Liberty Institute and which I’m a senior fellow. So, we talk a lot about voice already in our, podcast so far, voice means that people using the tech need to have meaningful input into how the tech gets to interface with the society. It should be domestication from this side, not domestication from the other side, right from the machines side.

And to do that, we need obviously, the person who had credentials, the verifiable credentials that we just talk about. Otherwise, any person can just create 50,000 fake accounts and call it voice. Of course it’s not voice, right. So meaningful participation grounded, in the proof, of that I’m actually somebody affected by this technology. This is very important for the voice track, for 2025.

For the choice stack, we’re increasingly seeing new designs that focus on protocol rather than specific applications. For example, Blue Sky, and the Mastodon, Veriverse gained a lot of ground in 2024. Nowadays is like more than 24 million people in the, at Protocode, the Blue Sky ecosystem alone, and many others, in the Veriverse.

And so the point is that if somebody sees the experience of interacting with the tech deteriorate, then previously, if you leave just by yourself, you lose all the network effects, you lose all the value, which is why people get trapped. Right. There’s a research from 2023 that shows, you know, an average undergrad in the US using TikTok.

If you want them to quit, you have to pay them like $60 a month. But if everybody quits together around them they’re willing to pay you 30 bucks per month right. So everybody is is that, and so and this kind of, decentralized design, means that people who are not very in favour of how Blue Sky show things can switch to Black Sky.

It’s not hypothetical. There’s actually Black Sky, and so on. And so this freedom of choice means that the operators then are incentive aligned to actually providing pro-social values, because people finally have a choice. And finally, there’s the stake layer that’s Glen talks about of meaningful compensation both monetary but also attribution and also just sharing the agenda setting values, right. If I contributed to something, sometimes I don’t care that much about money or my name on it but I want to have a say, it’s steering wheel. So, the voice, the choice and the stake principles, they work together.

Satwik Mishra

So wonderful, wonderfully put. Right, before we end the podcast, I have final two questions that I would love both your short answers to.

First, let’s start with Glen. What was the most promising and concerning AI development of 2024 from your perspective?

Glen Weyl

The most promising was Anno San in Japan. Anno Takahiro who ran for Governor of Japan, of Tokyo and used AI to enable collaboration with tens of thousands of citizens to build a platform that became one of the most popular ones in the election and now has helped transform Japanese politics.

And, the most concerning example, I would say is, is perhaps the way that TikTok, may have been abused in the Romanian election and, the way of nearly it caused internal conflict.

Satwik Mishra

Audrey.

Audrey Tang

Yeah, for me, the most promising development is the openness of AI is now seen as a safety layer. That was not the case in 2023. They were, in different blocks and now we’re increasingly seeing, for example, people investing in CSAM mapping, so that while it’s illegal to hold on to such images, there’s AI models that can, in the latent space, translate that into text which is legal. And then you can have federated learning at the edge and so on.

A grooming text which also helps facetime detection. So, this proven non-dual-use, use is actually very, very promising. And I think the main concern that I have is just that many people, many jurisdictions did not know like you don’t have to make a trade-off between safeness and openness. Just like many people thought, in order to make AI work more capable, you have to sacrifice privacy

or that’s in order to, you know, counter, deepfakes and disinformation online, you have to sacrifice freedom of speech and do censorship. You don’t actually, these kind of like nonbinary choices actually do exist. And the most concerning development is I think a lot of jurisdictions just passed laws and also some of the bilateral agreements that just, phrases this as a binary choice.

That is the most concerning. Hopefully in the new year this kind of non-binary choices can be made known to more people so that we see this as a co-creation process, not a either or process.

Satwik Mishra

It’s finally, what is both of yours one hope for 2025?

Audrey Tang

Glen

Glen Weyl

That we see the mainstream adoption of content serving protocols using these bridging ideas and that people start to see at scale the way that those environments can heal the social platform.

Audrey Tang

Great, and I’d like to see at the end of 2025 people collectively saying, oh, actually, we’re past peak polarization. It was a year ago or two years ago and so on. We remember being polarized, but that’s a issue that’s behind us. Now we are thoroughly depolarized and have meaningful bridges across communities.

Satwik Mishra

Audrey and Glen, thank you so much for joining us for this end of the year Trustworthy Tech Dialogues.

Audrey Tang

Thank you. Live long and prosper.

Satwik Mishra

Hello everyone! Welcome to another episode of Trustworthy Tech Dialogues. I’m Satwik Mishra, The Executive Director for Centre for Trustworthy Technology. Today, we’ll dive into one of the most transformative intersections of our time artificial intelligence and healthcare. Two areas hold as much promise as the integration of AI with healthcare. A field that touches every life, family and community. AI is actively redefining what is possible in healthcare, making care more personalized, predictive, and accessible than ever before.

It offers a future where early diagnosis, precision treatments and as importantly, operational efficiencies reshape how healthcare is delivered by automating routine administrative tasks while offering crucial, actionable insights. AI can free up healthcare professionals to focus on what they do best “Healing”. However, with this incredible potential comes profound responsibility. How do we ensure that this pivotal integration is equitable, transparent, and trustworthy?

How do we build systems that are not only effective, but also trusted by those that are tools, but it is meant to serve? These are the questions that will define the future of AI in healthcare. To help us unpack these themes, I’m honoured and privileged to welcome Maneesh Goyal, The Chief Operating Officer of the Mayo Clinic Platform. Maneesh is a visionary leader whose role at Mayo Clinic Platform includes shaping strategy, driving innovation and guiding investments in technologies like AI.

With a cross-disciplinary background in finance, engineering, product development and healthcare, Maneesh brings a wealth of insight. Today we’ll tap into his expertise to discuss how Mayo Clinic Platform is navigating this dynamic landscape, turning the promise of AI into practical realities while ensuring trust remains at the core of this innovation. Finally, we also explore what excites him the most about the future of AI in healthcare and what he’s wary of.

Maneesh, welcome to Trustworthy Tech Dialogues.

Maneesh Goyal

Thanks for having me and definitely excited about the times we live in and the time we have together here in the next 30 – 40 minutes.

Satwik Mishra

Hey, so I want to start off with your journey actually, you come from this very unique, interdisciplinary background. Tell us about your journey and how have these interdisciplinary lenses help you in the role today that you occupy with healthcare?

Maneesh Goyal

Yeah, no, it’s interesting. I often say that, my past experiences, really shaped me and made me ready for this role and I break it up into the first three jobs that I had. I was an engineer, so nine years helping to build product, semiconductor engineering, specifically.

And, you know, the company that I helped to build and, and, scale today’s, Fortune 50 companies delivering semiconductors for just about every device in your home and the pleasure of working with some of the smartest people on the planet. Around that, second was actually, moving into early-stage finance. So venture capital and and really, we dove into health care at that point.

I got a chance over a seven year period to really examine all the different parts of healthcare as an investor. And then finally, I jumped back into an operating role I realized that, I’m really good at operations and, help to build a health tech company where, we were selling into payers, providers, and employers.

Why is that interesting is if you think about the health care ecosystem, it really is governed by those of organizations, whether they’re the primary supplier of services like providers or they’re the primary funders like, payers or employers. And, and, when I left that organization, had an opportunity to help write the strategy of Mayo Clinic Platform and then got, you know, the role of a lifetime to help directly operate it.

So, if we sort of piece that together, understanding how startups scale, how to build a large-scale engineering organization or product organization, and then really understanding how the economics of health care work, those were the necessary steps and, in many ways, the last five years of Mayo, really got in a chance to understand how well-run provider organizations can deliver high quality care.

Satwik Mishra

It’s so wonderful and so for all of us for following this space, obviously, we know the wonderful work of Mayo Clinic and Mayo Clinic Platform what it’s trying to do. Can you share for audience just the general vision and mission statement for Mayo Clinic platform as to what it’s trying to do today and what is its vision going ahead with these emerging technologies?

Maneesh Goyal

Yeah. So, I always start with the notion of the word platform. Most people think of the word platform when they think of technology. For us technology is an enabler, platform is a business model, it is a way of doing businesses, a transformation tool. And you’re familiar with platform businesses because you’ve all used Vrbo or Uber or Lyft, right?

They’re asset light they are about building ecosystems together, they’re about sharing, they’re about establishing guardrails. Those are the key things that are common in platform organizations. Our CEO, Dr Farrugia, when he took this role, said, you know, healthcare has not really embraced platforms. There is a lot of proprietariness in health care, my patients, my data, you know, my intellectual property and Mayo Clinic made the bold move four years ago and said, we’re going to build a platform organization and we’re going to show the world that we mean it by sharing our data, because we recognize that that data is part of a social contract that we have and what that means is that they’re underlying that data is knowledge.

And it’s by sharing that knowledge do we make other health care systems, other providers better and if we have likewise participation back, then we become better and, in many ways, Mayo started 150 years ago, 160 years ago with that ethos. Physicians and administrators of Mayo dedicate their, their time to not only delivering care and enabling to deliver care, but also imparting knowledge.

Teaching and learning it is just a part of our ethos and so what we’re doing in platform is just putting that on steroids, making that very scalable and so our primary KPI is that today, Mayo sees patients from 150 different countries, 1.2 million, 1.3 million patients a year. We want to enable greater than 50% of the health care providers globally.

We want to partner with these institutions, whether they’re other academic medical centres or community health workers, and thereby enabling the impact on billions of lives, not just the patients that are fortunate enough to get on a plane or drive into one of our sites. And so, you can only do that under our platform model, which means you have to be very nimble, very agile, lean and be scalable so another academic institution can partner with you, but also, you know, community health worker where the economics are very different.

So that’s really hard.

Satwik Mishra

That’s wonderful and the principles of being nimble and agile for the scalability aspect, that’s a principle-based approach, which we’re all looking forward to see from Mayo Clinic Platform. But another aspect of it is dealing with these emerging technologies as the capabilities develop and when you’re thinking about the platform model and how do you think about how is the platform responded, how the Mayo Clinic Platform responds and possibly not only responded, responded, maybe the wrong word evolved alongside these developments in emerging technologies?

Maneesh Goyal

Yeah. No, it’s a really good question and as you were asking, I was just thinking about how, so if you if you go back five years ago, five years ago, putting 100% of your clinical data in the cloud was novel, right? And it was, we were arguing about BAAs with cloud partners. We were arguing about, are we releasing control of our data to a cloud vendor?

Those things are passé now. We take for granted that we have still maintaining patient privacy, we take for granted that there are security perimeters in place in the cloud.

Satwik Mishra

Right.

Maneesh Goyal

So that innovation felt like an innovation five years ago, but now it doesn’t.

Satwik Mishra

Right

Maneesh Goyal

3 or 4 years ago, we said, how do we de-identify patient data so we can make it usable without going through an entire IRB process. Of course, every patient you use, their data has got to be consented. But once it consented you want to protect that patient privacy.

So, we had to develop tools to de-identify complex, unstructured data sets. We had to develop tools to, to de-identify images. So, we’ve done all of that over four years. We’ve had to leverage tools that allow for federated learning, which is basically learning across multiple sites so that you’re not transferring data back and forth or consolidating data, which really is not beneficial to anyone.

You know, the AI world has been evolving seemingly on a nine-month cycle and maybe faster these days. Who knows? And so, ensuring that you’re building an architecture and evolving the architecture so that not only are you building and supporting predictive retro retrospective analysis but you’re supporting predictive algorithms and then evolving to support generative, models and algorithms.

So, we’ve had to evolve all of that. The thing that I think has made us futureproof is a set of guiding principles, and we describe that as data behind Glass, which is that we’re going to organize the data, we’re going to de-identify it, we’re going to get a third party to certify that it is truly de-identified, and then we’re going to disallow data to come in and out of that ecosystem or that infrastructure.

By doing that, you’re now completely safeguarded that data and you can create algorithms, you can create training models, you can do a lot of knowledge extraction from that, that protects the patient privacy and intellectual property and then also safeguards it against the evolving regulatory landscape. Because, you know, if you’re going to do this at a global scale, you have to support the AI laws in Europe, you have to support GDPR, you have to support HIPAA, you have to support all these models and by by being more constrictive

and how do you interact with that data and you really are focused on extracting knowledge. You can actually be nimble as these technologies evolve. That’s how we’ve approached it. I think our next frontier is going to be how do how do we make sure that we put these in front of the nations and physicians and, and administrators in a very seamless way, right.

We finally are, you know, reaping the rewards of the digitization of clinical workflow. Right. All of those, things that we’ve had to deal with, the digitizing and EHRs etc. We’re now seeing that that data sitting in our repository is beneficial to patients and physicians and so, I’m excited about it. as you can tell.

Satwik Mishra

It’s a very fascinating space and one of the key things that I think sometimes people miss is, yes, AI is helping along with the drug discovery process and core science. But one of the administrative processes and overheads, from the physician’s perspective or healthcare professionals’ perspective, which data can enable is a fascinating frontier. And in this context, how do you see AI contributing differently for for medical functions versus these support functions, which are so crucial

and how would you distinguish between the two as you think about the AI integration?

Maneesh Goyal

Yeah so I would say maybe let’s break it up into three sections. And I think that’s what you did. Right. So there’s science, there’s discovery work there’s clinical use cases and then there’s administrative use case.

Satwik Mishra

Yeah.

Maneesh Goyal

So, when we set up an architecture that basically said how do we define how AI is going to be used? So, if it’s used for these three kinds of uses, let’s make sure we understand the risk and in that landed itself to a responsible AI framework and so we can talk about that, in more detail. But the idea is that if we frame it as to how that how data is going to be used and how the results of solutions created from the data are used, then we can understand the risk and understanding the risk in healthcare is really important because bias creeps into it quickly or easily unknown or data, can, can can, find its way, outside of your control.

So, we have to be very careful. And so if you look at that range, administrative use cases, tend to be the easiest ones for solving a lot of the financial woes as well as the, you know, they’re most safeguarded because you’re not dealing with a lot of clinical data for the most part.

So, I think we as an industry have an opportunity to actually change how, how how every organization is scaled. So, you know, at Mayo we have thousands of people that are focused on, on coding and scheduling, you know, with, with a lot of these tools, we can reposition people into more patient facing or reposition them into creating solutions that give us more scale

and I think that’s the that’s the opportunities that we can take that added administrative swelling that’s happened because we’ve added more and more bureaucracy and and contracts. It can reduce that infrastructure and that ultimately means lower cost, hopefully.

Satwik Mishra

I do want to ask you about the network of stakeholders that the platform enables and which is absolutely fascinating.

But one of the things before I get to that, I do want to ask you a fascinating intervention on the Mayo Clinic Platform is the “Digital Hippocratic Oath” and can you shed some light on that, its value and its need in the times that we live today?

Maneesh Goyal

Yeah. So, it’s actually born out of Mayo’s primary directive and when you come to Mayo and you are whether you’re an employee or you, you are a visitor or you’re a patient, you quickly recognize that our primary directive is the needs of the patient come first.

And you can feel that. You can experience that by engaging anyone at Mayo Clinic. And so we said, how do we turn that into something that is captured in something as cold as digital technologies, how do you how do you human humanize them? And so we created this Digital Hippocratic Oath, which was effectively the embodiment of that and what we call our rich ties, which is a set of principles that we operate against.

And they basically distil down to the fact that you want to make sure that every time a decision is being made or needed to be made by a piece of technology or a mathematical equation, that it is being made for the betterment of the patient. So, you don’t focus on financials, you don’t focus on anything. But is the patient going to benefit from this decision?

And by stating it that way, you never go wrong with patients. Right. Because healthcare, very complex ecosystem, there’s a lot of competing interests and it is easy to get lost. So, let’s just write these down and and that’s what we did.

Satwik Mishra

We’ll link all of this, the Data behind Glass Model, the Digital Hippocratic Oath in our show notes and it’s fascinating and I encourage everyone in the audience to definitely look it up and read it up. I do want to ask you again on this multi-stakeholderism, which is something that we at the centre consider really valuable and Mayo Clinic Platform played a crucial role in establishing the Coalition for Health AI, which now includes over 14,000 organizations.

Tell us about its need and its early days and how it stands right now?

Maneesh Goyal

Yeah. Let me actually take a step back and so, one of the things we recognize is that when you’re creating AI or you’re looking at data, there’s bias there. No matter what you do, you’re biased, you have, and it comes in the form of the kinds of patients you see based on geography, the kinds of services you offer based on skill sets, where you are located, what kind of business model you have, or your capitated system are you a fee for service system, so on.

And so one of the things we recognized is that Mayo just put our data together and said, okay, we’re going to innovate. We’re not really great set of data for patients that look like patients that are, in, you know, in our Minneapolis, Rochester, Wisconsin area and our Arizona area and Florida area.

Well, those patients, you know, have a certain demographic. So, we said, let’s make sure we remove bias, or at least attempt to remove as much bias as possible. And so, so we went around the world and said, let’s identify the highest quality institutions and have them participate, ask them to participate in a data sharing collaborative. And so today, we have 40% of the top 11 institutions

and shortly we’ll have 50% of the top 11 institutions that are part of that network, which is effectively collaborating on data sharing. So, 100% of the data organized in the same way and accessible to each other for novel solution creation. So novel algorithms, validation, etc.. so when you have that across North America, South America, Asia, Africa, etc., you start to actually remove those individual biases because you have different institutions, different care models, different people.

That was one movement that we made. The second movement we have made was to make sure that we partner with innovators, whether they’re large multinational organizations like Barr Pharma or MEDevice, all the way down to small startups. Because we recognize that, innovation isn’t captive. It should be we should be encouraged and then finally, we said, how do we actually make sure that any innovation doesn’t benefit these small group of stakeholders that look like us.

How do we make sure that they serve all those institutions that are the critical access hospitals, or the community hospitals or the clinics, and I and I in that ecosystem today consists of, you know, 11 institutions across the globe that are partnered with us on data. Almost 100 solution developers and 50 health systems that are consuming solutions. And so we’re getting to the point where differentiation you know, the, the diversification is creating a lot of positive, positive momentum.

So, when we did that, we recognized that we have an opportunity to do good, but we also have an opportunity to create chaos, because if we have all these institutions and no communication framework, then it could be Mayo establishes our own way of communicating bias or fit for purpose and another institution has another way, and another institution is another way.

And we don’t want to have a proprietary anything. Everything we’re doing is about engaging the broader set of stakeholders. So, we created this coalition called the Coalition for Health AI, along with our partners at Duke and, and, 14,000 other organizations. And what that group has done is established a blueprint. So, when you’re creating an AI, here’s how you communicate.

And and you’ll often see them use the term nutrition label. So, when you go buy a food or packaged good, there’s a standard mechanism of communicating everything, it’s in their, ingredients, you know, nutrition facts, etc. that’s what we want to get to. And so what we did is to say, okay, you as an organization, Coalition for Health AI, come up with that or whatever you come up with we will implement them at Mayo Clinic.

Right. And so that’s what we’ve done and in that way when any institution leverages the goodness that that our ecosystem is created, it’s always in a standard way.

Satwik Mishra

Coalition for Health AI is an absolutely fascinating consortium in terms of course bringing together this diverse group of people and then obviously the blue blueprint around responsible AI which has come up.

Let me, from your experience right now, AI is like redefining several industries outside of healthcare as well. What in this multi-stakeholder coalition and coming together to come, to build out this framework, which everyone does follow, right from the startups to academic institutions to the big players, right. What lessons or advice, would you have for other industry consortiums trying to build out these trust frameworks, responsible AI frameworks

and there are so many of them today, right, but to have something which is universally accepted across our industry is still pretty hard?

Maneesh Goyal

Yeah. Yeah, it’s absolutely the truth. Right. So, I would say you start with basic principles. Right, you started this conversation around, the Digital Hippocratic Oath, you need to agree on something.

So, a Coalition for Health AI, I think all of the partners agree that the betterment of care, care quality is the number one imperative. So, then you build on on top of that. So, establishing a foundation that you can all agree on unabashedly is sort of the start and then second, you get to you have to check your ego at the door.

You know, if Mayo came in and said, we’re we’re the number one institution by many, many, reports or analyses, you should always listen to us, and we know best. I think we would have had a coalition of one organization instead we said, let’s all learn together, let’s all share and that ethos was very important. And, and I’d say if you want to establish an industry consortium, that’s another piece.

I think the last piece is that you can’t try to, cordon off things like intellectual property, too early, right? You have to build a framework; you have to build a safe space for people to share and learn. There’s always going to be room, and you have to think about intellectual property, but you have to create safe space early, especially when early days, when you just don’t know enough.

So, I think those are the rules that we followed to encourage participation.

Satwik Mishra

I have two final questions, which I’m most excited I always end up asking people is firstly. My second last question would be that how do you conceptualize trust while dealing with a technology whose capabilities are advancing and accelerating so fast while it is being integrated into crucial domains such as something that you lead around health care, which affects our daily lives right by the core infrastructure itself, its capabilities keep advancing in this sort of domain and intersectionality, how would how do you think about societal trust for this sector?

Maneesh Goyal

You know, I think first we we have a, I guess a mantra or saying at Mayo that we don’t believe that AI is going to replace anyone. It certainly is not going to replace physicians. Well, we do think that is is going to happen is that physicians who use AI are going to end with end up with the delivering higher quality care than physicians and organizations, they don’t.

So, it’s almost like no one would argue about using an X-ray to determine if a bone is broken. You might have argued about it a hundred years ago and so, it’s just another tool at the end of the day. And so fit for purpose is really important and that is how we’re going to create trust.

I don’t think that Generative AI today is ready to help make decisions around, clinical decisions, whether it be diagnosis or therapeutic pathways. Will they get there, probably. But in between now and then, we have a lot of room and a lot of opportunity to leverage Predictive AI, which behaves in a consistent, predictable, responsible, understandable, transparent manner and most importantly, explainable manner.

And so, recognizing that AI is nothing but mathematical implementations and so as long as you can understand the math, and the math is appropriate for the purpose that you’re using, whether it be clinical decision making or patient scheduling or, coding for reimbursement, you can use the right tool. And so, it’s just becoming smarter about what is powering the underlying tool.

That doesn’t mean you need to be a data scientist, but you do need to have a little bit of a better understanding than, you know, plugging in an X-ray and image back.

Satwik Mishra

And finally, let me ask you to do a little bit of scenario mapping from your position and tell me what gives you hope in terms of the technology’s capabilities and its integration with healthcare right now, looking ahead 2 to 5 years because, I don’t want to, five years may also be pushing it right now, but let’s say two years from now.

and what is, is there anything that worries you that we should be paying more attention to?

Maneesh Goyal

I often say that we’re in a cell phone moment, so I’ll give you the optimistic answer because I remain immensely optimistic. So so if you think about what happened when when cell phones were introduced across the world and they became, cost effective such that billions of people can could own them

and that happened pretty rapidly if you actually look at the diffusion curve on cell phones and, I don’t know, it’s like if you were raised in India but I, I was born in India, and I was when I was a child, it took a year or more to get a phone line established in your home. And up until even the late 90s, that was the case.

And along came cell phones. And now every rickshaw driver or or, you know, person that’s, you know, selling food, in stalls now has a cell phone and that changed the entire landscape in a developing nation like India. I think we’re we’re we’re going to have the same kind of impact as we think about idea around AI and how decisions are made and quality of care is delivered across the globe.

And so that the disparity between somebody getting care at Mayo Clinic and somebody getting care in a part of it, developing part of the world that doesn’t have a physician in front of them. They have a community health worker, I think you have a, you know, it’s never going to be perfectly levelled, but boy, do we have an opportunity to level that a little bit.

So that disparity is a little less and I think there’s no better legacy to leave, you know, to our kids than, than that kind of levels.

Satwik Mishra

That’s absolutely a wonderful optimistic note that I, I really hope comes through and I’m sure it will that all of our work. Thank you so much Maneesh for being here. 

Maneesh Goyal

Thank you. Really appreciate it.

Satwik Mishra

Hello everyone! Welcome to another episode of Trustworthy Tech Dialogues. I’m Satwik Mishra, the Executive Director of the Centre for Trustworthy Technology. Today, we’ll dive into some of the foundational issues shaping our technological landscape. The curation of inclusive data structures. The value of ethics for aligning our tech architecture to a human centered future and building societal trust in artificial intelligence.

Today we’re confronted with a choice, will we permit ourselves to be captivated by the simplistic binary narratives of doomsday and utopia, or can be instead turn our focus to a nuanced, shared, and profoundly human reality of the people that AI is truly meant to serve. At the centre, we believe it’s about this human reality, a path of hope grounded in authenticity.

But hope here is not naive. It doesn’t ignore alarming realities or troubling statistics. Instead, it embraces the urgency of today by pushing for systems that are just inclusive and trustworthy. Hope, which is grounded in authenticity, allows us to look at AI not as a silver bullet or a looming threat, but as a tool with the potential for transformative change.

It keeps us moving towards a world that prioritizes fairness, dignity and justice. Today AI is woven into us shared realities, creating powerful hubs of influence while also enabling millions to learn, innovate and access new opportunities. Our discussion today centers on this duality. AI may concentrate power, but it also holds the potential to amplify voices and empower individuals at every level of society.

With AI’s great power comes at corresponding peril if we proceed without due diligence, ethical guardrails and a commitment to human rights. Every tread off innovation decision on our data systems and the ensuing policies should hold up to a standard of reasonability and fairness to maintain public trust. But today, we’re asking some fundamental guiding questions. How do we ensure that emerging technologies like AI enhance our agency?

How can we balance innovation with accountability? Securing a future with tech truly acts as a force for good. What would it take to be hopeful with purpose to navigate forward with a vision grounded in authenticity and inclusivity. To explore these pertinent themes, I’m joined by Professor Renée Cummings. Professor Cummings is a world-renowned AI data and technology ethicist.

She’s currently serving as the first data activist in residence at the University of Virginia School of Data Science. In addition, she co-directs the Public Interest Technology University Network at UVA and holds the position for Non-Resident Senior Fellow at the Brookings Institution. Professor Cummings is a member of the World Economic Forum’s Data Equity Council and of the AI Governance Alliance.

She also serves on the Advisory Council for the AI and Equality Initiative at the Carnegie Foundation for Ethics and International Affairs. Her work delves into the nuance, ethical implications of data on society. Through her research and advocacy, she has elevated our understanding of AI systems, guiding us towards a future which is ethically resilient, responsible and informed for the benefit of all.

Professor Renée Cummings, welcome.

Prof. Renée Cummings

Thank you so much. It’s such an honor and a pleasure to be with you.

Satwik Mishra

Hey, to start off, tell us about your journey into AI ethics. How has your background in data science shaped your perspectives on developing trustworthy AI?

Prof. Renée Cummings

I will say that my background begins with my work in criminology. It is working as a criminologist, engaging with these risk assessment tools. Algorithmic decision-making systems are data driven systems that I realized that the predictions, the forecast that were being made with the zombie prediction that were overestimating the risks of black and brown defendants in the criminal justice system. It also had me thinking about risk and what do we define as risk in criminal justice? and looking at the connections between risks in criminal justice, risk in the financial sector, risk in the insurance sector, risk when it comes to credit scores and realizing that I was seeing some patterns and some trends that really needed, some critical analysis.

So that is how I got into AI, really thinking about the ways in which we were using a new technology, but we were actually deploying old, discriminatory, stereotypic stories steeped deep in systemic biases and wondering, what did we need to do as a collective at interdisciplinary thinkers to really amplify, this conversation, to really look at those risks, because I am so committed to, technology and advancing systems with technology that I believe the only way for us to truly reap the rewards of this extraordinary technology called AI.

We had to have a really sophisticated understanding of the risks, our rights, and most critically, what were our responsibilities? For those of us using this technology, designing, developing, procuring, adopting this technology. So it is through criminal justice, through criminology and criminal psychology and really working in the space of human behavior and the intricacies of the human mind.

Then I found myself really falling in love with data science and of course, the building of algorithms.

Satwik Mishra

That’s wonderful. Like, it has been such a privilege over the last two, three weeks going through your work and preparing for this podcast. It’s been absolutely wonderful to see your journey and listen to the things that you’ve spoken about from a very societal lens.

But on an emerging technology like AI and at the root of many of the largest questions around data and AI, you have explained a foundational what you call data dilemma like as powerful and I’m quoting you here “As powerful as data is, it is also very vulnerable”. Now explain to us what is this data dilemma and how does this data define the challenges of inclusive AI growth as we know today?

Prof. Renée Cummings

Most definitely. So, we have something that we cannot see which is data, but it is so powerful that it has been the greatest game changer when we think about geopolitics, when we think about the global power structure, you know, we’ve heard the saying, those who own the data, own the world. We’ve heard the saying our data is becoming our DNA.

We’ve heard this saying, you know, data power is a global power, it’s economic power. So, when we think about something that is so powerful, then we’ve got to understand how vulnerable it is. When we think about privacy risks, when we think about safety and security, when we think about questions around data governance or the lack of data governance, when we think about something that is being amassed in such great amounts and we cannot see it, we cannot feel it, we cannot touch it, but we know it’s growing and we know it carries an extraordinary value, not only a societal value, for the extraordinary things that it can do.

But it carries a monetary value because everyone wants to own it, to have access to it, to use it to rethink, redesign, reimagine, and reposition. If not their business model, their strategies, and of course, use data to design their solutions. What is so critical is that we’ve got to think of the question of equity. I always say the data equity is good economics.

Data equity is not only about equal rights, it’s about equal opportunities. And at the World Economic Forum, I was really privileged to be a part of a group, the Data Equity Council, that put forward a really brilliant document on data equity and also situated within the context of indigenous approaches to understanding equity, understanding data governance, understanding questions around data serenity and understanding something as critical as data provenance.

How do we look at the history of data? How do we You know, look at the ways in which, the trajectory of data is, being developed and, and the kinds of ways we need to maneuver that or how data is being manipulated. So, inclusion is critical because inclusive data gives you more opportunity. Inclusive data gives you more approaches.

Inclusive data gives you more dynamic ways in which to think about the solutions and the strategies and the system that you want to develop. So, when we think about this question of the data dilemma, we’ve got to think about questions around power, questions around policy and questions around politics, because we’re seeing the data has a political context, a social context, a cultural context, an economic context.

And all of these contexts, lets us know how particularly sophisticated we have got to become. You know, our own understanding and why critical thinking is also crucial to this data dilemma.

Satwik Mishra

We will definitely link, the Data Equity Council report in our, in our show notes, the wonderful report, and I encourage everyone to read it.

You have this thing around coming up with slogans which just inspire the community. And from the data dilemma, you also have something you call the “Duty of Care”. And you said around, let’s move from the data dilemma to AI ethics. You talk about like navigating AI ethics requires a duty of care across a range of stakeholders.

Now, can you share your vision around what is this duty of care and how should it be exercised across like governance right now?

Prof. Renée Cummings

Most definitely. You know, when we think about risks, we think about responsibilities, and we think about rights, and we also have to think about questions around protection and liability. Liability is critical to AI safety, to AI governance, to AI ethics as well as data.

I always say there is no AI without data. So, when we think AI we got to be thinking data, when we’re thinking data, we’re also thinking AI. So, duty of care is our responsibility as data scientists, as technologists, as leaders, as social scientists. So, when we call for this interdisciplinary approach to data, we’re looking at interdisciplinary duty of care as well.

So, we all have a responsibility, it is all of us who have this duty to ensure the work that we are doing in this space really presents an attitude of care. Caring about not only the communities but caring about the long-term impacts of this technology, caring about the, global, geopolitical, relationships, caring about the future, caring about the environment.

We have a technology as super as it is. It is also super destructive when we think about the environment, when we think about the resources required to create these massive data centers or for compute power or for building the, you know, the, the semiconductors that are, that are required. So, when we think about our duty of care, it is our responsibility.

And it is also the liability that speaks to the fact that those who may be harmed, because we did not deploy that due diligence or that duty of care must have redress or questions around restorative approaches, using AI or questions around reparative approaches using AI. So, duty of care is this care and concern and liability that must be a part of any conversation around data, AI, safety, ethics. Of course, the future, the ways in which we are building our future using data and AI.

Satwik Mishra

So, this is a fascinating topic and we’ve, all in the community been working at it and made some progress, not to the level I think any, any of us would want, but we are still at it, in this duty of care and building these inclusive structures like across our domain, what progress do you find encouraging

And where do you think more of our energy needs to go?

Prof. Renée Cummings

I will start with the the end of the question. I think our energy needs to be focused on the governance, and we need to understand that governance is local as much as it is global. So, we’ve got to have these local conversations. You know, when we think about local governance, when we’re thinking about city council, when we’re thinking about universities, when we’re thinking about, communities have got to become part of the conversation for us to have, the kind of global governance that’s required, it cannot only happen at that level.

It must be, a bottom up, you know, a ground up approach. We’ve got to have these conversations happening. We’ve got to ensure data and AI literacy are critical to the ways in which we are thinking, when we think about that, that is so important. I think what we have done really well is that we have built these brilliant communities of practice around data, around AI.

So, I think, when we think about many of the scholars in this space, the ethicists, the technologists, the leaders, public policy, there’s a passion that we are all bringing to this because we want to see it done well. But we have got now to find ways in which we can distill these conversations from the C-suite and the boardroom to Main Street.

And that is what the challenge is. How do we lift up, you know, communities and, and, and members of the public, how do we provide the right kind of public information, public education? And how do we include these voices in public oversight, which is now even more critical when we’re thinking about the advances being made with data and AI.

Satwik Mishra

You also speak like getting into the practical aspects of building this from the ground up, from high level conversations to local conversations around this and empowering people, at the at the lower levels where these systems are deployed, like you also use this term, which you called “Ethical Resilience for Professionals”. And tell us about it. How do you cultivate ethical resilience in your work specifically with data and AI and how would you encourage professionals to do the same?

Prof. Renée Cummings

So, I always say that there is no perfect data set. So, there is going to be no perfect algorithm. When we think about data, we know that data are people. Data are everything that we do our smiles, our eyes, you know the ways in which we are thinking, our likes, our dislikes we understand that. So, I always say that we have got to be ethically resilient, because so much is changing in this space.

When we thought we were governing AI, then we got GenAI. Now we’re thinking about Agentic AI system. So, every day, every week, every month, it’s something new when something often, more exciting, but something that also needs critical thinking, something that needs a sophisticated understanding. So ethical resilience means that we have got to be resilient, intellectually resilient in this space.

We have got to understand that the ethics questions and the safety questions keep changing, keep evolving and that is very important. And we have got to state ethically vigilant, which means as data scientist, as technologists, as policymakers, as leaders, we have got to continuously try to be one step ahead of the game. We know that it is said that the law is always behind the technology.

We’ve got to move away from that and we’ve got to start to prepare. We’ve got to think about the future of law. We’ve got to think about the future of just every discipline, every industry and we have got to bring the kind of risk management that is also futuristic, not reactive. At the moment, we’re doing very reactive risk management.

We’ve got to do proactive and futuristic risk management, which really creates, you know, what these hypothetical situations and scenarios could be. And what would be our response? So, it’s like, red teaming, you know, 5.0 is where we need to be at this moment.

Satwik Mishra

I agree, we need to do more of a scenario mapping and prepare as this technology just accelerates at a speed that we’ve probably never seen before in our careers.

But we are in towards the end of this year, and it has been a pivotal year in many ways, for a lot of reasons in the AI space. Where do you see some industry wins and government wins across the world, whether in the United States, or anywhere else where what would you identify as a place where we made some progress, either in an industrial ecosystem or even for governments?

Prof. Renée Cummings

I think we made an extraordinary amount of progress. When we think about 2024, when we think about what’s been happening globally, a lot of traction and globally when it comes to data governance and AI governance countries all over the world, big and small thinking about national AI strategies, thinking about data protection and data privacy. We’ve had a lot of conversation, from the global majority when we think about, what’s been happening, let’s say in India, in Africa, in Australia, when it comes to online safety and child protection, really forward-thinking approaches there.

When we think about the U.S., we have had a many, dynamic discussions in this country. When we think about what’s happening with AI safety, and we’re seeing that we’re coming together on questions around AI safety, questions around ethics. We know a few years ago, we got our blueprint for our AI Bill of rights. Then we had our executive order, and this year we had several federal memos really charting an ethical, responsible, and trustworthy course for the ways in which we interact with this technology and the ways in which we manipulate the technology for a full benefit of humanity, as opposed to the technology manipulating us all the time.

So, there’s been extraordinary progress. There’s been major growth, there have been brilliant conversations, there have been event that I have been involved in. I’ve spoken at I’ve participated in panels where we really try to raise consciousness and really try to come up with very practical approaches. Well, that’s not enough. We’ve got to do so much more.

And this call of action should really be a call now at this moment to have the broadest spectrum of communities globally, really having conversations about AI and about data. It’s about civil rights. It’s about civil liberties, it’s about human rights. It’s about equal opportunity, equal access, the ways in which resources are going to be divided and the ways in which we can engage in the future.

So, it is really, really important because this is a very, very powerful technology that is already creating some inequities and some challenges that we are struggling to deal with. So, we need all on board because, I think 2025, this ship is going to be this AI ship is going to be on some rocky, rocky waters.

So, we need to have all hands up there on deck with us.

Satwik Mishra

I started off with, yes, this is a very big collective issue that we all need to put our hands on. And you’ve shed light on it beautifully. But there’s also the other lens on the micro, on the individual agency which this technology enables. And one of the things that you speak a lot about are creative processes. And in creativity, how do you see AI influencing creative processes and in our creative processes, what are the ethical implications of AI’s role?

Prof. Renée Cummings

So, AI and creativity, the intersection of both, absolutely exciting for me. When you think about the imagination, when we think about the possibility, and now we have the ability to combine our imagination with this technology. So, there are two challenges there.

There’s a school of thought that believes that the technology is, making the imagination muscle very weak because many people believe that imagination is independent of, of this technology and there’s a school of thought that believes that collective augmented intelligence and collected augmented imagination is critical to the future. My approach is that we have got to ensure more and more individuals.

and communities play with this technology. This technology has the ability to really advance, so many of the challenges that we have had as a society and to do things in ways that we, have yet to imagine, you know, many moons ago, I worked in, disability rights and, I worked as a rehabilitation counselor.

And I remember thinking in the 90s about questions around accessibility, working with clients who were visually and hearing impaired or who had other developmental challenges, people who were neurodiverse and thinking about neurodivergent communities and what we are seeing within the realm of accessibility, just from a technical standpoint where individuals are able to do things that we never imagined, that’s that’s brilliant.

But then there’s the question of access and who gets to play with the technology, who gets to experiment with the technology, who gets to design and develop and deploy the technology? Those are the questions. And when we think about, the creative industry and many, creatives thinking, about the impact, the long term impacts of this technology when it comes to the future of their jobs and this technology actually taking away jobs from creatives and many of the lawsuits that we continue to see when it comes to intellectual property, when it comes to copyright issues, we’re seeing that, IP law are really taking, you know beating from the kinds of cases that are being presented and our, our judiciary and our criminal justice system having to stretch your own imagination to really understand the impact of large language models and frontier AI and Agentic systems.

So, this is the time for creativity in every discipline. And this is a time for us to even have a deeper understanding, again, of how we can use this technology, the kinds of new creative approaches that are being developed, the fact that we never thought a prompt engineer would have been a profession, because those who are more creative come up with the most creative problems and the most creative prompts give you the most creative kinds of outputs.

So, we have got to really reimagine imagining in the world of AI and that’s something I’m very committed to when it comes to the imagination, when it comes to play and when it comes to, the radical imagination that is now required, for existing in a world of AI, GenAI and Agentic systems.

So, we’re built for this task, you know, the human brain is what has created AI. So, when we think about imagination, for me, we have the ability to just imagine so much more possibilities. But an ethical imagination, an imagination that understand the, you know, the context of this technology and why accountability and transparency and explainability and all of these things are so important as well to the process.

Satwik Mishra

The reimagination of imagination that’s so wonderfully put, but also so while we talk about this inclusivity and, bringing more people, to play with this tool to realize its potential in a lot of your work you’ve developed and a lot of accessible pathways for people to participate meaningfully in this entire process, right from the design to the development to the deployment.

I want to get to a question that I think a lot of us think about very deeply, and I wonder if anyone has a very comprehensive answer. Every answer is right in its own way. But how would you define meaningful participation and like from your work, what are your key takeaways from building inclusive spaces such that we create what you talk about ethically resilient AI systems? What does meaningful participation in the world that we occupy look like to you?

Prof. Renée Cummings

I think it looks like access, it also looks like equity, and it also looks at the structures that are required to protect. I think if we don’t have access to the technology and we don’t have the creative confidence to play with this technology and understand how this technology can be used for the kinds of solutions that communities, societies, you know, globally, we will require, then you’re not a part of this movement.

So that access is critical, resources, also critical. We know that across this world there are billions of people who do not have access to the internet, who do not have access to this technology, who have, probably never engaged, with its technology. And that is also critical when we think about resources, when we think about the infrastructure that is required.

And also it’s about, the exposure as well to those ethical questions because when I think about data and I think about AI, I always think about data rights, and I always think about algorithmic justice, because the other part of that exposure is understanding although you may not have access and you may not have the resources, you already part of a large data set. AI is already being used to make decisions about you.

AI is already a part of your future. So, I think for me it is how do we broaden the access, how do we share those resources, how do we engage all communities and particularly when we’re thinking about young people, for me, it is a great concern that some groups may have more access than others, denying other groups and opportunities to thrive.

One of the things I always speak about is legacy and when I think of an algorithm, I always think about this as, algorithm can deploy a legacy because it either creates or denies it creates access, denies it, creates resources, denies it, creates opportunities, deny it. So, we’ve got to think, when we’re building these algorithms, questions around agency and questions around self-actualization and questions around identity development.

And this is why accountability and transparency and explainability and impact assessments and auditability are so critical to the work that we’re doing within the space of AI and data, because, to get that kind of interdisciplinary participation, we’ve got to think about questions around intersectionality, and we’ve got to think about how do we create the most open access to this, this critical resource called AI and of course, data.

Satwik Mishra

On intersectionality and the need for interdisciplinary lenses for these systems. I know that you currently teach a course on how to communicate effectively about ethical challenges in data driven technologies. While if we recognize the importance of meaningful participation and you shed light on it so beautifully, what would you think are the lessons for this communication, for this technology which is so important. What are we doing right in our communication strategies to inform society from industry, from governments, from academics, from organizations like Centre for Trustworthy Technology, or from any of the people working in this space

What are we doing right in communicating effectively the stake which we are operating at today, and what are the places that we need to work more effectively on?

Prof. Renée Cummings

I think what we’re doing right would be those conversations around ethics, safety, governance. We’re doing that really well, and I think we’ve built an extraordinary amount of momentum around the things that we need to do when it comes to really looking at research around AI risks when it comes to really looking for common testing practices when it comes to developing and promoting the kinds of protocols for for global inclusion and for evaluative processes.

You know, we’re doing that well and from that geopolitical level, we’re doing that. To me, I mean, there’s always room to do more, but I think we’re doing that very well. Our challenge continues to be, how do we build these conversations from the ground up, how do we communicate to just everyday people the kind of social responsibility that’s required at this moment

Because people are using this technology, people are engaging with AI systems and many individuals continue to find themselves in, in really challenging places while we think about criminal justice, when we think about health care, when we think about public benefits, when we think about, you know, even the financial sector, the housing sector, we’re seeing those biases and discriminatory practices there.

So, I think when it comes to communicating, we need to really bring together, the communities and bring them into this conversation. You know, I’m, I am so forth at it and so committed to that. But we also got to understand that many of the things that we’re communicating, creating these binaries, you know, the doomsday, versus whether or not, you know, the, the evil, versus the excellence, you know, we’ve got to understand this technology is a tool one

It’s something that you can press into service. And it’s also a style of communication. We now communicate using data. We communicate using AI. As there are many languages in this world, a new language that we are using is this language of data and this language of AI. So, it comes back to the literacy in real time. We need to upskill all of us globally in this new communication style.

And of course, we’ve got to ensure that people have the skills to use the technology for it to deliver the efficiency and the effectiveness and the excellence that it can deliver. But we also need to give that other critical education, which is around the ethics, which is around the bias and the discrimination and the risks and of course, we need to really lift up that conversation on rights.

And I think our legal system, or our judiciary have got to also upskill themselves in real time, because there are things that need to be said at the level of the judiciary. That is so critical. There are things that need to be said at the level of world leaders that they need to be a little more direct in articulating, those risks and of course, really encouraging people to engage and really looking at who can play with this technology and who cannot.

Satwik Mishra

Back in engineering more than a decade back, I had this professor tell me, one of the biggest things you should always fight for is indifference, not passion. People will disagree with you, but as long as they’re passionately disagreeing with you, it’s fine. And I think effective communication in this space is so important, so that nobody becomes indifferent and realizes the value of checking into the space and gets involved.

So, Prof Cummings, this was absolutely beautiful, my final question to you is, what is your most hopeful vision for this inclusive and trustworthy AI future? What are the practices that you really hope we will be able to embed and adopt in let’s say a 2-to-5-year horizon.

What are the practices you see embedded and you think that we would have done some good work, in building an inclusive and a trustworthy AI future?

Prof. Renée Cummings

Well, I will say this. I cannot even imagine what AI is going to look like in five years. I think we can’t even imagine two years. But my hope is that we truly commit as a world to ethical, safe, responsible, and trustworthy AI.

 My hope is that we understand to truly reap the rewards of this technology, we have got to protect rights, and we have got to evaluate and respond to those risks. We’ve got to provide the requisite systems to deliver redress, to ensure that there are reparative approaches to ensure that there are restorative approaches as well. We’ve got to also look at the resources and look at the environment and ask ourselves, do we need all that we are creating?

You know, when we think about this technology, when we think about the kind of capacity we are trying to deploy, “Do We Need It?” is a question we’ve got to ask ourselves and what is the cost? Not only the financial cost, but the cost on humanity, the cost on us, the cost and on our future? And what is this technology doing when it comes to our legacies?

And one of the most critical aspects of understanding AI is understanding the legacy it deploys and understanding dreams. I’m so committed to dreaming. And, you know, now we dream in a, a datafied world and my approach is that we must never allow data to defer a dream. We must never allow data and AI to undermine a legacy.

So that is why data equity is so critical. That is why equity is also very critical. And that is why we have got to understand how important AI and data governance is. And we’ve also got to understand that we have always, got to, to protect, some of those things that make us truly human and in protecting those things, that means that we understand why we need to protect each other and that is so critical to this work.

Satwik Mishra

Professor Cummings, to building a trustworthy future and creating a legacy that we all will be proud of. Thank you so much for participating in Trustworthy Tech Dialogues.

Prof. Renée Cummings

Thank you so much. It has been wonderful conversation. Thank you.

Satwik Mishra

Welcome everyone to another exciting edition of Trustworthy Tech Dialogues. I’m Satwik Mishra, Executive Director for Centre for Trustworthy Technology. In this episode, we will dive into the power of designing data architectures that have the potential to shape a future that is both trustworthy and innovative. Today, industry has emphatically embraced the transformative power of data to drive emerging technologies and unlock their full potential.

But is the public sector keeping pace? Governments are pouring resources into AI and other emerging technologies, aiming to transform the economies. However, there exists a challenge in effectively monitoring and measuring the spending. To truly understand its impact, a paper released by IMF this month brings out a stark analysis. In the 1980s, the total U.S. R&D investment was 2.2% of GDP.

Today, it stands at 3.4% of GDP. Conventional economic models suggest that this rise in R&D spending could have spurred faster economic growth. But contrary to expectations, the economy has experienced a slowdown instead of an anticipated growth. Between 1960 and 1985, the productivity growth averaged 1.3% annually. In the following 35 years. Productivity growth constantly fell below this historical average.

That brings forth the question how can we ensure that these investments are smart, strategic and backed by data and evidence to truly make a difference? Where is the public sector excelling? Where does it need to improve, and what changes are necessary to stay ahead? Today, we dive into the transformative potential of building data architectures which can ignite innovation, inspire fresh ideas and power economic growth.

We’ll also analyze the impact of artificial intelligence on the economy and the future of work. This remains a hotly debated issue, with predictions often pulling in opposite directions. Some argue that a productivity boost is on the anvil. Others warn of a possible hype. Some say we are headed for a jobless future, while others argue that AI will surely elevate the quality of work.

Increasingly, it seems that when it comes to AI and future of work, the only thing we can truly count upon is uncertainty.

That brings about another question, are we capturing the right data to truly understand what’s happening in our economy? If yes, what does this data actually indicate? And if not, what do we need to do to get the answers that matter?

To discuss all of this and more? I’m delighted to be joined by Dr. Julia Lane. She is a renowned economist and statistician who has brought her quantitative skills to public service. Dr Julia is a tenured professor at New York University’s Wagner Graduate School of Public Service. She recently served on the Advisory Committee on Data for Evidence Building and the National AI Research Resource Task Force.

She currently serves as the Secretary of Labor’s Workforce Innovation Advisory Committee and the National Science Foundation’s Advisory Committee on Cyberinfrastructure. She’s also the author of a fascinating book, “Democratizing Our Data: A Manifesto”, which I highly recommend to everyone. In her extensive public sector experience, she has developed innovative and practical approaches to design in varied data infrastructures. Let’s dive into the potential of building optimal data architectures. To help us navigate today’s uncertainties, enable smarter decision making and as importantly, advance societal trust.

Dr Julia. Welcome. 

Dr Julia Lane

Thank you so much. I’m thrilled to be here.

Satwik Mishra

Walk us through your journey from the quantitative background in economics and statistics to the world of public service. How did you come here?

Dr Julia Lane

I was really interested in answering the actually partly the same questions that we still have, which is what is the impact on investment on the job training on the economic productivity, the earnings of workers and the survival and growth of firms. So, this is 30 years ago. You can’t answer those questions with survey data because most workers work for big firms. But most firms are small. And you can’t get information about workers over time and firms over time. So, the only way to do it was to develop new sources of data.

So, I thought, okay, let’s go ahead and try to do that. That turned into a long journey. I created the LEHD program, Longitudinal Employer-Household Dynamics Program at the Census Bureau and that integrated data infrastructure in New Zealand, as you can tell from my accent, I’m a New Zealander.

And what I found was this foundational investment in data is way more valuable than writing a paper that maybe half a dozen people read, or even writing a book that thousands of people read. But that was really what hooked me into investing in data. That massive gap for firms, for businesses, for government, it’s a public good, and you can make a huge difference by investing in it. But that’s how I got into it, just by chance.

Satwik Mishra

I do want to dive into all the current affairs that are happening, and I would love to know your views on all of them, but I’ve spent the last two weeks reading your fascinating book, and I don’t want to, get into current affairs before getting you to speak about the book and its key ideas.

Give us a snapshot of, and I wish we could do an entire podcast on that book itself, but, give us a snapshot of that book’s recommendations and what is that vision for democratizing data?

Dr Julia Lane

The book really came out of a sense that despite the enormous potential, when I first started the LEHD program at Census Bureau, which was in the mid-1990s, you could see the world changing potential from data, from moving from surveys to new types of data.

I went from census to the National Science Foundation; I could see what the computer scientists were doing in security and access and converting texts was in 2004. So, I thought, oh my God, we could just securely make data available, we could democratize data. There’s all this potential that would galvanize the statistical agencies.

After 20 years of pushing and saying and opening the doors that the federal government couldn’t move and didn’t. It was still the same old stuff and I was brought to the White House to work on the federal data strategy.

And I was having a series of conversations with people, and they were saying, it’s never going to change. There are all these wonderful people. There’s a lot of money getting spent, but nothing is ever going to change. So, what the book is saying – “What are the consequences of not changing?” What does that mean for the people of the country when they don’t have the money to make decisions?

Not the big companies, they always kind of have the money, but economic intelligence, the small businesses, the students who are making career decisions, the policymakers who are trying to make informed decisions, one what are the consequences and how do we get out of it? How do we reimagine public data? So, that’s the notion of the book.

And the recommendation is very much the hap build within the government. You need an independent, trusted, data driven resource that is complemented by high quality research but driven by local needs. So, you get finely actionable information that can be used proactively, not just reactively. So that’s the book in a nutshell. 

Satwik Mishra

There are so may practical recommendations in that book. But if you are to look at today’s environment, give us one example around how informed, granular or dynamic real time data would have helped make better public policy decisions. What is that one aspect in the economy, you feel like if he had better data right now. I know there would be so many, but we would make better public policy decisions.

Dr Julia Lane

Well, I’m not going to take one. I’m going to take two. So, one is COVID, and I’m going to give you a concrete example in just a minute. But then the second is the point that you opened up with which how do we react to this and better manage the investments in R&D, which we hope are going to stimulate the economy. So, can I do the quick thing on Covid first and then the R&D?

Satwik Mishra

Absolutely.

Dr Julia Lane

The COVID example is a classic one. COVID hit and, we had this massive impact on the workforce. People went home. They were getting laid off.

How you went from .5% of all people, or 2% of people going and asking for unemployment insurance, to 25%. The workers who were administrating unemployment insurance were sent home. They didn’t have access to the data. A lot of them had the data on their physical machines. They didn’t have laptops, they didn’t have access.

They didn’t know how to produce information to the governor’s offices and the workforce boards at a local, granular, timely level so that you knew who was getting hit, how to get help to people and infrastructure laid bare, the inadequacies of a system that was designed a 100 years ago. Now, I had set up the Coleridge Initiative so, the states that had put their data into a secure environment, informed by my NSF experience on cyber security and so on, and they were able to get access to the data, produce dashboards that could be provided to the workforce boards and the governor’s offices. Illinois was the leader in this. They had they got the data about the layoffs and the unemployment insurance claims on a Monday.

The data were in the summaries, the tables, the dashboards were in the governor’s office by Thursday. That is transformational to answer your question, timely, local, actionable. So, take the same basic idea we had. I was on the National AI Task Force, asked to figure out what we need to spend money in order to do better industrial policy, essentially, what are we going to spend it on?

And the task force was asked to say, what are we going to spend? And of course, people were saying compute, data training and so on. And I said, how are you going to evaluate that?

Exactly the question you opened up with. We don’t have an infrastructure to answer those questions, proactively not reactively. You actually said that 20 years later, like with NAFTA, the North American Free Trade Agreement, 20 years later, we didn’t have enough data on there were these deaths of despair from the middle parts of the country that lost their jobs.

How do we avoid that?. So, as part of that national AI task force, I said, what data do we currently have? The data doesn’t make sense. The measures of AI that were put out, for example, by the Stanford AI ethics, which is what the committee relied on. They’re changing their ways, is based on Bibliometrics, which doesn’t tell you anything about the workforce.

In the MIT press piece that I circulated, and you’ll probably provide a link to it. The Census Bureau provides an estimate of 5%. What are the consequence of that 5% of firms who are using AI?

So, what are the consequences of, of not knowing the answers? Well, let me get very concrete. Since you opened up on productivity and the differences at MIT’s Daron Acemoglu will think that AI will increase by GDP by 0.93% over the next ten years.

Economists at Goldman Sachs say AI will increase GDP by 6%. Our task force had to figure out which one was right and where does the money go. In that order of magnitude between those two estimates is 1.6 times larger than the world’s third largest economy, Germany, over the decade. So, these are not small amounts. These are really, really big amounts.

And it means part of the issue with this is one of the questions is that is knowable. The data could be created and figured out, what proportion of firms are using AI, what proportion of jobs are going to be replaced, what types of new jobs are gonna be created and finally, how skilled is workforce and how can it be retrained?

All that’s knowable. So, building an infrastructure that can answer the question of what is the impact of R&D investment on productivity and the workforce not currently known. Big differences. But knowable and in relatively short time period.

Satwik Mishra

On this. This is a perfect segue way to, speak to yours industry of ideas, which is such a fascinating idea again and I’ll link again that paper in our show notes. Speak to us about the industry of ideas, not only its potential, but also where do you see it making a difference, and what do you think are its pitfalls or challenges which may come about if we are to go down that path?

Dr Julia Lane

So, here I am on the NAIRR task force. We’re trying to figure out where should, we were charged by the President, Congress writing a report on where to invest. Many economists say “you need data to decide”.

You don’t just make big numbers up. So think about it. AI quantum computing or not, all of these new areas of investment and industrial policy, none of them has an industry code. None of them is a scientific field. So, if you can’t even measure what you’re investing in, when will you get it? So what you measure is a management saying what you measure is what you get.

So again, I’m an economist. You think, okay, how do you think about what these terms mean? And really it is a new form of economic organization. So how are we going to describe that. And I’m going to argue industry of ideas. And let me tell you how I got to that piece. A 100 years ago, which by the way, as I said before.

Satwik Mishra

Can I just intervene just one. One sort of quote from your nature article, which I found fascinating. And then we can absolutely get into this idea, which will set it up for the audience. So, in the nature article, you wrote, which we’ll link in the show notes, the North American Industry classification system was modified in 2022 to add a single category for AI Activities, AI Research and development laboratories. In February, Adam Leonard, the Chief Analytics Officer of The Texas Workforce Commission in Austin, applied this new NAICS classification to Texas data and found a mere 298 AI research and development firms employing just 1021 workers in total, real workforce,

Dr Julia Lane

Entire state in the entire state of Texas

Satwik Mishra

Exactly. The real workforce involved in AI related activities, meanwhile, is likely to be much larger and spread across multiple industry sectors, ranging from hospitality and healthcare to oil exploration and more. So yes, what is the industry? Let’s segway into the industry of ideas.

Dr Julia Lane

And so, thank you for jumping in right there to make a very concrete. So, 100 years ago, we classified firms into what they produced, and that was mainly manufacturing and agriculture. So, we had thousands of manufacturing, agriculture industries. Then 50 years ago, it was, we’re not just producing physical goods anymore.

So, we need to change from SIC codes to what you referred to as NAICS codes, NAICS codes that are based on services. So, how things are being produced. That was 50 years ago, and it took ten years to get those categories. Now, as I’m looking at it, those new industries – AI, quantum computing, these are industries of ideas. The basic idea, the innovation is ideas, new ways of putting things together.

As Heidi Williams has pointed that innovation is what breaks the shackles of scarcity because you no longer have to just invest in capital, labor, energy, material services which we have. But if I have a better idea, you can put the same inputs together in a different way, like Steve Jobs did with the iPhone. That was nothing new. It was just pure innovation.

There are so many things, that’s what AI is. So how do we track those ideas? the reason that the industry of ideas is such an important one is ideas are embodied in people. It’s not things, it’s not services. Its ideas embodied in people. So, the whole idea between investing in R&D is investing in great ideas in great people.

So, and not just to give you credit, Paul Romer got the Nobel Prize for this Endogenous Technological Change back in 2018. He called it the “Economics of Ideas” that you think about that is, ideas don’t get consumed like capital. Right. Physical capital that one of the key ideas that comes out of that is, first of all, that creating a market with incentives for people to share ideas is massive because you have a great idea.

The more you share it, the more impact it has. Unlike steel when I consume it, it’s gone. The second thing is, going back to your opening statement, which is then you need to figure out where to invest in people. That has demographic consequences, geographic consequences, workforce consequences. And then it tells you how to build the data infrastructure that has been being built over the past ten years, about which we wrote the Nature piece.

So that’s the key point here. It’s a new economy. Firms are organized around ideas much more than services and industry. And so, the data infrastructure needs to reflect the economic organization.

Satwik Mishra

So, from the industry of ideas, give us an analysis of the issue, which everyone’s debating right now. What would the future of work look like? How would you think about the measuring and monitoring of how jobs are changing?

What needs to change in our measurement of jobs so that we’re able to analyze artificial intelligence and impact on our workforce, our industry of ideas translate into that measurement?

Dr Julia Lane

My first answer is going to be you don’t send out a bloody survey that no one’s going to answer. But, you think strategically about how are you going to trace that through digital fingerprints? Right. So, what has been being done on AI is the first thing is the shock to the economy is coming, as you just pointed out from R&D spending.

That’s the first thing that they’re going to need to figure out. How do we track that? Well, that part is straightforward, because when a government or a grant, a contract is issued, there’s a code that’s generated “Abcdefg”. When that contract hits a firm or a university, I’m going to focus it on universities for right now, a code gets created within the university or within the firm.

Everyone who charges time to that project is captured, and so are the vendors who are paid. The people who are producing new types of computers, the GPUs, the Nvidia’s. All of that is tracked in the HR and finance system. So, immediately you have a data infrastructure that describes the scientific production function and who is working on these new ideas.

And this is what we talk about. Now, how did those ideas that are in the university get transmitted to the private sector? Well, it’s not just the principal investigators.

Our research shows using these data, which have been built up at the University of Michigan, the Institute for Research on Innovation and Science over the past 15 years. So, this again, and it goes back further because universities were able to pull it out.

So, it goes back 20 years. We show that there are 22 graduate students, postdocs, undergraduates, clinical researchers, the every principal investigator. You can see those people. Those are the sparklers like Larry Page and Sergey Brin. Those are the sparklers that go out and create new fire into the economy. So, what those people were taught because they described it in their proposal.

The funding agencies know what’s going on. You can then track that. You could see the firms that they landed. Many of them get hired in at high wages because they’re expected to create that spark and then you can see what happens to the jobs in those firms and the skill contours of the workers in those firms, because that skill information is available from additional sources, both the links and to the educational information from the public schools, as well as what’s hiring information with the job vacancy data.

That immediately tells you, its local, it’s timely, it’s actionable, and it enables you to figure out proactively what’s happening to the workforce and the hiring patterns of firms that are affected.

Satwik Mishra

Absolutely. So, this answers the question around why the industry of ideas. Another fascinating piece of intervention that you make in your writing is, answering the “How”. And you speak about your proposal for a Centre for data and evidence.

The institutional intervention, which is, if I may say, which is required in our data ecosystem. Walk us through, what would what is that proposal for Centre for Data and Evidence? Why is it important and what do you think should be the structural arrangements around it?

Dr Julia Lane

So, I think what’s important is getting new eyes in to solve these problems. There are so many smart people out there who are already cited MIT people, but also people in other institutions throughout the country. We shouldn’t just focus on the top institutions that need to be many voices. So, the basic idea is to get new ideas in, there can’t be the Federal Statistical System. Just to be clear that survey methodologies and statisticians are not the ones with the ear to the ground.

It’s the researchers, the local researchers, the people who are on the ground trying to provide services, community colleges and so on. So, the questions need to be driven from outside. That centre should be independent, nonpartisan, trusted think tank, but a different think tank from the ones that we historically see it needs to be, grounded in the idea of bringing together data from multiple sources to make sense of it, just like IRIS has done, for example, at the University of Michigan.

But many other things like the multi-state data collaboratives, that have been set up by the NASWA, the National Association of State Workforce Agencies. So, people who know how to analyze data and people who want to understand why it needs to be built on top of an institutional infrastructure. So, you’re not building castles on sand. You’re building a firm foundation.

What does that look like? In what I’ve argued is philanthropic foundations are going to be central to getting that set up. They’ve been very successful in setting up institutional infrastructures in the past, but they have to be pure of heart.

You could also have funding lines that go to the states through federal agencies that also want the answers to the questions. Department of Labor, Department of Education, Department of Commerce, Department of Defense, Department of Energy. All of those could provide funding to this independent centre, but the centre would be demand driven, agile, and had the right incentive structure.

Satwik Mishra

We answered the why, we answered the how. All from your writing. But there’s an old piece of yours which I went through last week. It’s a piece you had written in science and technology, issues in science and technology where someone who advocates, as you do most importantly for building a robust data ecosystem, you also identify three challenges of relying a little too much on data and being wary of that.

And I’ll give you the headers, and I would love for you to expand on them. The first one is what you call “Campbell’s Law”, which I didn’t know about but is very interesting to hear. The second one you speak about the lag in investments in things like science & technology and education. That is something that I’d love you to explain the readers.

Why that lag happens in science and technology and education? And thirdly, obviously on privacy and confidentiality. So, expand on a little bit on the challenges on all three of these challenges and what what do you see? How do you analyze these three, particular challenges?

Dr Julia Lane

Why do you start with number one? Why don’t you spell out your take on that law that you’d never heard of?

Satwik Mishra

So as much as, for a novice in stake, I understood Campbell’s law, as the more we rely on a quantitative analysis or to make decisions the more they will be proclivity from society to possibly corrupt that ecosystem because of decisions are made from it. There will be a tendency or a proclivity to corrupt that ecosystem.

And I found that like, it was almost like a light bulb moment when I read about Campbell Law. But this is as I understood it. But I would love for you to expand a little bit more on it. How do you think about it? How do you envisage getting around something like that?

Dr Julia Lane

So, and I wanted you to say it so that I wasn’t I, I feel like if I talk all the time, I’m talking at people. So it’s helpful. So, I’m going to ask you to also summarize the other two. But once I go through that, so, again what you measure is what you get. So, the challenge is coming up with a measure that can’t be gamed.

And people are really smart. So, anytime you set up a measure, you’re going to get gamed in some way. The advantage of investing in ideas and people is harder to game that, right. You immediately switch the attention away from mindlessly producing bibliometrics and papers and patents. You’re swamped and the stuff to really investing in your students and your trainees and pushing that out.

But that can still be gamed. So you just have to pay attention, basically, and change, if you’re getting the wrong results for not the wrong results. But if you feel like there’s bad actors in there, just be aware and change, because people are always going to figure out how to mess things up.

Satwik Mishra

So yeah. And the second one, which is investments in science and technology and education, take time from the time of investment to its impact in society. And it’s, we need to make space for assessing that complexity and almost acknowledging that complexity of those investments.

So, we need to make space in a data ecosystem, walk us through that risk as well.

Dr Julia Lane

So, it comes back to a very basic concept, which is what is your theory of change and what kind of lags do you think?. You’ve got these inputs which are the spend a new immediately in the introduction you said, well it was 2.4.

Now it’s 3.2. So, everyone focuses in on the inputs and the Lisbon protocol said every European country should spend 3% and you’re like, well, why 3%? Why not 2.91 or 3.1? What’s the magic number? So, the theory changeṣ. Okay, we’re going to spend money. Also, the charge to the National AI task force that I was on.

What activities are going to be generated? What are the initial outputs, what are the outcomes and what are the goals? And then how are you going to measure at each step of the way and then, what are the compounding factors? Now for a lot of things, people are complex, organizations are complex. There are a lot of compounding factors.

And so of course, what scientist tend to say when they’re going to be evaluate it but it sounds so complex, you can’t possibly measure how wonderful we are because there are so many things that can happen. And there are two answers to that response. One is, this is why they call you welfare queens in white lab coats on the hill.

The second point is, if you think that ideas are random and serendipitous, you get that argument a lot, you don’t need to send money. If, however, you think, put it simply, why which is an idea is equal to X, which is money beta plus epsilon. Epsilon is your noise, The serendipitous part.

You’re going to say money makes a difference. Then you have to give me some idea of what beta is and the process whereby that beta and that X generate innovation. That’s the theory of change. And as a scientist, you should bloody well be able to write that down. That’s the core idea here. And yes, science is complicated.

So is investment in early childhood, so is investment in health, so is investment in many areas. But you have some idea. And as you go through the process you write, you learn, and you have a better understanding of how that money should be structured to be most helpful.

Satwik Mishra

And the third one the final one which is a little more straightforward. But leads me to my final question. The third one, around is the insufficient protection of privacy and confidentiality, including disclosure of intellectual property and national security risk could lead to a loss of trust and, more importantly a consequent disincentive to participate in sharing knowledge, which is so important.

Dr Julia Lane

This is what I’ve spent my life worrying about these things that privacy and confidentiality issues are massive. That’s partly why setting up secure environments are so important. What I learned from NSA about security is you can have multiple tiers of access. So, as long as your biggest risk of security is people sharing information.

So, figuring out how to indemnify individual behavior, which is what they do in the Defense Department, for example, is going to be critical. And that’s by the way, going back to the Illinois example, there was specialized customized tabulations that were sent to the governor’s office and the workforce board because they were trusted partners. So, you have levels of trust that are institutionalized.

There can be standards that can be set up around those that get that provide that trust. FedRAMP is an example that is being put in place so.

Satwik Mishra

FedRAMP is a fascinating example and how it functions. Right. Yeah. My final question is always looking ahead. As we said in 2024, with all of these debates happening around us as to what’s happening in the economy. What is the best case outcome, as you see going forward and what are the pitfalls that you worry about that we may need to work more on otherwise we are going down that path?

Dr Julia Lane

I’d like to see the centre set up immediately. It’s been 30 years, there’s lots of reasons kicking cans down the road. A philanthropic foundation or a set philanthropic foundation dedicated to the public good should set this up and get it done. All the pieces are there, join together, make massive opportunity that I don’t think we can miss.

So that’s the opportunity that I think can be capitalized on. What are the challenges? I think the incentive structure needs to be rationalized, it’s not publications and patents that rich publishers, it should be investing in people. Immigration is going to be a big issue because it is just a few ideas or you need as many of those bright ideas opened up as possible.

So, that’s going to be a major challenge.

In terms of the privacy and confidentiality, there’s been a big focus on things like differential privacy and synthetic data. The secure environment having multiple eyes on the data is what’s going to generate trust, not black box fake data. So. And all of those things can get addressed in this new set of the data and evidence.

Satwik Mishra

Dr Julia, from the book to the Centre for Data and Evidence to the Industry of ideas, all of these would be individual podcasts. And I could just keep asking you questions because it’s such a fascinating body of work. Thank you so much for being here. 

Dr Julia Lane

You’re a joy to work with. Thank you so very much.

Satwik Mishra

Welcome everyone to another episode of Trustworthy Tech Dialogues. I’m Satwik Mishra, Executive Director, Center for Trustworthy Technology. Today, we are exploring a topic that affects each of our professional lives, ‘The interface of artificial intelligence with human resources.’

In a world where AI increasingly interfaces with workplace operations, particularly in HR, the question arises how do we ensure that this technology is used ethically, responsibly, and with a clear commitment to civil rights to advance society trust?

Now, on the surface, AI offers tremendous benefits. It can streamline recruitment by efficiently analyzing resumes. It can remove unconscious bias in hiring decisions, and it can create more tailored, and data driven approaches to employ management. But with these capabilities come significant challenges and considerations. How do we develop AI systems that are human centered? How do we ensure that AI in HR (Human Resources) respect civil rights, promotes equity, and avoids perpetuating discrimination?

Trustworthy AI adoption isn’t just a technical challenge, it’s a deeply ethical one. It requires the collaboration of technologists, policymakers, business leaders and civil societies to ensure transparency in design, development and deployment. It demands a vigilant focus on safeguarding the dignity, integrity, security and privacy of data that AI systems are built upon. Because at the heart of every HR decision is a person, and that person’s rights and autonomy must be respected.

Today, we will explore all of this and more. We’ll talk about the importance of building AI systems that not only work for businesses, but also enhance fairness and protect civil rights within the workplace. And of course, we’ll dive into practical aspects like – What does AI mean for the future of HR? How can we harness this power responsibly? And what are the steps that we need to take to ensure that AI is used to empower, rather than undermine, the human experience at work?

To discuss these pertinent and timely issues, it is my absolute privilege to welcome the Commissioner of the U.S. Equal Employment Opportunity Commission (EEOC), Keith Sonderling, in today’s podcast. Commissioner Keith was confirmed by the US Senate with a bipartisan vote, to be a Commissioner on the US EEOC in 2020. Prior to his confirmation, he served as the Acting and Deputy Administrator of the Wage and Hour Division at the U.S. Department of Labor.

Before joining the Department of Labor, he practiced Labor and Employment law in Florida. He was a partner at one of Florida’s oldest and largest law firms, Gunster. At Gunster, he counseled employers and litigated labor and employment disputes. Commissioner Keith also serves as a Professional Lecturer in the Law at The George Washington University Law School, teaching employment discrimination.

Since joining the EEOC, one of Commissioner Keith’s highest priorities has been ensuring that artificial intelligence and workplace technologies are designed, developed and deployed consistent with longstanding civil rights norms.

Commissioner Keith, thank you so much for being here.

Keith Sonderling

Thank you for having me.

Satwik Mishra

All right. So, let’s dive in. I want to start off with a little bit about your journey at EEOC.

You’re currently wrapping up almost four years as the commissioner. Walk us through your path to this role. How were you introduced to the intersection of technology and politics?

Keith Sonderling

Well, it’s a great question. And to get there, you have to take a step back and learn a little bit more about my agency here.

So, the US EEOC, for all intents and purposes, we are the regulator of human resources. So, our mission is to prevent and remedy employment discrimination and advance equal opportunity for all in the workplace. So, when you think about some of the larger issues the workforce faces with discrimination, preventing discrimination, all the issues related to the #MeToo movement, diversity, equity, inclusion, social issues in the workplace, pay — That’s my agency.

So, you may ask, what is so significant about this agency and why we’ve been able to take such a global role in ensuring workers have protections related to not being discriminated against in the workplace, which are civil rights protections in the United States? That is because of our founding.

And this agency was founded out of the civil rights movement in the 1960s. So, Martin Luther King Jr. marching in Washington DC, led to the Civil Rights Act of 1964 that created Title VII of the Civil Rights Act, which gives employees, applicants and even former employees, civil rights protections in the workplace to not be discriminated against. Such as characteristics, their sex, national origin, color, race, religion, age, disability, pregnancy — all these very big-ticket items.

So, the United States was one of the first to create these civil rights protections in the workplace. Because of this very storied history, a lot of governments around the world have modeled their own employment laws and civil rights protections for workers in the workplace. So as the global leader, when it comes to these important issues, we really can set the pace for not only the compliance side, but also the enforcement side, because at our core, this is a civil law enforcement agency.

So, that’s a little bit about our mission and how we’re here. And I would like to call that the day job because, what’s also unique to US based employment law is that for most claims, you cannot just go sue your employer in court. You have to come to the EEOC first. Whether you work for a private company, state or local government, or even the federal government.

We see every case of employment discrimination for the most part in the United States. So, it really puts us in a very good position to see what those trends are and to see what the hot issues are. So now, to answer your original question, how did I become interested in the intersection of technology in workplace laws and workplace rights?

The answer is – when I got here, I really wanted to focus on an issue that was going to be proactive, focus on an issue that would help not only workers, but employers, be able to comply with these very complicated federal laws. And if you look at just at the recent history, the last 10-15 years of this agency, very much like HR departments, very much like corporate governance departments, we’ve had to shift what our priorities are, based upon what’s in the news.

So, let’s go back to the recession post 2008, there were a lot of claims of employment discrimination related to age discrimination. Because a lot of older workers were disproportionately laid off after the recession because they were making more. So, we started focusing on age discrimination. And then happened, the global #MeToo movement. And we had to stop everything and then focus on preventing sexual harassment, which of course has been illegal since the 1960s.

But that raised awareness. Then the U.S. women’s soccer team with pay equity became national news. Then, COVID happened. And everything related to vaccines and working from home and those accommodations were right in the center of our agency. Then George Floyd and the social movements within, then racial injustice within the workplace and a lot of those movements. So, we had a focus there.

So, I say this is that there’s always going to be something, for us, for companies on the compliance side, that is going to distract based upon what’s happening in the news. So, when I got here in 2020, I said, how do we get ahead of the next #MeToo movement? How do we start thinking about, proactively, what is the biggest issue for the workforce going to be? And how do we tackle that?

So, I started talking to corporate leaders and a lot of different stakeholders, both on the employee and employer side. And they told me that artificial intelligence in the workplace is going to be the biggest issue moving forward. And like many, especially in the government, I didn’t really understand what that meant. For me, it kind of took me to this dystopian vision of these robot armies replacing human workers.

And I thought, maybe that’s happening. You see that in manufacturing and logistics, some retail fast food where actual robots are doing the job. Well, here at the EEOC, we regulate every single industry across the board. So, I needed something big enough to tackle, and not just a couple industry specific things. But then when I started learning about it, I said, wow, it’s a lot more than just replacing workers.

Which is funny because four years later, after generative AI, now we are actually talking about replacing a different kind of worker with our knowledge workers, which we can talk about. Then I realized how artificial intelligence, machine learning, natural language processing, all these buzzwords I had to learn, were already being vastly used within the HR function, not to displace the workers, but to actually perform HR functions that HR managers have been doing in their entire career.

And to do that was really having software designed, developed and deployed to make decisions better than humans making them. Not just on being efficient, not just being more economical, but dealing with the biggest problem in human capital management, in human resources. And that’s the H (human), because a lot of it was based upon and still is. If we can remove the H from these decisions, if we can remove the humans from these decisions, then we will be without bias.

Then, humans can’t inject their bias into employment decisions, which we know have plagued employment decisions before, during and now where even though these laws are here and taking a step back again. In the last two years, this agency has collected $1.2 billion from employers for violating these laws. So, as entrepreneurs and technologists look at stats like that and they say, well, how can we have artificial intelligence, computers, do that better? And as I started to explore it, I said, yeah, if properly designed and carefully used, AI can potentially make decisions without bias. But then I also realized at the same time, if the AI is not properly designed, or if it’s not carefully used, it can scale discrimination to the likes we’ve never seen before.

So, it really took me to understand not just the AI industry, not just technology, but how impactful it has been, it will be and it is to our space. So that’s why we decided, and I decided to make this a priority for the agency. And a lot has happened since then.

Satwik Mishra

The EEOC absolutely has a fascinating history.

And like you said, with your term, I feel like building that proactive lens of getting ahead of the curve, you really embodied that. And it’s fascinating. And might I say you are being too humble when you say that you didn’t know enough, because I’ve read everything that you have published on the topic, and we’ve all learned so much from your work.

So, it’s absolutely fascinating. Let’s talk a little bit about your work at the EEOC now. As you say that over the last three-four years, the focus exclusively has been on artificial intelligence and the changes which have come about. And now you’ve focused extensively on the balance between the benefits and the harms of using this, at the workplace.

Walk us through the factors which go into weighing this balance. And secondly, it would be really interesting to know from your perspective what potential benefit of harm do you think is most misunderstood or mis-valued at the workplace right now?

Keith Sonderling

Not being a technologist myself and being in the executive branch and not in an agency, like Congress, that can create new laws, I’ve always understood the restrictions and the confines that we’ve had, and that’s how I knew we had to operate to be impactful. So, just taking it a step back. At the end of the day, there’s only a finite amount of employment decisions – Hiring, firing, wages, training, benefits, promotions, etc. And that’s what we regulate.

And that’s what we’ve regulated since the 1960s. And from our perspective to look at how to analyze these AI tools, don’t forget companies are making those decisions with or without technology. And they have been making those decisions, since companies have been in existence. So, in a sense, I try to simplify it and say like, we regulate the outcome of the employment decisions, whether you’re using AI to completely make that employment decision, whether you’re using AI to supplement or augment that employment decision.

There’s still going to be an employment decision. And that’s what we know best. And just that the employment decision is based upon the candidate or the employee’s ability to perform the job, based upon their skills and their merit? Or is it based upon bias, which is unlawful so that just, to simplify things, I always went back to the basics there.

And, when you talk about artificial intelligence making all those decisions in the HR space just so everyone understands, there’s AI out there related to every type of employment decision an employer is making. There’s software out there that will completely draft the job description, a job advertisement. There’s software out there that will do the advertisement or software out there that will collect the resumes, that will review the resumes, that will rank the resumes, that will determine which resumes to interview.

And then, of course, there’s software out there that will completely conduct the interview from scheduling to substantively asking the questions to grading the interview, and then software out there to determine whether or not to make an offer, how much to pay the person. And then once they accept the offer, there’s AI software out there that will look at your social profile and tell you where you should physically sit within the office, what colleagues you should interact with based on AI.

And then when you start working, there is software out there that will be your boss, that will tell you what your assignments are for each day. They will grade you. It will determine what your performance reviews are. They’ll determine if you get promoted, or demoted. And then there are AI algorithms out there that will tell you – you’re fired, if you’re not meeting expectations. So, this isn’t hypothetical. This is out there right now related to every type of employment decision that employers are making, and they’re making it with or without technology. So, when it comes to using technology for these decisions, we’ve historically relied on humans.

I’ve taken the approach that – Can AI absolutely help you make better, more transparent, more fair employment decisions based upon the ability to do the job based upon neutral metrics, based upon industry standards without the human bias?

Absolutely. Is it the best thing in the world? Is it going to completely revolutionize everything? Maybe! Is it the worst thing in the world? Is it going to discriminate against everybody? Maybe! At the risk of sounding like a lawyer, which I am. It all depends. And that’s the trickiest question. And I always felt here at the EEOC, we shouldn’t be in the position to tell employers not to use certain software, because if we are, then they’re going to rely on some of the issues that we’ve been dealing with – Humans – which has led to the creation of this agency.

The decision you have to make is what software are you going to use? What purpose are you going to use it for? What are your existing policies and practices for that? And when you’re using these tools, how are you going to comply with the longstanding civil rights laws?

And that is so complicated because every use of AI technology has a different analysis. So, if you want to use facial recognition in a zoom interview to determine a candidate’s score, whether how much they’re looking at the camera, how much they’re smiling. Grading that on a score, you are allowed to do that except for, in certain states, of course.

But is that going to give you the best candidate or will it potentially lead to disability discrimination because the person can’t smile, or national origin discrimination because it’s not culturally appropriate to smile in that situation, even though that would be the best employee. That is something you need to account for and assess.

And even with some of these programs that look for the best candidates, are they going to be sophisticated enough to deal with some of the very classic discrimination examples we’ve seen in the AI space and specifically employment saying – we can’t take into factor whether you’re a male or female and, you’ve seen cases where the vast majority of applicants are men.

And the AI thinks that must be the number one qualification for that job because there are so many men applying. That must mean something more than skill, more than the ability to do the job, and that will then play an unlawful factor in making a determination if the AI isn’t sophisticated enough to exclude those characteristics, that you’re not allowed to make an employment decision on and actually look at merit.

So much of it is what I said earlier. It’s the design side which the vendors need to deal with, and then it’s then the use side, which the companies can use, and have to deal with. Because in both of these examples, whether it’s the data, SAT discrimination or in the use of not being able to account for disability or national origin or color, if the vendor can then design the software to say, well, I’m not going to be able to have the artificial intelligence look at any of those protected characteristics, such as male or female. And even though 90 out of 100 resumes are male, the AI is not going to look at that. It’s going to exclude all that. It is going to mask all that, and it’s going to look what are the underlying actual skills that these potential men, who were the more qualified applicants in the first round, had? And then where do the females rank within that?

Where does everyone rank neutrally based upon that skill, which is how hiring works. Or in the other example, how are these tools, especially when you start dealing with natural language processing going to account for foreign accent? How are they going to account for disability to allow those individuals who are in that class to be treated equally, which is what the US law requires, ‘Equal’. It’s in our name. And that is so much on the design of the program that the vendors can do. And then on the flip side, you can have the most perfectly designed algorithm, you can have the most perfectly designed tool. And if you don’t have those governance constraints within your organization, if you don’t have that training, the vendor can do everything they’re saying.

And then somebody within your organization can use these AI tools now, that is faster and more efficient than ever before to make an unlawful hiring decision whether that’s including certain people based upon characteristics you’re not allowed to or excluding them. I say that in the aggregate, it’s difficult to just say one broad brush about if AI is going to revolutionize every side, everything in HR space, or if it’s going to cause significant discrimination, it could do both.

But that is on this community, whether it’s the vendors, the users, the employers, developers, all these different stakeholders, we have to come together to deal with all these factors because unlike using AI in other areas, reviewing documents, making shipping routes faster in employment, you’re dealing with what you said in the intro “Fundamental Civil Rights”.

So, if the AI makes a mistake here, you’re excluding somebody from the workforce based upon their religion, based upon their age, based upon their national origin, based upon a disability, etc., and not allowing them to provide for their family because of that at scale, I think that’s the difference here. And why leading with the workforce, leading with the HR from our perspective here at the EEOC, has brought us that attention because everybody gets it.

Everybody has entered the workforce, everyone’s had a resume, everyone’s had a boss and the issues that we’re dealing with related to this AI software is being used equally, whether it’s in the United States, whether it’s in the EU, whether it’s in Asia, it’s the same core issues.

Satwik Mishra

You’ve said this in several other interviews and in your papers, but one of the things is to look behind the wheel.

And it’s not just technology, it’s affecting people, it’s affecting different communities. And it could elevate them, or it couldn’t. Basis on what you said, it depends on how we design it and how conscious we are in our decisions, why we are deploying these technologies. But in your role, what you’ve done is also engaged, like you said in the beginning, engage with a lot of corporate leaders, engage in society, engage with community to understand what is happening, walk us through that approach.

Like how does the EEOC approach and engage in multi-stakeholder dialogue to work these critical issues, and how important has that been in the framing of what you have thought about the current situation?

Keith Sonderling

I think it’s probably the most important thing that I was able to do here related to the AI governance world, related to regulations, related to the use.

And let me let me just explain why. Because, again, you have to take a step back and look at our history. Since this agency has been around, we’ve had a very small world. I mean, a very big world. Every employer, basically every employee, I don’t want to minimize that. But our stakeholders were limited.

They were employee groups, employer groups, unions and staffing agencies that was everyone in this world and we never had technology that nurses, making or assisting with employment decisions. So, we’ve operated in this world of people who understand the issues here. But now you have that technology coming into play and essentially making those decisions about being a big part of decisions.

And then I learned that, taking a step back, who is this new world when it relates to tech and AI, especially in the workplace, it’s such an expansion of those stakeholders. So where does it start? From the very beginning. The VC money that’s going into this that is used to going into tech, I realized that they need guidance and tools of what technology to invest in because they’re not steeped in federal employment law, and they do not want to invest in a product that’s going to violate civil rights.

Then you have entrepreneurs who are extremely talented in actually making the technology, whether it’s from actually being able to code and make the AI or the business ideas to be able to revolutionize some of these hiring processes. And most of them, the common theme is that they are entrepreneurs. They’re tech savvy. They’re not HR professionals.

They speak a different language as well and they don’t want to build with the money they just received a product that’s going to violate civil rights laws. Then you have what I like to say, the group that is in the toughest position, is in the middle, and that’s the companies who are needed to make the decision of what products to buy, what products to deploy. on the front end, dealing with the vendors.

And now they need to speak the language and understand the language the vendors are speaking, and they’re used to buying off the rack SaaS software, which doesn’t work in our space because, the rights and requirements for each individual employee are different. And then, after making the determination of what products to use, they have to then determine how are we going to use this on our applicant pool? How are we going to use this in our workforce?

Ultimately in our space, which we’ll talk about, the company using the product is going to be the one that’s 100% liable for those decisions, the final stakeholder. Arguably the most important are our consumers. And who are the consumers in our space that’s different than other spaces? The consumers are the applicants. The consumers are your current workforce.

They’re you’re going to speak a different language and there’s a different level of trust required for them when being subject to these tools. When I realized that’s our world and everyone is speaking different languages, this is going to be a lot of work getting everyone on the same page, although everyone is on the same page about not violating the law,

You’d hope. We’re not violating fundamental civil rights. But in all sincerity, everyone was looking at it from different directions. For VC – how do we invest in the right products and how do we return our money? For the entrepreneurs – how do we make these companies very large, the companies buying it, how do we use these properly and have more efficient hiring decisions with less bias?

So, it’s taken a lot of effort to bring everyone together to have those discussions to see what all the different, unique concerns are. But more importantly, explaining how these tools are, how we’re going to look at these tools. And in federal law enforcement investigation, which of course is the fear of using all these products. Here is what we’re going to look at.

So many people wanted to say, this is so novel. We don’t understand how an investigation is going to work. And that’s where I’ve been trying to change the narrative and saying, look, right now our investigators are trained to look at employment bias. And how do we do that? By interviewing people by, unfortunately, subpoenas and depositions and litigations.

But if this AI chain can be done correctly through transparency, through accountability, robustness, all these words that I know you’re very familiar with, here’s what they mean in the employment space. What is your data set? for most people in this world, they don’t know what a data set is. But to simplify that saying, your data set, let’s say, it’s your current workforce.

If you’re going to be using AI, what are the dynamics of that? Is it imbalance there, what going to lead to an imbalance, result potentially on one side and how do you potentially justify that. If not? We’re just talking about the scalability here when you’re using AI and then if you’re ever challenged on an employment decision, instead of, having to retrace that now you may have a contemporaneous record that’s defendable exactly what you asked the algorithm to do.

Let’s just say, in a hiring case where you have AI help you with the job description, and you have AI do those things based upon industry standards, based upon your own standards. And you can then show that the AI was only looking at these qualifications that are necessary to do the job, and not the other factor.

That’s how we selected these applicants, we didn’t know what religion they were. We didn’t know what their sexual orientation was or pick any of the factors. Then you potentially have an undeniable record of a proper employment decision in real time. And that’s taking a step back to explain how then you can use that to disprove discrimination.

I think it really helps everyone understand saying – now I understand why these need to be properly designed and carefully used, now I understand why these factors need to be accounted for. Because if starting with the end of a class action lawsuit or any EEOC investigation, which everyone fears, if this is what they’re going to look at, well, I can show that and I can build an algorithm around that.

That’s going to account for that. And I think that was just a different way of thinking. I think most people have never experienced a regulator trying to be proactive, because it goes to my theme of why I got involved in this. It’s because not only do I want this to work, but because it can help eliminate bias if it doesn’t work.

The scale of these lawsuits, the scale of potential victims of discrimination through AI, is just far greater than anything this agency or courts had ever seen before, and that requires learning different languages and dealing with different stakeholders. Many, especially when you start looking at how the global community, had never heard of the EEOC, yet they’re creating products directly regulated by us.

Everyone has heard of the FTC on the advertising side, but that’s just the first layer. When you actually get to the core, when you get to the bias side, which is one of the hottest topics in AI. We’re the core regulators. So, it really took almost injecting myself in that conversation and then have a stake in it.

And once we did, everyone was like, no matter where you are in the world we want to get your side right as well.

Satwik Mishra

So, one of my former professors is Defense Secretary Ash Carter at the Kennedy School, used to always, tell us, just look under the hood. Look under the wheel.

Keith Sonderling

It’s not that complicated. If you’re able to build out the pipeline of decisions which are being made by technology, and if you can create out those structures, you’ll be able to navigate through all of this very well. But you managed to do was absolutely fascinating in the sense of and also you mentioned this changing the narratives, changing the narrative when you came in and took over and went ahead of the curve, tried to go ahead of the curve about what the current marketplace challenges might be.

I wanted to talk one more thing about changing the narrative, because, it gets into the Washington D.C. part of this where it’s just easy for people in D.C. to say, this is bad. And before generative AI, people started citing the benefits. If you look at some of the oversight towards this agency, sort of the original interest from Congress and civil rights groups about AI was all bad.

It’s going to discriminate, It’s going to cause issues for older workers, for disabled workers. It’s going to discriminate based upon race, based upon national origin. And you had heard some of those original horror stories, with the discrimination against females, just the very basic data set discriminations, which are used largely in the AI space, whether it’s from the employment, for the housing and the finance credit. The original kind of thought leadership on how I can discriminate was all bad.

Because, when I realized early on, it’s no longer a question, especially for large organizations are going to use AI in HR that’s when we had to flip the narrative to, like I said earlier, how are you going to use it? What purpose are you going to use it for? Don’t use it because it’s going to be really bad and discriminatory.

So, I think that is also something really important where a lot of governments inherently, because we have a very important role to play to prevent fraud or prevent discrimination, to prevent violations of the law that you really need to take that compliance side of it as well. And saying, if this is where the markets are going, here’s what we’re going to require opposed to scaring you into not using it.

And that was a very, very different approach, that I tried to lead.

Satwik Mishra

This is absolutely wonderful. But on exactly this aspect, I’m going to ask a little bit, to step back and maybe put on a lawyer hat. Tell us, you’re the interface of artificial intelligence at the workplace and EEOC’s work, which has been absolutely fascinating.

But tell us, how does this role shape your perspective on the role of federal government in general, emerging technology guidance. There are a lot of issues that we’re dealing with at the marketplace. And like you said, there are several agencies looking at it. From your experience, what do you think about the federal government’s role in technology guidance?

Keith Sonderling

Well, this is a very complicated question. Again, I’m in the executive branch. So, I take the very narrow approach, we cannot make new laws. We have to enforce the laws that are in the books and the laws that are in the books of this agency are from the 1960s although they may be old, they’re not outdated.

They’re going to apply to all the decisions. And it doesn’t matter to us whether it’s an AI making decision or whether it is a human making the decision, because the employer is the only one that can make an employment decision under our laws. It’s easier for me to say this with the authority and power we have here, this is how the law works.

This is what Congress, all of us doing, and all we’re going to do now, I can’t ignore the complexities of the changing dynamics in how governments want to regulate AI and how they want to regulate AI in the HR space. So, taking out that side of the executive branch limitations. You’re right. I’ve learned a lot and I’ve seen a lot.

And I think the trickiest thing is understanding that without additional resources, without additional help, we are not technologists here. And if Congress decides to give us a lot more money to hire technologists to actually dive into the algorithms, dive into the development and the deployment from a technical side. Yeah, that will significantly change things.

And then it would be a different conversation than just the lawyers dealing with the results and dealing with the technologists on the design, which we just don’t have the ability, we don’t have the funds. Really most agencies aren’t able to do this. I think what you’re seeing now, in the meantime, is states and foreign governments trying to get involved, trying to fill some of those gaps that we can’t do without congressional authority.

And if you look at the HR space in particular, it’s one of the more active ones when it comes to AI regulation. There is just a quick brief history is that the first laws related to AI in the workplace were out of Illinois and then Maryland, essentially banning facial recognition in interviews. Then obviously New York City local law 144 – was the first comprehensive AI HR law anywhere in the world related to artificial intelligence being used to make actual employment decisions and hiring, promotions. Then obviously the EU AI act diving into some of the cases for employment being in the higher risk. And same with Colorado and a lot of proposals and California and New York state.

Recently, just last week in Illinois, they passed another law related to some of the disclosures, and then not being able to use zip codes as proxies for artificial intelligence because of the longstanding discriminations there. So, you’re seeing now everyone wanting to get involved in that space. And my personal thought of this, having that national perspective is that, I commend them for trying to dive into this very complicated issue, but without a federal standard, without these requirements coming from us, especially for national or global employers, it’s going to cause potentially more confusion than compliance.

And I would like to use the example of New York. So, New York’s local law 144 requires that if it’s their own definition of what an AEDT (Automated Employment Decision Tool) means, well, for them, it has to essentially, for the most part, make the employment decision, right. The employment decision versus, being able to say, we have some humans in the loop.

It’s not actually one of many factors. Then it doesn’t qualify as AI even though it’s clearly AI. And if it does qualify as AI need to do a pre-deployment test and yearly bias, not even post those results. But if you look under the hood, what does that bias audit entail on the front end? It allows you to use other employers or synthetic data, which the EEOC would never allow.

And then it only requires you to do the audits on race, sex and ethnicity where we have many more protected characteristics. So, I just don’t want people to lull into thinking that they’re fully compliant if they do these little one offs in certain areas where states care about whether it’s video, whether it’s certain disclosure. And then, I think if you’re completely compliant and be surprised if the EEOC comes out and says, well, you’re using it for terminations and show us that.

And, we didn’t think that was regulated. So, you see kind of the issues there. And then, you’re starting to see some common threads now across the board of whether it’s overseas or here in the US in the HR space of saying, well, what are the actual themes we’re starting to see? And what’s emerging is consent, disclosure of the vendors, of how it’s actually being used.

And also, that bias auditing requirement. And even in the EU, with the high-risk systems requiring those audits then that takes the step back of saying, okay, well, now we have to do audits if we’re using AI in HR, whether we’re in Colorado, California, New York City or in the E.U., how do we do that?

And then if these states and foreign governments are going to start diving that well, what’s the new standard for auditing bias in HR? what everyone said across the world will look to the EEOC. And if you look at EEOC, it is based upon our uniform guidelines and employment selection procedures from 1978.

All right. So, it all comes back to that as progressive as some of these new laws are, the basic standards of how you actually get there, are based on us and are not going to change. And I’ve argued too, look whether or not you’re subject to the EU AI act, whether or not you’re subject to New York local law 144. What are the potential benefits there that you can see and use as yourself as a risk mitigation tool such as the bias-lines? And I’ve encouraged self-audits, since I started diving into this. Because if you’re using the EEOC standards, which of course, all these places require you to do, you have the ability now to test these tools before ever making a decision on someone’s life.

Well, prior to this. When you do an employment test where you have some kind of assessment, you hope it doesn’t discriminate. You do everything in advance to mitigate it. And then if there’s discrimination, you’re liable for that now with these AI tools, you can test in real time to see if the skills and everything you’re asking it to do related to your business is actually finding you the best candidates, or if it’s disproportionately excluding certain groups, based upon something that you could potentially change that is not necessarily for the job which we see all the time. So, even though that’s not required under federal law, even though you may have to do part of it for some places, if you’re doing this in advance voluntarily, you get to root out employment discrimination, not have that risk that it’s going to do wrong and feel much more confident than you can with your regular employment tools.

So that’s a long-winded way of saying, if these state, local, and foreign laws are pushing people to do things that you could be doing voluntarily any way to prevent discrimination, I think that’s a really good thing.

Satwik Mishra

So, one aspect of this is definitely we are starting to see the emerging trends of regulation, what it will look like.

And you’ve done, this was great answer to understanding what those emerging trends would look like, what’s missing and what has been done well. Let’s talk about the two questions that I definitely want to ask you is one of the issues that all of us deal with is what would the future of work look like?

What do you think from your perspective going ahead? Let’s take a foresight. I would never ask ten years ahead because no one would know. But let’s say 2 to 5 years ahead, what would be not the regulation side, but the future of work side. So, what do you think should and will change? And more importantly, as you said, as we’ve been talking about the constants.

So, civil rights movement, the civil rights laws, the 1970s interventions which have stayed constant. What do you think when saying ‘consistent’? And what do you think would change in the next 2 to 5 years and then going.

Keith Sonderling

Well, I think this conversation has completely changed since generative AI. And everything we’ve been talking about so far is using machine learning to make human employment decisions.

The decisions that humans were making, whether or not to hire somebody, whether or not to pay somebody, how to pay somebody. But now we’re seeing generative AI come into place and the displacement of jobs is no longer related to factory workers where you actually have a physical robot. And it’s coming for everyone for the knowledge workers across the board.

So, I think that immediate is the most number one immediate impact in the workforce. And from my perspective on that, I have significant issues like I do with any kind of AI in the workplace. How is that going to be implemented? And I think now with the rush to implement some of these generative AI tools, because every CEO, every board of directors is reading those same studies, we read about – How it’s going to displace 300 million jobs?

Is it going to make everyone 80% more effective in work? You don’t need to hire for these jobs anymore. People don’t have to do what they don’t want to do. Work and they’ll be happier working. You’re seeing a rush to say, well, how do we get that with an organization? And, if that is not built in with not only these ethical and legal safeguards, but more importantly trust, it is not going to go anywhere.

It’s going to lead to more claims of discrimination, because workers are going to feel that they’re just really being pushed out and that they’re actually not the employers not using these tools in good faith. They’re actually using these tools to displace them, and they’re having their employees train their robot replacement. And that really goes to the messaging, the trustworthiness of the name of your organization, Centre for Trustworthy Technology.

And it really resonates in this context, because if you’re not going to account for how are disabled workers going to be able to use generative AI, whether they need more time, where they need more adaptive devices, how are older workers, going to be able to use these tools who didn’t grow up on their phones like the Gen Z workers?

And are they going to really believe that the company is trying to implement this to help them, or are they trying to push them out? And we’ve seen a lot of workers who’ve been doing their job great for 30 years, the highest performance review of their job. And now suddenly, if they don’t become a prompt engineer 30 years into their career, they’re going to be forced out.

So, I think a lot of that on the front end is just how we message this, ensuring that workers with disabilities, workers who are older, who are more vulnerable to this, these issues or job displacement are going to have the tools we need as well, or it’s not going to work.

So, before we can even get there, from our perspective, we don’t want to see this led to more discrimination. We want to have all those safeguards in place to ensure that everyone can use these programs properly and that there’s no fear and anxiety. There’s been some studies out there that most workers, a lot of them, are very anxious to these tools.

And that is leading to mental health issues in the workplace. So, you can see from my perspective, from the laws we enforce, my head immediately goes there and says, how are we going to do this right? And what are those safeguards going to be in place to make sure that’s happening? And like everything else switching gears here, when this inevitably comes, it makes people much more efficient, they don’t have to do the job they don’t want to do, etc., all those studies.

And then we start getting into, well, who are going to get those employment opportunities to reskill, to retrain, to start getting the newer jobs that are going to be created out of this. And look, historically, we see some of these job displacements impact certain groups over the others. And there are some studies just on the quadrants of workers who are going to be displaced immediately, are lower wage jobs, and it’s going to impact certain groups more than others.

So, I think these are just really important dynamics that have to be in play. Because the worst thing is for a company to spend millions of dollars on a generative AI tool that they think is going to help their workers make them more efficient and it leads to mass claims of discrimination because it disproportionately eliminates certain jobs that you can correlate to, national origin, race, etc., which we see happening.

So that’s where I think before we can get into this whole future of work conversation, I have immediate questions and a raising immediate awareness on the implementation of this.

Satwik Mishra

Here, a lot is at stake. And no one does. Like no one brings light to it better than you in this space when we are in HR space. My final question before I let you go.

And we’re coming on time. But this is one of my favorite questions I would love to ask different leaders in their own fields. How do you think about trust? What is your perspective on trust? Certainly, within this domain on AI in the workplace?

Keith Sonderling

Well for me it’s a little different because I’m a lawyer, I’m a regulator.

On the compliance side, there’s going to be no trust if you’re violating the law. There’s going to be no trust. Like I just said, if you’re disproportionately impacting certain groups, whether females over male, right. And like some of these classic examples of who is being discriminated by AI, not just in the employment space, but the housing space, the credit space buying in, you see a lot of these studies of some of the same characteristics of those who have been historically marginalized and discriminated against groups is just being replicated and amplified when it comes to, technology. So, I define trust more uniquely. I’m saying that, well, you can’t even get to the trust conversation, which is incredibly important. In some cases, more important than some of the issues we were discussing.

Because you can legally have everything solid and not have trust and you’re going to potentially lead to those issues. Well, how do you get the underlining right on this? And for me, that’s the civil rights protection. And before you get to define what trust is, show me how you’re going to comply and ensure that these are very important protected characteristics in the US and around the world are being protected before you even tell me the benefits of the technology. So, I think that’s how we get there in the HR space, showing that, like, look, most people don’t trust employment decision making made by humans. And there’s a really interesting study that I encourage everyone to read by the Pew Foundation about this issue in HR. And we live in a bubble.

The vast majority of Americans don’t know this technology exists in the workforce, right. So, at first glance, most people don’t even know the topic. But when you dive into it, you ask people, would you rather be rated by the human or machine? And a lot of people said human first because that distrust of the technology, that distrust of big tech, the distrust of algorithms.

So, when they flip the question, they said, does bias exist? Yes. Bias exists in employment and hiring and promotions. Most people said yes. Well, understanding that exists. Flipping the question now, would you rather trust that same hiring manager or same boss who just had a bias or, a potentially neutral algorithm, and then people said they’d rather trust the algorithm.

So, I think that’s really a good example of trust, because if you could build that trust, then people are going to potentially want to be subject to this technology more than humans. So, a lot of that is from the good work you’re doing in other organizations of trying to lead with trust, lead with ethics, lead with legal compliance.

And this is not a criticism of the tech industry. I just think it’s new because a lot of the products that have been developed previously, whether it’s making data transfers faster, cloud tech, all these things, you weren’t dealing with human, you weren’t dealing with civil rights. You were dealing with business issues of reviewing documents story that all these things and now you’re using this for good, in civil rights areas.

It’s just a completely different mindset of what that trust is.

Satwik Mishra

Sure, Keith. Thank you so much for being here. It’s always such a privilege to talk to you and I always end up learning so much.

Keith Sonderling

Thanks for having me.

Satwik Mishra

Welcome everyone to another exciting edition of Trustworthy Tech Dialogues. I am Satwik Mishra, Executive Director of the Centre for Trustworthy Technology. Today, we are diving into a topic of great significance, ‘Leadership in the era of emerging technology.’ As we navigate these uncertain times, exercising leadership has never been more critical. In today’s world, innovation, creativity, and curiosity are not only buzzwords, but the foundational pillars of a successful corporate strategy.

Leaders are increasingly recognizing that fostering these qualities within their organization is essential for staying competitive, driving growth, and navigating the complexities of a rapidly changing global landscape. Cultivating a culture that sparks new ideas and encourages out-of-the-box thinking has become the key to developing products and services that are not just relevant today, but future-ready. For this culture, curiosity is the engine that drives continuous learning and adaptability.

It is the key to staying connected with the ever-changing ground realities around us and staying ahead in today’s fast-paced, competitive world. Leaders who nurture curiosity within their teams create a culture of exploration and learning, leading to higher engagement and ultimately more satisfied customers. This commitment is particularly vital in industries where stability and trust are paramount, such as the insurance industry.

We find ourselves in an area characterized by profound economic uncertainty, unprecedented technological advancements, escalating climate change, and the imminent reality of looming global health challenges. In this context, the insurance industry is increasingly seen as vital in delivering the stability we so urgently need. It is an industry which uniquely stands at the crossroads of the challenges and the opportunities presented by rapid change.

At the heart of this intersection lies trust, which I believe is fundamental to the industry’s mission. To explore these themes and more, today, it is my absolute privilege to be joined by a global luminary of the insurance industry, Mr. Tapan Singhel. With over 30 years of rich experience in the insurance sector, he has been a key figure at Bajaj Allianz for more than 22 years.

He has served as the company’s MD and CEO for over 12 years. He is a member of the Insurance Advisory Committee of IRDAI and the Pension Advisory Committee of PFRDA. He serves on the board of the Institute for Insurance and Risk Management. He has been the president of the Indo-German Chamber of Commerce and continues to serve as a board member.

He is also a member of the Governor’s Council of the World Economic Forum. Under his visionary leadership, Bajaj Allianz has emerged as one of the largest private general insurers known for its growth, profitability, and customer centricity. He has been awarded innumerable awards and recognitions over his illustrious career, including most recently the CEO of the year at the India Insurance Summit 2024.

Beyond these astounding accomplishments and an enviable body of work, I have had the distinct privilege of knowing him for over a decade. Throughout this time, I have witnessed firsthand, leadership embodied by him, which is not only strategic and steadfast, but also profoundly humane. It is an absolute pleasure to welcome you to the Trustworthy Tech Dialogues. Thank you so much for being here.

Tapan Singhel

My pleasure, Satwik. It is amazing to be with you on this forum.

Satwik Mishra

Let us start off at the beginning. Walk us through your journey at Bajaj Allianz. What initially led you to the insurance industry and how has the industry evolved over the last few decades?

Tapan Singhel

If you go back to the era when I joined insurance, at that time, hardly anybody wanted to join insurance.

It was like somebody who did not do good in life would join insurance. That was the image. And the funny part was that when we had this JB (Joint Business) with Allianz and speaking to some of my Allianz friends, they said, ‘No.’ In Germany, it is a bit different. When you fail in life, you become a bartender, and when you fail as a bartender, you join the insurance industry.

Imagine! That is the image that the industry had in that era. I am a scientist by education, and I wanted to get the Nobel Prize for my country. That was the burning desire as a child growing up. But because of some challenges, my friends said – This is interesting, take this government examination. I joined, public sector, which in India, you know, is run by the government insurance company.

Very early on, I paid some claims, and, when I saw the difference it made to the lives of people, I realized it’s a much bigger thing than what we understand. You contribute to society, to people at a very, very different level. And you can make a huge difference to your own country also by being part of this industry.

And that’s why I stayed on in the insurance industry for over three decades. But if I look at the last decade, not only for insurance but across the globe. Earlier on, when we used to do an innovation or do something new, we would reap the benefits for a decade or two, and you feel like you have done something really amazing.

Today, you do something which you feel is amazing, in just about 20 days, 25 days, everybody else has done it just like that. The speed at which things are happening now is so phenomenally fast. It is exciting. So, if you are not used to this excitement, it can become overwhelming. But if I use excitement and constantly keep on thinking about what next, then you really enjoy this game.

The industry has gone through massive changes in the past 10 years. And we will talk about it as we move further. But I will leave you with the thought process that if you meet somebody on the street, they will have distrust in insurance.

And they would say, these guys do not pay claims. And this is happening globally. You ask anybody, any part of the country or world. You are in the U.S. right now, go and ask somebody what do you think of the insurance industry?

But look at the insurance industry, they pay millions of claims, and their profit margins are actually much smaller than any other industry which is known.

So, the thought is, why should this be the notion when they are paying claims so well? Why should there be distrust in the industry? I will leave you with this question to answer that later on, as we progress in the conversation.

Satwik Mishra

So, a little bit on the distrust. And you have seen the shifts in the economy and society over the last two, three decades in this industry. What I wanted to specially do and learn for myself, and the audience is your distinct leadership style over the last three decades, how has it shifted?

And you speak specifically about the last decade and the changes which have come about. How has the changing environment, the economy, society, changed your leadership style at the company?

Tapan Singhel

Well, is it the economy or my age or the years of being there? Satwik, I can’t define what would be the reason for the changes. But let me tell you how I saw myself some years back, how do I see myself now and how do I see myself going forward?

So, when I was younger, aggressively building the company, I would like to be, very, very strong like an alpha male, as you define it. That is how it would be. And I would feel that ‘this is it.’ Nothing can go wrong. And this is how the world should be, and this is how people should think, and this is how it is.

And way back I remember, there was a professor from Harvard, so he was doing a course about two decades back, and he was doing some profiling. And he told me that, you are like an eagle but the worry is that you will scare the doves off. People would not speak when you are there.

That hit me really hard. I was thinking, why should it be, why should people not speak up? They said, no. You are too aggressive, and you do not listen. But my style was doing very good. I was very successful in what I was doing.

But that conversation really made me think. And I started making efforts to see that I listened to everybody. And then, when I would be on the ground, looking at, people, their difficulties connecting with them, it brought a lot of different perspectives, to leadership, to business, to the way I see people, to what business should do for society, how organizations and corporates can make a huge difference to the country, to society, to individuals.

Those thoughts came to my mind, and I saw a shift in the way I was behaving, with people, with customers, with stakeholders. The level of empathy moved up significantly and much higher. And this also gets very good results. It was not that my early alpha male perspective was the only one giving results. When I moved forward, I realized that it is not only about the materialistic way when we look at things.

How do you connect to the higher plane? How do you see a much bigger purpose in building organization and company and team, which actually is sustainable and last for hundreds of years? So over time, this has been a huge shift in the way I have looked at things.

But if you ask me, the current phase, I think this is better than the phase that I was in the past years.

Satwik Mishra

Eagles, doves, engagement and empathy. That is fascinating. But one of the things which always stands out with your leadership style is your continuous emphasis on engagement. We have seen you engage with probably your employees and more importantly, even customers.

I have seen you travel all across the country and even abroad, making sure that you meet people outside and someone at your position meeting people regularly, what is the value of that engagement with people, customers and your regular employees? What drives you to do that and what is the benefit of something like that in today’s age?

Tapan Singhel

So, Satwik, somebody asked me a question, how do you innovate so much?

So, as a company, we have done a lot of stuff which was first in the world for the industry. Not only first in the country. And I was thinking, how does it happen? And the answer which I gave was – To be able to innovate you should have, two things I feel.

One is you should be eternally curious. Eternally curious means we should be like a sponge, pick up every bit of information and knowledge and be eternally curious. Ask people, be with people. Figure things out. Because your mind is a data lake. The more you fill it up with knowledge, it may be random knowledge, but it is perfectly good.

And second, you should have a high-level of empathy to be able to feel the pain and the concern of an individual, of the society, of the country, of the world at a very high level. When you combine these two together, your mind will figure out the solution and that is what we call innovation. And then you should obviously have the understanding to implement it.

See the transformation and push it through to the last mile to be able to get to where you want. Now to get the understanding, be it business, be it what is happening or what do you feel is going to happen in the immediate near future.

You don’t connect to people at an amazing level in which you are just being how should I say, at your humblest form, where you are able to listen and understand every sentence, every word the person is saying. Be it your customers, your stakeholders, your own team members, be it a person on the street, be it a person in Parliament, or anywhere.

You will not be able to take your organization to the next level. So, meeting people is very essential. Essential to understand business, to understand what’s the requirement, to understand what you do. Also understand from people’s perspective, what they are missing and what you as a leader should be doing.

Because leadership is not the way you think, because it is not about, leading or something like a very powerful marshal on the top. It is the lowest rung in the hierarchy. Where you actually have to be at every place to figure out how you can ensure that people are more comfortable and happier from the time they met you, to the time they meet you next.

Satwik Mishra

It is not just your engagement, if I may say so. Like, not only are you meeting people, but also, you are meeting people with a lens of curiosity.

And I say this because I have been following your work for over a decade, like I said. And I spoke about curiosity upfront because I cover your blog, I regularly read it, and your blog speaks to not only corporate strategy, but also philosophy or personal development, things which are very wide ranging and help individuals at different stages of their lives. How does your curiosity drive this engagement and what, how has that changed over the years?

Tapan Singhel

So, when, if I go back again when I was younger, I would feel that I am God’s gift to mankind. Whatever I say and do is the right way in the world and nothing else. Today, I feel that it’s unbelievable.

Every individual that I meet, everybody that I see, even if I look at a tree. They have so much to give, and I have so little time to understand and absorb. It is a huge shift. And that is why I said is it age? Is it understanding? Is it life? Is it the economic condition?

But this transformation I have seen. And I am very happy with this equation of mine in which I am eternally excited to meet people and to know and be awed by their experiences, the knowledge, and try and figure out, wow, this is something which is so amazing. I think it just hits me. Some people say it happens when you let go of your ego or you become more humble.

I don’t think I really worked on any of this. But maybe the equation that you have when you think that you are God’s gift to mankind, to a place where you feel that every individual is so amazing, I think that shift in your thinking actually opens you up much more to a bigger inner experience and better understanding of life and things in a much better perspective.

So, this has been the shift, Satwik.

Satwik Mishra

It’s absolutely fascinating. And let me go back to another thing that you mentioned earlier about short cycles of product development and innovation, where you need to be at the forefront every 20 days, every quarter, if I may say so, going ahead. And one of the things that Bajaj Allianz is especially known for is innovation and staying ahead of the curve as a leader.

How did you imbibe this within the work force, and how does that drive the company going forward? And what value would you give to this for leaders today?

Tapan Singhel

So, let us understand this. If you are too ahead of the curve, then it doesn’t really benefit your business a lot.

So how do you do that and how does it happen? I will give you two examples and then I will answer my first question I left you in the beginning saying I will answer later. One example is as a company, we are doing very well. You know, with the mid phase.

But the challenge was that we had the old government companies which have been there for close to 100 years. They had bigger sizes in terms of volumes and number of customers. And then I was thinking the difference is that they have more offices in a huge country like India compared to what I have. So, if I had to be ahead of them, I must have more offices.

That was first. Now to open more offices means I have to invest a lot of money in the infrastructure, which would be a huge capital requirement, which means I have to go to the shareholders and my return on the ROI would be much less. That was one thought which was there. The next thought was that I was at that time the chairman of the Indo German Chambers.

And then when I go to Europe and people would talk of demographic dividend. And that India has demographic dividend, and you would not have it. So that was the first time in early 2000, I was thinking, what are the speed brakes? I know about the dividend. But if we are able to provide employment to the youth here, it could actually become negative.

And then, you normally would say the government should do that, but as a professional, how do I contribute to that is the next question.

So, can I generate employment for a million youth? That was the second burning question. Now, if I am able to open offices across the country and I don’t have to invest in capital in terms of the infrastructure, then I can have a model.

Now you have two statements. I can provide employment as a professional to youngsters in my country. I can generate more business and take on competition. Now to do that, it looks very easy nowadays as you have, internet, good cameras and mobile, and tablets. But in those times, the internet was not so good.

But then you have the prime minister speaking about getting internet access to the villages. And maybe NITI Aayog you would also be not working on those stuff in that era. And that conversation is going on. So, when I started seeing the conversations happening, I realized that this is going to happen well and the speed at which technology is progressing, the cameras and the internet speed, and the tablets are going to get better.

So, I told my team to start working on virtual offices. Where you do not have physical presence, everything can be on the tablet. Today, it looks like everybody is doing it. But that time, people told me it was a foolish idea. My board members told me it is a foolish idea. It does not work. That there is no connection, nothing.

My bet was, by the time we develop this, the connections and other things will happen. And that is what happened Satwik. I think by the time we developed the technology, the infrastructure, did recruitment, you had internet in all places, the camera got better. So, we were just ahead of the curve.

We were the first to do that, and we launched one thousand locations, provided employment, generated business. And now we are ahead of most of the government companies in the Indian market. So, this is an example that you start preparing before the technology has arrived at the stage where it is. Because it takes time to prepare.

After it arrives you may be at the curve or just a bit behind it. But after everyone else has done it, then you are too far behind. But the one who just goes ahead of the curve, and you know you have to prepare in a way that by the time the technology hits, you already are ready to deploy it. That is where the money and the reach and the confidence come.

The second example I want to give you is as I told you is the distrust on the insurance industry, even though they are paying so many claims. And they are really paying through the nose a really high amounts of claims.

It is because the process of paying claims is so cumbersome. You lodge a claim, and you say they ask so many questions. They ask so many questions simply because they have public money and they really cannot be, giving it away to fraudsters and all. And that friction has to go. So, you have to make claims, friction free.

And that is when it happened, with machine learning. Now you have advanced AI, generative AI, and blockchain. We used it initially to ensure that as a customer, when you click pictures, upload and I’m talking about like 15 years back and we transfer money to your account. We tried to make it as friction free as possible.

So, when you put all these thoughts together, you realize that, for innovation, you have to put thinking which would take into account the developments happening. And you are actually at the time it happens.

Satwik Mishra

It’s almost about developing the foresight to see the curve not being far too ahead, but definitely not being behind.

It is absolutely fascinating. But also, this is about developing foresight and in the insurance sector you would also navigate you would have navigated through various crises which have happened in the economy. As a leader, is there a crisis that stands out which brought about some lessons for you?

And how would you suggest is the best framework to think about a crisis as a leader today?

Tapan Singhel

So, in India right now, there is a price war which has become very intense. So, I had to address a team some days back and I could sense that they were saying that the time is bad for business.

Then I got on stage. The first question I asked them was that was let’s say in the 23 years since we have built this company, which year that any one of you came and told me that this is an amazing time, whatever target you give, we’ll give you 500 times of that. That means, every year, there has been some crisis or other.

Every year is a difficult year. That is how business will be. You cannot have everything falling into place the way you want it. And you know, then you sit at home, and business starts growing. It doesn’t happen. Which means that crisis is not something to be afraid of. It is something to be loved. And the worse it is, the better it is for you.

Because in a crisis, the good companies become stronger, and better. And the weak and shallow companies, disappear.

Now, are you a good company? Do you have that kind of conviction and standard? This is what will determine how you move forward.

Okay, let’s say these many years I have been, when was something where I felt everything was perfectly fine and I am just happy? It does not happen. There would be some pressure. It’s how do you handle that? How do you see it and what is the point at which you make a company stronger? What are your competencies? In your own life, how do you handle that and how do you make it stronger than what it was, takes you to the next level.

Crises are good. Bad times are amazing. You should be excited about it. The worse it is, the more excited you should be because that gives an opportunity for you to really know, transform and push things. That is the first thought process that you should have.

The second is that you start looking at it. Be realistic about where you stand and how things are.  Have a dispassionate look at it and start figuring out from the end consumer perspective. End consumer does not mean only the client. It’s also your own employees. I gave you an example of Covid also, and how we handled that. And what exactly is hitting them hard?

And how, with what constraints you have, are you going to stand by them and make a huge difference to them? And how do you transform yourself to deliver that? That is what should be the obsession for the entire unit or organization, that you will lead. And that is when you put these two together, you start enjoying it. And then as time progresses, you start emerging stronger.

So, in our case, we are in 2007, completely free pricing. Or when we started coming in 2001, we only had government companies. So why would somebody trust a private company at that time. People would say that Tapan we know you, but how can we trust you? Government agencies not accepting our quotes, to the price going to 90% fall, to an environment not for the profits of the market going to disappear. In 23 years of building this organization, I have seen at least ten huge crises.

But fast-forward today, if I look at the past ten years, 70% of the industry’s profit was made by our company alone. We have 150 million customers organically grown brick by brick. We have the largest distribution of bank insurance in the world. We have 240 banks, and a hundred thousand agents. If we look at it, it is all built organic. Every crisis made us stronger than what we were before that. It is just the thought process that we just have to do.

So, in Covid, I remember when hospitals didn’t have beds, we actually set out 18 ambulances in different cities with six hours of oxygen competence. So, if something goes wrong with the customer or employees, we can actually keep them alive on oxygen and take them to a different city where the beds are available.

So, in every type of crisis, you keep thinking on your feet and see what you can do to be able to handle that. Not just sitting back. I think that is what keeps the excitement alive.

Satwik Mishra

So, this is one aspect of not becoming overwhelmed by the crisis at hand, the crisis will always be there. The second aspect that you spoke about was a little bit of building resilience within your company and within your organization such that you are ready for any crisis which comes ahead.

But there’s also the reality today that there are too many lenses of uncertainty around us. And you spoke about 150 million customers. That’s 150 million different uncertainties, that you have to drive through and navigate through. How do leaders think about uncertainty in today’s time? Like what do you see? And specifically with the lens of emerging technology.

I don’t think emerging technology has ever been as pervasive as it is right now in society. How do leaders think about driving through the uncertainties which have come about from these emerging technologies?

Tapan Singhel

So, somebody asked me this question, I think that is early on when generative AI, when it was just starting. And they said that, how do you look at technology?

I said, to me technology is a tool. I solve problems. And that problem, whatever technology has to be used, I use it to solve the problem. I think that is how we start. It is not that technology becomes all for what I am doing. Now, anything which is new, we are curious and excited about it. And keep on thinking how and what problem that you have where this can become a huge fit.

And how do you do that? It will have risks. There will be disruptions. But the biggest risk is if you are not able to use it well for solving issues, problems or customer experiences. If you are afraid of it and you want to be away from it, that is a bigger risk.

Risks evolve over time. I think when I started working, you would get, salary in terms of a cheque which we had to go collect in cash from the bank teller and carry cash home. And the biggest risk was pickpocketing then, and you try to see how to handle that. Today, the biggest risk is phishing and people taking money away from your account using mobile hacking or whatever it is.

Risks have changed. But obviously convenience has also moved in. In my experience as a customer, it has moved in. Things have become much better. So never be afraid of new technology. It will have risks.

The second thing is how do you build a mechanism where risk correction keeps on happening? Let us look at generative AI. Now there are a lot of hallucinations which would happen. It can go to unethical methodology. So, people are thinking of guard rails. But there should be continuous feedback because there are more good people in this world compared to not so good people. Let us put it like that.

The majority or overwhelming majority are of good people. That is why the world still exists. If the majority was not of good people, the world would not have existed. So, if you have a feedback mechanism of self-correction, where people start contributing and more and more people contribute in the right way, the Generative AI actually gets more powerful in the right ethical manner compared to what it is.

Now if you try to remove all that mechanism of feedback and you are so scared and you put so many guardrails, you don’t allow development to happen. You can’t stop it. Somewhere else. Somewhere, far off, maybe a rogue unit, organization may actually start building, using more feedback from negative. And make it more negatively powerful than possibly it can be.

So, my view on technology and the way I look at risk is, first and foremost, be super excited about all the development. Second, what problem does it solve? Use it well. Third, just keep it very open and allow society and the world to keep on adding there. Don’t make it very restricted because there are more good people than not so good people. Remember this.

That is how it works. And that would actually ensure that it actually gets better and goes in the right way. So that is the way I look at it, Satwik.

Satwik Mishra

So the way I’m thinking, one of it is, from your answer, it is like a philosophical lens in terms of not being awed by the uncertainty brought about by the technology, but flipping the paradigm and thinking about how the technology can be used to address the uncertainty and strategically, developing a framework such that you have continuous feedback channels, which is again, extremely interesting to think about while you are thinking about emerging technology.

But you also, like in your answer, spoke about phishing and the challenges and risks that stem from tech, like in the world today. And I remember Bajaj Allianz was probably one of the first firms to come up with cyber insurance. Now, walk us through that journey of coming out with cyber insurance. And what have been the lessons since then?

Tapan Singhel

Yeah, Satwik. It is again, a story. So, in one of my interactions, one elderly person lost some money because of phishing, and I met him, and he was so sad. And I was talking to him and thinking, as an insurer we are missing out on solving the worry of this person and it is going to happen again.

This was much earlier. Today, every second person is getting this phishing message and people have become smarter on both sides. But this was the early era. And then we said, we have to solve it. So, we came up with a retail cyber cover, the industrial cyber cover was there. But now any individual can buy that and be protected against that. And then we thought, there is also stalking happening on the cyber.

So, people normally do not know how to react with somebody stalking them. How can an insurance company provide assistance in taking care of the legal fees? How do you stop that? What are the issues that are happening in cyber? How are the insurance companies going to protect that? And we came out with the first retail cyber cover in India.

My daughter’s buying it. She was one of the first customers to do so, and we’re super excited. But those were early times. At that time the cyber risk was not so perceived as it is today. So, the policy did not take off as well or as much as I thought it should be. Because normally I am more excited than the time is.

But it started picking up, as the risks are moving in. But again, it came back from same thought process, the risks have changed. You know, nobody’s pocket is getting picked now, we don’t have pickpockets anymore. So, the insurance companies still providing cover against pickpocketing has no relevance, it has shifted.

So, how do you keep seeing the shift happening, the risk and how do you keep moving towards it? And how do you solve issues? And what all risks can you think of even putting out, so the product is also involved, coverage is also involved.

Also, a lot of companies also came forward and the risk also has moved up a lot now. But obviously we have the pleasure of the first to be launching it. And it started from that conversation with the elderly person on them losing some money to phishing. And that again, comes back to the question. You asked me, why do I keep on connecting with people? Because that’s when you realize what is the mess that you’re doing and how do you make yourself better.

Satwik Mishra

Wonderful. So, one particular question that I’ve always wanted to know, and I’m sure that a lot of young leaders today would be very interested in understanding is, you had this distinct pleasure of seeing your firm grow over the last three decades.

And scale to where it is right now. How do leaders think about their business models as their businesses scale, as the industries scale, as they get more consumers, as they probably navigate through the need of a larger workforce in times like this, as companies, think about scaling as we move from startups to probably a resilient company becoming a more standout organization in the world.

What are some of the lessons that you thought have worked? Why Bajaj Allianz scaled over the last three decades?

Tapan Singhel

So, when you are a startup, every decision moves very fast. When you become a large organization, the biggest worry I have is, it becomes bureaucratic, it becomes silos. You have, as I call it in my word, you have emperors with their own kingdoms.

And they do not want to know anybody else’s move. And you have a lot of arrogance which comes in. And that is what destroys organization. Organizations and individuals get destroyed when you reach this place. So, you have to be on a continuous, one, startup pace. Second, always be excited that you have to do much more.

And there’s so much to be done. And know what can you do. Be aware of the use of capital and cash. Be aware that it has to be leading to the benefit of society, the country and the world. And what you do, it has to have an impact on that. Be aware that organizations exist for society. And for that, they have to run it with an ethical, good governance perspective.

And be aware that the individuals in your organizations define the organization, not you. The culture that you have, the people that you have. Be very sensitive towards that. You can’t have a very toxic culture inside the organization and expect them to be taking good care of us when that happens. Be very aware of that.

Be obsessed with the thought process that every day that you get up, what difference do you make. So, when you keep this on and you don’t allow arrogance, the bureaucracy, the layering to come in and politics to come in, you have to keep on nipping at the bud.

Continuously. Be it in communication. It is not that one culture, you change, and it happens. It is the continuous work that you have to keep on working on and build that sensitivity towards that. Then you can have organizations which outlast the average age of organizations which normally is 20-25 years. Because look at the stock market and pick up companies at the top, 20 years back.

Most of them are not there today. So, to outlive that you have to be very, very sensitive on this part. And this is fun and challenging also because the natural tendency of large organizations is to become bureaucratic and to create silos, and to become arrogant.

Satwik Mishra

Tipping the arrogance that may develop as you scale, being very careful about not ensuring that bureaucracy comes in the way of you scaling and obviously being very mindful about the culture within the organization and so that it isn’t leading to any toxic work environments that you are able to grow sustainably.

One of my favorite questions to ask leaders today is – you spoke about the distrust which exists possibly amongst people, with the changing times with so much uncertainty, how do you personally think about trust, both within the firm while you are dealing with your employees and also, more importantly, while dealing with external stakeholders with partnerships, which Bajaj Allianz essentially is a partnership.

What is your take on the importance of trust today?

Tapan Singhel

Trust is everything, Satwik. You know. And trust does not mean that you are judging anybody, trust means that you accept everybody as how they are and who they are, and you believe them. Start from the point that you trust everybody. Never start from the point that you will trust people after they have proven themselves to be trustworthy.

I think that is the first point.

Let’s say if I meet even a stranger, I will start with complete trust and faith in the person and awe of what the person has done there. So, the way you have to look at trust is when you give a person that comfort, that trust, that feeling, they also trust you back. Human beings actually mimic. It’s a very interesting thing.

And, to understand this better, try and do this. So, next time you meet somebody just look a bit cranky. And observe, after some time if you watch them, they will also become stiff. You have done nothing; you have not spoken a single word. And next time you meet somebody, just keep on smiling for no reason, and you will see the results.

So human beings mimic. If you want trust. You decide to trust. It is not a game that somebody has to prove before you trust them. I think that is the first fundamental. You start with trust. What will happen? 100, you will trust. One will betray. Fair enough.

99 did not. So, you’re still winning more than losing. As I said, the world is full of amazing people.

There are a few who are not so amazing. It’s fine. Majority are. So why would you start with the negative?

So, in my view, trust doesn’t come from the expectation of somebody else showing it. It comes from us and demonstrates that we really trust everybody that we meet and talk to and go all out in all that we can do. Even if somebody is not kind to you, you be kind, and you watch the change happen.

I’ve never seen the change not happen. You be more kind. Somebody doesn’t trust you; you trust that person more, and watch the change. So, I think the basic rule is you start strong as we have to trust more and blindly. Even the cost of getting betrayed a couple of times, is perfectly fine.

Satwik Mishra

And, so it’s almost reciprocity, like building a reciprocity from first, from your lens, and then you expect. Once you start building, engaging people with trust, you will get trust back.

Tapan Singhel

Don’t expect. It will happen. You expect, you will have disappointment, Satwik. So, don’t expect.

Satwik Mishra

And my final question is – What advice would you give to young leaders today who are trying to navigate this intersection of technology and starting off in leadership positions, maybe in the government, maybe in the corporate world, maybe in terms of society, what are some essential steps that you think that young leaders should keep in mind in today’s times?

Tapan Singhel

The first thing is – ‘It is not about you.’ Always keep that in mind. It is about everybody around you, first.

Second, what do you feel for the most? Figure that out because that is what you have to get into to try and help make it better. Third, the more you try to make the world better, the happier and more prosperous you will be.

Now remember that. So, if you look at any business, becoming a unicorn or has billions of dollars, at the root of it, they made the world better, than what it was. So as a young leader, try to think how you would make a difference to the world? Then think how can you make it better? Or, forget the world, if not the world, the country, if not the country, the society, not the society, the family. And if nobody else, you. What difference can you make?

I think that is the first thing I always think about is to have a high level of empathy. And it is not about you.

The rest will follow. It is something that if you are so obsessed about position, if you are so obsessed about money and you chase that, then that may not come so soon. If you are obsessed with trying to solve things, to make a difference in society, these will follow very easily.

It is a very funny equation, like the one I told you about how people mimic others. So, do not be obsessed. Designations, power, money, fame will come to you if you do not obsess about it. If you obsess about it, it will take a lot of time. Be obsessed about what difference do you make.

‘It’s not about you.’ Just remember that. And in the end, never forget, be happy, be joyful. Keep smiling. Come what may, no crises will last forever. Even when it comes, smile it through. This too will pass.

Satwik Mishra

Those are wonderful words to end this podcast. I wanted to understand leadership in today’s times and there was nobody better that I could have discussed it with.

Thank you so much for being here.

Tapan Singhel

My honor, Satwik. Always great talking to you.