Watch on YouTube; listen on Apple Podcasts or Spotify.

Episode Summary:

In this engaging conversation, Aashka Patel interviews Deeba Kazmi, co-founder and chief data scientist at Finbots.ai, exploring the intersection of AI and FinTech. Deeba shares her journey from lead data scientist to co-founder, the mission of Finbots.ai in revolutionizing credit risk management, and the significant impact of women in leadership roles within the industry. The discussion also delves into compliance with regulations, the importance of explainability in AI models, bias monitoring, and the challenges of rolling out AI in legacy systems. Deeba emphasizes the need for high-quality data, the role of human oversight in high-risk scenarios, and the emerging trends in fraud detection. The conversation concludes with insights on AI literacy, the skills needed for the future of FinTech, and the relationship between AI and universal basic income.

Timestamps:

00:00 AIR Bites (Precap)
02:21 AI in FinTech – Everything You Need to Know
04:41 Deeba Kazmi & Finbots.ai – Founder Journey
07:27 Women in AI – Leading the Change
10:18 AI Compliance – FinTech Essentials
13:08 Explainable AI – Why It Matters
16:01 Data Quality for Better AI Models
18:54 Fighting AI Bias – Monitoring & Fixes
21:58 Upgrading Legacy Systems with AI
24:43 Human vs AI – Who Makes Financial Decisions?
28:23 How AI is Evolving FinTech Decisions
29:37 AI Guardrails – Keeping Finance Safe
31:41 Testing AI – Navigating Regulations
31:51 AI & Fraud – New Threats to Lenders
33:29 Credit Assessment – Insights from Emerging Markets
35:05 AI Literacy – What FinTech Teams Must Know
38:14 FinTech Future Skills – What to Learn
41:59 AI & Universal Basic Income – Financial Inclusion
45:01 FinTech Founder Tips – Thriving in Change
50:31 Outro

Transcript:

Aashka Patel (00:04)

Hello and welcome to On AIR with Aashka. Today we are diving into the world of AI and FinTech with one of India’s trailblazing women in AI. Yes, I am joined by Deeba Kazmi, co-founder and chief data scientist at Finbots.ai. She has spent over a decade at powerhouses like Citi, Morgan Stanley, Experian. So welcome to the show and thank you so much for joining.

Deeba (00:28)

Thank

you, Aashka. Thank you for inviting.

Aashka Patel (00:31)

Yeah. Yes. Let’s dive right into the questions. So you have been recognized as one of India’s trailblazing women in AI. Huge congratulations on that. So can you shed some light on your work at finbots.ai that brought you this recognition? And what do you do?

Deeba (00:51)

So at FinBots I am leading the product development enhancement aspect. So when I joined four years back, I joined as a lead data scientist and my journey from lead data scientist to co-founder has been over a period of two years. In these two years, I started working on the development of the product from scratch.

Aashka Patel (01:00)

Mm-hmm.

Deeba (01:20)

Not just that, but I was very deeply engaged in the pre-sale conversation, understanding what are the client requirements so that I can build that into the product. So that our product is not just a flashy product, which has AI built in it, but it actually solves real client problems.

Aashka Patel (01:42)

Yeah. Can you give us a little intro about the product? Like what space is it for the audience?

Deeba (01:49)

So FinBots was actually launched with the mission of developing AI products to solve the challenges that are there in the credit risk area of the banking industry. Now credit is a very important aspect when it comes to banking because it’s the money making part of the business. So although there are other banking products, but the real money really comes from credit.

Aashka Patel (02:02)

Good one.

Deeba (02:15)

All at the same time, there is a lot of risk involved in it. So, yeah, so because of the trade off between the money that it brings in and the risk that’s involved in it, there is a lot of need of coming up with products and AI solutions that help you solve this trade off. So, and this can be done by leveraging the lead data that’s out there and using the

Aashka Patel (02:20)

Yeah.

Okay.

Yeah.

Good

Deeba (02:44)

state

Aashka Patel (02:44)

night.

Deeba (02:44)

of the art AI algorithms and processes to make use of those data sources in the most appropriate manner, which can then lead to low risk credit lending.

Aashka Patel (02:55)

Mm-hmm.

got it. Got it. Yeah, that sounds very exciting. And of course, as you mentioned, it’s a high risk area, also classified by the EU AI Act. so I had this question about beyond the headlines, since you have spent over a decade in the FinTech industry itself, not just in AI. So what’s one

Deeba (03:11)

Yeah.

Aashka Patel (03:22)

concrete way, having more women in fintech AI leadership has changed actual business outcomes or product decisions at companies that you have seen or worked with.

Deeba (03:34)

Yeah, absolutely. There have been real and measurable impact on business outcomes, whether it be growth, whether it be innovation, ethics, team performance, and lot of such areas, which has led to a remarkable shift in the way the AI products have been designed. So now if you see the AI products are much more user-centric, inclusive, and effective. And a lot of...

Aashka Patel (03:42)

Thank

Mm, yeah.

Deeba (04:02)

This has to do with the inclusion of women in the leadership because they have brought in the aspects of the different data parameters to be considered, looking at the product from a human aspect, a lot of collaboration across teams, which has led to developing much more inclusive products, which are more effective, negotiation of adoption of AI.

Aashka Patel (04:08)

women leaders.

the day.

Mm.

Deeba (04:32)

So there is always a huge gap between the business team and the innovation team. So there has been a good amount of contribution by the women leadership in this negotiation and bringing them on board. Yeah.

Aashka Patel (04:32)

Mmm.

Yeah.

Yeah, yeah, yeah. That’s very powerful to hear and I totally agree with you and we need more women representation in the AI world as well. So let’s go back to the Finbots conversation and talk about some Singaporean regulations.

On the Finbots website, it’s mentioned that Finbots has completed MAS veritas and AI verify frameworks launched by the Singapore finance and technology regulators to objectively validate the quality and performance of AI solutions. So can you explain what they are and how did Finbots comply with them?

Deeba (05:27)

Okay, so it was a good two to three months of pilot that we worked upon in collaboration with MAS and IMDA. So these are the two government bodies of Singapore, which had a lot of technical tests on the product itself and the AI models that are being generated by the product, as well as process checks.

Aashka Patel (05:34)

Mm-hmm.

And.

Mm-hmm.

here.

Deeba (05:52)

So a huge list of questionnaires where you have to provide answers on the process that has been adopted to make the AI solution and the models that are being developed completely compliant with the regulatory policies that are out there. So as mentioned, a combination of technical as well as process checks, which includes all your major pillars of...

Aashka Patel (05:53)

Mmm.

in.

in.

Deeba (06:19)

your feet principle, which is fairness, explainability, accountability, and transparency. So it was much detailed process to really analyze the product. So if I go into the detail, the technical test really talks, gives you results upon what kind of AI models are being generated in terms of accuracy.

Aashka Patel (06:27)

annoying.

good.

Deeba (06:44)

in terms of fairness. So there are fairness parameters on whether the model is biased with respect to any of the gender class or age groups or ethnicity, explainability, whether it’s able to explain and come up with the features that are contributing to the result and are they accurate or not, whether we are using the approved algorithms and the latest algorithms for the different steps that are involved in the modeling lifecycle.

Aashka Patel (06:51)

and move.

Deeba (07:13)

what we are doing to maintain the model repository, keep track of the versioning of the models, keeping track of the models even after they are developed, validated, and pushed to production. So end-to-end check and assessment was done, and after that only we were provided that certification by these government bodies.

Aashka Patel (07:33)

Mm.

Yeah, yeah. So does it happen? Like, what’s the frequency of the check or the audit that happens? Like, does it happen yearly or

Deeba (07:45)

So no, so it happens once unless until your product undergoes a drastic change. For one AI product, it will happen once end to end.

Aashka Patel (07:48)

good.

a major change or something.

Deeba (07:57)

the process checks happens once when you have launched the product and it’s ready. And if it undergoes some bit of huge transformation, then you will have to go through the process again. the technical tests are something that can be infused within the solution. And every time you’re developing a model, if the client needs a certification, can.

Aashka Patel (08:10)

again.

Mm-hmm.

Deeba (08:23)

we can go through it really quickly because there’s a tool kit and we can provide it, just provide it to them on the go.

Aashka Patel (08:26)

Okay, yeah.

Yeah, that’s actually quick and very smart way to approach it. Yeah, because with I think SOC two compliance or something you need to get it done like every now and then type one. Yeah,

Deeba (08:34)

Exactly.

Yes, yes, yes, yes. Those are the, those

certifications we get on a yearly basis, the SOC two type and engineering certification.

Aashka Patel (08:52)

Yes. Yeah,

yeah, that’s interesting to know. So you mentioned about the explainability part. Let’s dive right on to that. So the EU AI Act also requires clear and meaningful information about automated decisions. But here’s the paradox. When you give customers detailed LIME or SHAP explanations, some become very suspicious or even litigious, especially seeing factors they

Deeba (09:14)

Yeah.

Aashka Patel (09:20)

considered irrelevant. So there’s also a technical issue like LIME creates simplified approximations that can sometimes be misleading. So it might highlight income as the top factor when your model is actually weighing complex interactions between income, spending patterns, regional indicators, like all of that. So how do you basically overcome these kind of challenges while

Deeba (09:21)

Hmm.

Aashka Patel (09:47)

infusing explainability within your solutions.

Deeba (09:50)

Yeah, so that’s a very important question. So the USP of the product is that it has automated all the model development cycle. So although there is human oversight and control, but you can run all of this very quickly at a click of a button, along with having control. So you can provide your inputs as well. How it helps is once you develop your model, at the same time, you get a very well

Aashka Patel (10:04)

Okay.

Boom.

Soken.

Deeba (10:19)

automatically documentation available, which gives you all the details about the model, the overall development process, the top variables, the contribution of each of the variables. And then this model, because it’s a credit risk space and it’s not any other space, it’s a credit high risk models. So it cannot go to production without getting an approval from the regulators or the model risk management team of the organization.

Aashka Patel (10:24)

Go for it.

Yes.

in the room.

Deeba (10:49)

modelers can straight away pass on this auto-generated document to them and let them go through the top contributors. And if they have issues with certain model variables, they can just mark them out. And over here, the modelers can just quickly at a click of a button, rerun the model with the updated set of variables and share that with them. before going to production, the...

the model risk management team has to approve those variables, which happens really quickly through an automated fashion through the documentation that’s auto-generated. That’s one. So in no point that the regulators will come to know, why this variable is being used or why that variable is being used, it has to go through that approval process. Second is, this is global explainability when we are developing the model.

Aashka Patel (11:17)

Good.

Really.

You know.

Yeah

Deeba (11:43)

now talking about the local explainability when the model is in production and the explainability is provided at a customer by customer level. At that time, also given the risk involved, not everything goes through an automated yes or no or decline or accept kind of scenario. We do understand that at times there is human oversight involved.

Aashka Patel (11:47)

No.

Yeah.

Okay.

Deeba (12:12)

And so wherever we feel that there is a risky region, and let’s say the top percentile of the customer base model is very good in identifying and giving the top parameters. But let’s say there are segments where there might be some scope of misalignment between a human inspection and the AI inspection. we, again, through an automated fashion, it will go through a manual review.

Aashka Patel (12:13)

Mm.

room.

Yeah.

Deeba (12:42)

So where the decision maker can go through it and then pass it on, if everything looks fine to him, then he can pass it on to the next stage.

Aashka Patel (12:43)

Okay.

Good one.

That sounds very interesting. you mentioned about the local thing, localized explainability. does cultural relativism play a huge role when you are defining? Because there is a human in the loop or human oversight is there, so it’s tuned accordingly.

Deeba (12:58)

local. Yeah.

No, it depends upon the values that that particular customer has for the different data points in the model. So the local explainability again comes from the AI algorithm only. So it will assess, yeah, it will assess maybe at a global level and for majority of the people, income is playing a big role in his application getting accepted or declined.

Aashka Patel (13:22)

Mm-hmm.

Only, okay.

Mm-hmm.

No.

Deeba (13:39)

But for this particular customer, it might be something else. Maybe it’s his income to expense ratio or loan to value ratios for his past credit score. So it can be totally different because the values that this customer has and the interaction that’s happening between the variables for this particular customer might be very different. So that’s why local explainability plays a key role for decision making.

Aashka Patel (13:40)

Mmm.

Boom.

Boom.

No,

Yeah.

Yeah, So explainability isn’t just like, our models are explainable or explainable AI is implemented. It’s much deeper than what we see on the surface level. So yeah, it’s very interesting to know that and localized is very like mind blowing to me at least. Yeah, yeah.

Deeba (14:20)

Exactly.

Aashka Patel (14:30)

So within your product, you have this model builders right? Within the creditX product. So EU AI Act requires high quality data sets for forecasting and risk models. As we mentioned, it’s a high risk scenario. So since you are letting the clients build their own models from their own data, so how do you maintain this kind of...

Deeba (14:47)

Mm-hmm.

Mm-hmm.

Aashka Patel (14:56)

help maintain this kind of high quality data and integrity on your platform.

Deeba (15:01)

Okay, so there are two ways we do it. Stage one and stage two. Stage one is when we are deploying our solution in the client’s environment, we will definitely do a initial level of analysis to see, to check the quality of data and to understand what’s the appropriate usage of data. It’s not a...

Aashka Patel (15:05)

No. No.

Okay.

Deeba (15:23)

every time and for every data that we would recommend that you go ahead and develop an AI model. Maybe that client has just started lending and they don’t have enough data points, or they don’t have a mature repayment history. So whatever, or they still need to procure information from third party sources, which are still not there.

Aashka Patel (15:26)

Yeah.

Mm.

Hmm. Hmm.

Mm-hmm.

What?

Deeba (15:47)

So depending on

any other reason, so depending upon the quality of data they have, we will make recommendations around what type of model they should be developing, whether they should be going with an expert scorecard, what we call as an expert scorecard, which is just based on business judgments, because there is no data to train. So if there is zero data, so there is no training data available for the AI algorithms to learn the patterns, right?

Aashka Patel (16:02)

Mmm.

Yeah.

Deeba (16:17)

recommend them to go ahead with an expert model, which can be implemented, created and implemented through a product called DecisionX, which is a business rules engine. through simple rules, you can have an expert judgment-based model. So that’s stage one. And if there is good enough data, then you go ahead and develop an AI model. Again, we take a call between going ahead using a traditional

Aashka Patel (16:17)

Mm.

Okay.

Yeah.

Yeah.

Deeba (16:46)

approach or an advanced approach again, depending upon the data quality. So those recommendations is something that we provide. Stage two is built in the product. So we have a specific data curation pipeline which takes care of all the issues in the data, whether it be missing value, anomalies in the data, fluctuations in certain distributions.

Aashka Patel (16:59)

Good.

Yeah.

Deeba (17:11)

bias in the data, imbalanced data. So all of those which can be curated through algorithms are built in the product to take care, to maintain the quality.

Aashka Patel (17:18)

in.

Yeah,

does your product also gives the ability to create those model cards that are required by many regulations like the model cards or the system cards that we see with OpenAI models or

other AI models. So do you also help them create those kind of documentation Got it, got it. Yeah.

Deeba (17:49)

Yes, yes. All the required documentation

are generated automatically and it gets downloaded at a click of a button.

Aashka Patel (17:55)

Mm-hmm.

Okay, got it. So the one thing that I loved while I was researching for this podcast was the AI toggle feature. it’s very interesting. Since I’ve been a product manager, and I’ve seen that people are, overusing AI or maybe like applying AI to problems where AI is not even needed. Traditional ML can do the work for us.

Deeba (18:22)

I that’s

Aashka Patel (18:24)

So that’s an amazing feature to have. yeah. Yeah.

Deeba (18:27)

Absolutely. Yeah, absolutely. That’s very much required because

one should make a decision to whether to use AI or use a traditional approach, looking at some stats, Not just go ahead and if I feel like I go ahead, use it. So that’s more intentionally included in the product. So people who are AI ready, they can go ahead and start using the advanced one. Who are starting new, they don’t want to take that jump.

Aashka Patel (18:37)

Thank you. Mm-hmm. Mm-hmm. Yes.

Yeah.

English.

Good

Deeba (18:57)

Start with a traditional approach, check out the results they are getting using AI. if they see a drastic improvement, then they can make their own choices on the basis of some stats rather than human

Aashka Patel (19:00)

Yeah.

Yeah, yes, yes, yes. That’s very powerful. So since we touched upon bias a little bit in your answers, so let’s talk about bias monitoring. So you have talked about identifying and eradicating biases. So can you share a specific instance where your post-deployment monitoring caught the model that had drifted and become less fair over time? And what

demographic got hurt and how did you recover it and what was your 48 hour response plan?

Deeba (19:43)

Yeah, so Aashka in today’s day and age where our environment is so dynamic, population shift in your model is eventually going to happen. And unlike earlier days when the model used to be stable for a year, two year, it’s not happening these days. And if it is happening, there’s something definitely wrong because there are changes happening.

Aashka Patel (19:52)

Yeah.

Happen,

You’re wrong.

Deeba (20:12)

Either you’re not monitoring it carefully or wrongly monitoring it. So what we have in place is automated monitoring reports. So every week these reports get generated. Usually what the reports capture is, I’m talking about the regular approach, they only capture the performance of the model and if it’s doing good, they’re okay. However, your model can still be doing good, but there can be a population drift, right?

Aashka Patel (20:17)

Yeah.

Boring.

Really.

Deeba (20:41)

like they are able to capture bad from goods, but the cutoff at which you started defining your bad versus goods might now need to change because there has been a drift in the population. Now this drift can be based on either gender. So maybe now more females, we are seeing so many females more actively applying for credits these days. Earlier it was to be only the males of the house.

Aashka Patel (20:41)

Yeah.

Mmm.

Yes.

Deeba (21:07)

that were going to the banks and doing those activities. So that’s there. Then age also, a lot of younger people. Earlier we used to see only mid-aged people applying for credit Now the youngest people are also getting jobs and they are applying and these trends are shifting from time to time. So what we have captured in our reports are these bias aspects as well.

Aashka Patel (21:08)

Yeah.

Yeah.

Yeah.

Deeba (21:37)

how well the model is doing across different gender segments, age groups, ethnicity, and if something goes haywire, it will send alert to the credit modeler and he can go back and refresh the model very quickly on the most recent data and then push the model to production. So that can happen within a period of one or two days that you can take that action and stop any kind of...

Aashka Patel (21:47)

Thank

Boom.

Yeah.

Deeba (22:06)

wrong decisions being made.

Aashka Patel (22:09)

Yeah, that’s a very good incident response plan also, Like having that quickness and everything. So since you mentioned about this, you with some customers you

try the non-AI or the traditional ML approach first and then based on the stats then you help them get into the AI component of things. So you must be working with some banks who have been using legacy systems like hardware, infrastructure and everything.

So what have you found works best for rolling out AI in environments with old infrastructure, especially in banks across emerging markets because you have been working with emerging markets a lot. So any lessons from that?

Deeba (22:56)

So the good thing was that we did this research earlier before completing the development of our product so that we can take care of all the reluctance or hurdles that are there from the flying side in terms of IT and in infra and tech So that was one important research that we did at the very beginning. And that’s where we decided that we are going to develop a lightweight containerized solution.

which can be deployed in any environment. So there are no restrictions that this needs to be deployed only on cloud or our public cloud. No, the product can be deployed on-prem, private cloud, or on our cloud. But we do mention the benefits of deploying any solution on cloud versus on-prem so that later on, if they want to make that shift, they can do it with our help.

Aashka Patel (23:29)

Mmm.

Green.

That’s a good transformation that they can have. We have talked about human oversight a lot during this conversation.

I have an odd analogy. Like when we shop at a supermarket, there’s often someone at the exit checking your bill against your cart. Like sometimes they verify every single item. Other times they just scan one or two items at random. if you have an unbilled item, you would definitely get caught with the thorough check, but you might slip in just with a spot check.

So in like how much amount of human oversight is enough would you like someone to be manually reviewing every decision or does random checking work in this high risk scenario?

Deeba (24:39)

Okay, so as you rightly mentioned, it’s a high risk scenario. So we have to be very careful where we are allowing automated decisions and where there is a need of human evaluation. So there are segments at the top and the bottom the size that the model is extremely confident and we know the level of confidence that the model has. All those stats are available. So we put cutoff so that

Aashka Patel (24:50)

Okay.

Yeah.

Deeba (25:06)

anything passing through these regions can just go straight through. Whatever decision is being made by the model, just go straight through. But wherever we know there are risky segments, and within the risky segment, there is also high, medium, low. We channel them through human evaluation, but different degrees of human evaluation. There can be where the model is very less confident about the decision, then

Aashka Patel (25:11)

Okay.

Good night.

Deeba (25:34)

it will go through a longer process of human evaluation, maybe asking the customer for more documents or something. But if it is less risky or edge cases, we would, the human evaluation would be just review the reason codes if it looks fine, then pass through. So those kind of categorization in terms of how the output should be rolled through is taken care of within the solution. It’s just that the cutoffs

Aashka Patel (25:41)

Hmm.

Boom.

Okay.

No.

Deeba (26:02)

can be adjusted accordingly for different models.

Aashka Patel (26:04)

my.

Got it, got it. But this cutoff that you are talking about might have evolved over a time, right? Like when you just started with Finbots or this product, like was that the case from the very beginning or did any incidents happen?

Deeba (26:19)

No, Yeah,

earlier we started with binary decisioning, which was accept, reject. But then of course, clients were quite reluctant about we cannot just do state, say true for everyone. We have to be able to get deeper into the kind of the quality of output that’s being generated in terms of the confidence. And then we segmented further for the type of action to be taken on

Aashka Patel (26:26)

Yeah, okay.

Yeah.

Yeah.

green.

Got it, got it, makes sense, makes sense completely. since we talked about this human oversight thing, the world is moving towards more and more autonomous AI. And recently there have been reports around AI agents executing unauthorized financial transactions despite safety guardrails.

in your Finbot scenario, it’s credit scoring or credit rating. So a similar vulnerability could mean wrongful loan approvals or denials affecting thousands. So what specific guardrails or safeguards have you implemented that you believe most FinTech AI systems are missing these days

Deeba (27:16)

Yes.

the first thing I feel is in a high risk area, people shouldn’t rush through the AI wave that’s happening. I understand to adapt something for marketing, for campaign offers, that’s absolutely fine. But if something as important as credit risk, with their one decision,

Aashka Patel (27:35)

Yeah.

Good morning.

Mm.

Deeba (27:47)

And just think about very high ticket size loan, where one decision, wrong decision can lead to huge, huge, huge loss. So the guardrails there has to be very, very well thought upon, researched, tested, evaluated over a period of time. And it has to be rolled out in a phased approach. So that’s.

Aashka Patel (27:57)

Yes.

Bye.

Yeah.

Deeba (28:14)

very important rather than just developing something and really feeling happy about it and sending it across. And also it should be used in patches. For example, there is an overall modeling life cycle and decision making process. So not in one go you are including everything in one go where testing and evaluation and tracing back the issues become a huge challenge.

Aashka Patel (28:20)

Yeah. Yeah.

Mmm.

replacing everything.

Deeba (28:42)

You doing it in patches. We are also doing, but we are not replacing the entire thing by LLM or, or AI agent. We are doing it in patches, for example, for data. So we have brought in the VLM which is visual language models to transform and let’s say image data or scan copy, digital copies, and then to extract information from all of that, to use it for decision. Yeah. So it is, it should happen, but yeah.

Aashka Patel (28:43)

and good night.

Thank you.

No.

Thank you.

Deeba (29:12)

it happened in patches where you can very well validate it.

Aashka Patel (29:18)

Yes, yes, makes a lot of sense.

so what’s the most sophisticated new fraud vector that you have encountered in the last six months that specifically targets AI powered lending systems?

And how are fraudsters using tools like deepfakes or synthetic identities and what’s your counter-strike against them?

Deeba (29:37)

Yes, deep fakes and identity fraud. See, identity frauds have been happening for a very long time, but there was no solution for it. But now with, of course, the new AI out there, it’s a boon and a pain. So the pain is there will be more frauds and new types of frauds. But yeah.

Aashka Patel (29:45)

Yeah.

Thank you.

frauds.

Deeba (29:58)

So

a deepfake might not be, it might be difficult for a human eye to identify deepfake, but the AI again is much more capable of identifying deepfakes. So the need is just to keep updating your AI algorithms to identify fraudsters to include all these aspects which can then be taken care of.

Aashka Patel (30:04)

Yeah.

Thank

Any recent incident that has occurred that you would like to share? Beware of such.

Deeba (30:28)

No, think people, bands are

much more vigilant. they will introduce us. They ask us to introduce these things much before they encounter anything like.

Aashka Patel (30:34)

Yeah.

Yeah

Yeah, yeah, makes sense. Yeah, so you have worked across India, Singapore and other emerging markets. So can you share one specific insight from African or Southeast Asian fintech that could revolutionize how Western banks think about credit decisions like something that they have been missing and the South region is doing much well, Southeast region.

Deeba (31:08)

I think the scale at which the banks and the fintechs really work in this part of the region is immense. that scalability is absolutely missing in that part of the world and diversity in the data being used. Of course, here there is a need because they don’t have formal

Aashka Patel (31:11)

Mm-hmm.

in the room.

and

Deeba (31:30)

credit bureaus and data repositories to use. But because of that, people here have become very creative in the sense of what are the different data sources they can tap into and use it for their credit risk assessment for other purposes as well as the marketing, digitization or any other space. So a lot of work is being done in the space of using alternate data source.

Aashka Patel (31:42)

Mmm.

Deeba (31:58)

which we hear a lot in that part of the region, but to be honest, it’s not being implemented that well as it is being implemented here because of course the need over here is much more digital wallet data, transactions data, your telco data, your website data, digital footprints.

Aashka Patel (31:59)

Mmm.

Mm-hmm.

nice, that’s interesting,

when

Deeba (32:22)

We cannot even, like five years back, couldn’t have thought about Southeast Asia tapping into these data sources and using it. this is reality. This is what we are doing. And they are doing it at scale.

Aashka Patel (32:29)

Yeah.

Yeah, that’s very important. Yeah, so let’s move on to AI literacy. AI literacy is also a huge part of the EU AI Act.

Deeba (32:44)

Thank

Aashka Patel (32:45)

for non AI companies, it’s like legal services require advanced proficiency while marketing departments need only basic level awareness and everything. But for AI companies, what does it mean to be AI literate?

Deeba (33:00)

Being an AI company, we are really responsible of the knowledge that we have and the knowledge we are spreading. So if we call ourselves the AI company, we have to be up out there and have the latest information available and also assess it well. So if I’m going to talk about something, I should know what are the pros and cons. I cannot just read two articles and give a speech on a new algorithm.

Aashka Patel (33:08)

and then.

Amen.

can you?

Deeba (33:25)

We always,

what we have is a small research team. So whatever new comes up before talking about it or even making a decision around implementing it or including it in our solution, we have a research team where we run a couple of experiments to get deeper into it and to understand what it is, what actually it is and what is being talked about. So that’s our knowledge base.

Aashka Patel (33:50)

Yeah.

Morning.

Deeba (33:56)

First responsibilities with every new thing coming in, create a right, trustworthy knowledge base. And that’s the AI team. That’s the responsibility of the AI team within the AI company. But not everyone is part of that. But then once we have that information, we make sure that we spread it across. Of course, there are monthly meetings where we introduce it to the other team members also. And the very important part about being part of FinBot is

Aashka Patel (34:11)

yeah.

Amen.

Deeba (34:25)

even people, whether it be marketing, whether it be engineering, whether it be finance, everyone knows the product inside out. If you go and talk to anyone about the product, they will be able to. Although the AI product is very technical, but that’s the basic goal that we have set up for everyone. So you cannot keep working for an AI company and say, I’m from marketing. I don’t know anything about it.

Aashka Patel (34:34)

Yeah.

no.

Yeah.

Yeah, yeah, yeah, that is actually very powerful. Like, I have worked with a company where the salespeople used to come back with questions on security, privacy And they are just knowing how to sell the product and what the product is like just on the outside or the surface level, but they don’t know the nitty gritties, the safety, privacy, security nitty gritties.

Deeba (35:02)

you

you

them.

Aashka Patel (35:11)

So I have seen that in the corporate That’s very powerful literally great work. when we talk about this, Anthropic’s economic index shows AI use leads more towards augmentation, like 57 % as compared to automation, that is 43%. So it also reveals like usage.

Deeba (35:12)

you

Aashka Patel (35:37)

peaks in limit to high wage occupations like computer programmers and data scientists. So from your economics background and hands on fintech experience, how is this reshaping the skills that fintech companies need and who is getting left behind in this scenario?

Deeba (35:53)

See, whoever is thinking that this is not going to go too far, they will be left behind. So I think the problem first to tackle is the mindset rather than the skill set. So having the right mindset and going with the time is very much important. And that’s why adapting to what is coming in new. So like computers came in, internet came in, then...

Aashka Patel (36:00)

Yeah.

I mean

Deeba (36:20)

traditional modeling came in. So with everything coming in, we always thought that this transformation will lead to job losses, job losses, job losses. It will happen if you keep sticking to your current skill set. it’s always very important to adapt, whether it be data scientists, now from coding, they have to move to where should this app apply. So it’s not like the opportunities are going this.

Aashka Patel (36:29)

Yeah.

Bye.

Deeba (36:48)

new technology, new algorithms are going to open new opportunities. So opportunities are there. It’s just that you have to have the right mindset and transform your skill set to tap into those opportunities. Then your jobs won’t be, your jobs will be secured.

Aashka Patel (36:50)

in one.

Good night.

Yeah.

affected,

replaced. Got it, got it. So can you point, of course, like can you point some specific skills that FinTech will be demanding in the future that people can develop alongside the actual hardcore skills that are required, but like any other skills that you can point out?

Deeba (37:28)

Yeah, so prompt engineering is very, very, very important. Now, when I say prompt engineering, it’s not just a prompt engineering that we do to plan a vacation or something like that, but for our Inbuilt solution. So how you’re asking, what you’re asking, it’s exactly like when you have a big business problem to solve for, you always break it into tasks.

Aashka Patel (37:35)

Mm-hmm.

Yeah, very complex level.

Amen.

me

Deeba (37:56)

and or smaller

Aashka Patel (37:57)

the best.

Deeba (37:57)

problems depending upon what it is. And then you assign a different process to solve for it. So prompt engineering has become that important. So that’s one. And second space where you have to, anyone has to become more developed and have great skill set is on the validation and guardrail.

Aashka Patel (37:59)

Mm-hmm. Mm-hmm.

important.

Deeba (38:22)

You can keep developing things, but you have to come up with a skill set on what type of guardrails should be there. It should be now that a guardrails AI based human in the loop. Again, another layer of LLMs, multiple layers of LLMs. There multiple ways of doing it, but when what should be adapted, how to set up the entire process for it. So a lot of.

Aashka Patel (38:24)

Mmm.

No.

No.

Deeba (38:48)

thinking

through, thinking from solving a problem kind of mindset needs to be there. And of course, how to set up that overall architecture, so architecture design. are coding and coding and all is good, but then those skillset will become less important and these skillsets will become much more important.

Aashka Patel (38:55)

my

Yeah.

Thank you.

Yeah.

Yeah, yeah, I agree to your point. like, of course, if the employees know prompt engineering well, then it will cost less to the company as well. So they can write very contextual and precise prompts and get the output that is needed. So one last and deep question. Sorry.

Deeba (39:22)

Every hit and miss is a cost to the company.

Aashka Patel (39:35)

Yeah, exactly, exactly. Yeah, yeah. So last question. The UBI discussion is going on, right? The universal basic income.

So studies have predicted AI will displace about one third of existing jobs worldwide within a decade and universal basic income is gaining momentum as a policy response. So this creates a paradox for financial services, especially with your AI credit inclusion tools, like they expand financial access right? But the same technology might eliminate the jobs that create income streams to service those loans. So from your economics background,

How do you like, or do you see AI financial inclusion and UBI as complimentary or competing approaches? And practically how would you design credit models for a world where a significant portion of borrowers might receive guaranteed government income rather than traditional employment? how would this world look like? Paint us a picture.

Deeba (40:34)

I think it’s a transformation.

think the AI AI advancement and UBI are going to be complementary and will help this transformation to go peacefully rather than creating a lot of disruption in the society and in different economies. Talking about the need of credit, no, the need of credit is going to be same or increasing over time irrespective of the UBI coming in because

Aashka Patel (40:38)

Good night.

Mmm.

Mm-hmm.

Deeba (41:03)

Credit basically gives you more power, more purchasing power, while being able to repay your credit with the help of UBI that’s going to be provided. So it’s not going to impact financial inclusion. It’s not going to impact the credit industry. In fact, it will increase more demand for credit because there will be more products and services and other things that people would want to use at the same time they didn’t have the sorority.

Aashka Patel (41:08)

Power.

No.

Deeba (41:31)

of an income flow coming in through the UBI. But I feel all of this is going to be just transformational. After the transformation is over, we have gone through the other side. People will be settled in and they would have identified their respective space where they can contribute and start earning. Because there are things that will get disrupted. The jobs will go, but again, it will definitely create more opportunities, newer jobs.

Aashka Patel (41:36)

Mmm.

Bye.

Mm-hmm.

more.

yeah.

Create new, yeah.

Deeba (42:01)

for people to get employed into.

Aashka Patel (42:05)

Yeah, yeah, yeah. So it will be not a steady transformation that we will go through, but after that, the grass will be greener on the other side. Yeah, yeah. That’s very important. So one last piece of advice for FinTech founders, like.

Deeba (42:08)

New world

again.

you

Aashka Patel (42:24)

Firstly,

let’s start with the biggest mistake that you have seen them making. And secondly, what would be the one piece of advice that you would give to fintech founders trying to survive in these environment of tight regulations and aggressive regulations?

Deeba (42:39)

I, the first mistake I would say is thinking too much can be harmful because I think if you keep on thinking and planning and not really doing things, you will not know what’s working, what’s not working. So in the beginning, when we were developing the product, we were like, no, we will think through this perfectly. And then only we will start the implementation so that the development goes smoothly.

Aashka Patel (42:44)

Hey

Uh-huh.

Mm-hmm.

Bye.

Deeba (43:07)

So that’s one example, but that can happen with anything with any kind of AI adoption. By the time you, if you’re taking too long to think and plan, there will be 10 new things that would have, would have come up, could replace 10 things in your plan already. So now it’s the time to learn, to learn quickly, develop quickly, fail quickly. It’s okay to fail. You are bound to fail. If you don’t fail, also you’re doing something wrong.

Aashka Patel (43:13)

Good night.

Yeah.

in the name.

Mm. Mm. Mm.

Deeba (43:38)

to learn again and then get back to track of development. that’s the thing. I really realized that sitting back, thinking and planning, those were the old good days where we used a month or two, one to do the planning only. here, yeah. And the thing,

Aashka Patel (43:38)

It’s wrong, yeah.

Yeah.

Yeah, the waterfall model in a way. Yeah.

Deeba (44:04)

The learning for me, so this is the thing that I’ve changed about myself. The other thing that I have kept is, as mentioned, me keeping myself relevant and keeping myself updated and creating my own knowledge base to make my decisions. Because there is so much coming up every day and at times it gets over whelmed

Aashka Patel (44:05)

Mm-hmm.

And you’re.

Morning.

Mmm.

Deeba (44:30)

with the kind of information that’s going in, the kind of research and development that’s happening. So there’s always a requirement of a reality check. What’s true, what’s just in the air. So that is something that I’ve always done. And I really want to continue doing it.

Aashka Patel (44:32)

Yeah.

Mm, yeah, yeah, yeah, yeah.

Mm-hmm.

Yeah, yeah, that’s very powerful. are there any go-to sources that you like refer to? Because with AI, content generation has become huge and like referring to the right resources is also a hassle. Can you point any specific resources or people that you follow to get the latest updates or keep yourself updated?

Deeba (45:19)

Anything

that’s not supported with proper research in terms of a research paper with code and data, I don’t fall into that so easily and nobody should fall into it. So go for well-reputed research papers which are trustworthy.

Aashka Patel (45:24)

and

Amen.

yeah, we yes.

Even.

Deeba (45:44)

and are supported by codes and data, which you can very quickly check on your side as well. Articles and all, it’s good to read, but again, as you mentioned, the content in LinkedIn articles, don’t know how, not even one percent of the people are writing it themselves and you can make it out very easily. So I don’t think for those are...

Aashka Patel (45:44)

Mm-hmm.

anyone.

Okay.

Yeah.

Yeah.

Deeba (46:12)

the right sources of information to gather knowledge from But the re search papers are always my go-to place to get that trustworthy information.

Aashka Patel (46:14)

Come on.

.

Yeah, yeah, yeah. Like a proper data scientist kind of a response. just, yeah.

Deeba (46:32)

Yeah, and for people who don’t have a technical background, for

them, I think it’s difficult for them to really, they want to know what’s happening in AI. Look at articles where it’s been talked about the AI being actually implemented.

Aashka Patel (46:49)

No.

Deeba (46:56)

Yeah. So if they are talking about it and they have implemented and they’re talking about something they have implemented, it’s OK to trust that if it’s publicly available, try it out. But if it’s something in the development stage and they are saying that they will do this and that, it might be all in the air unless and until Because to be very honest, there are thousands and thousands of pilots that are being run today.

Aashka Patel (47:01)

Okay.

Mm-hmm. Mm-hmm.

Evening.

Thank you.

morning.

Yeah.

Deeba (47:24)

in the GenAI space with in-reputed consultancies and big tech firms. But if you talk about what’s in production, you will not get an answer. So if you really want to do a quality check, ask what’s in production and what results.

Aashka Patel (47:25)

You know, anyway.

Yeah.

Mm-hmm. Mm. Yeah. Yeah.

Yeah,

that’s a shrewd response and a shrewd way to kind of cut through the hype and make a judgment on your end. Like, okay, this is to be trusted and this is not to be trusted. yeah, completely makes sense.

Deeba (47:53)

you

Aashka Patel (48:00)

So thank you so much, Deeba Thank you so much for your time. And I think right on time we are completing this. Let me stop the recording.

Deeba (48:01)

you

Yes.

you

Watch on YouTube; listen on Apple Podcasts or Spotify.

Episode Summary:

In this conversation, Aashka Patel engages with Ryan Carrier and Jimmy Farrell to explore the critical themes of AI audits, governance, and the implications of AI on job displacement. They discuss the necessity of independent audits for AI systems, the challenges posed by autonomous systems, and the importance of incident reporting as outlined in the EU AI Act. The conversation also touches on the need for international coordination in managing AI incidents, the ethical considerations of AI's role in society, and the future of AI compliance initiatives. The speakers emphasize the urgency of addressing AI-related risks and the potential for significant economic impacts due to job displacement.

Timestamps:

00:00 AIR Bites (Precap)
01:52 AI Audits EXPLAINED ft. Ryan & Jimmy
02:50 Sam Altman: Genius PR or Guilt Trip?
05:13 Will AI Kill Jobs? UBI vs Taxes vs Sovereign Funds
09:30 Auditing AI: The Next $55B Industry?
13:22 Why Auditing Autonomous AI Is 10x Harder
18:05 AI Incidents: From Risk to Reality
20:37 EU AI Act: What MUST Be Reported
23:37 Do We Need a CERN for AI?
25:52 What Happens AFTER an AI Incident?
28:24 Maternal Instincts in AI?! Geoffrey Hinton's Idea
32:44 AI Alignment Is Broken: Here's Why
35:57 India's AI Law & Free Auditor Training
40:11 Australia's AI Stress Test: What They Found
43:10 AI Liability Law Killed by Lobbyists?
45:31 AI Suicides, Deepfakes & Agents Gone Wrong
47:49 If AI Harms You, DO THIS
49:30 Final Thoughts & AI Risk Literacy
49:53 Outro

Transcript:

Aashka Patel (00:04)

hello and welcome to On AIR with Aashka Today we are stepping right into the heart of AI audits and AI governance and joined by Ryan Carrier and Jimmy Farell. Ryan is the executive director of For Humanity, the global nonprofit building independent audit for AI and algorithmic systems. With 25 years in finance, Ryan knows what it takes to make audits credible, enforceable, and essential for trust.

Jimmy Farrell is the EU AI policy lead at Pour Demain where he works on EU AI regulation, incident reporting, and codes of practice. Jimmy’s work directly shapes how we respond when AI causes real-world harm, from financial loss to human rights violations. So welcome to the show, Jimmy and Ryan.

Ryan Carrier (00:52)

Thank you.

Jimmy Farrell (00:53)

Thanks very much for the lecture. Pleasure to be here.

Aashka Patel (00:55)

So let’s dive right into the questions. Sam Altman had spent $60 million on the largest UBI universal basic income study in US history, giving people $1,000 a month for three years So the study found UBI gives people flexibility and autonomy.

but didn’t solve deeper problems like housing or chronic health issues. Meanwhile, Altman just told the Federal Reserve that entire job categories will be totally, totally gone due to AI agents. So the same guy who is building the technology that displaces workers is funding research on how to support them afterward. Is this tech billionaire guilt or brilliant PR? What do you think?

Ryan Carrier (01:42)

From my perspective, neither of the above. It’s the ultimate conflict of interest.

Jimmy Farrell (01:43)

I’m gonna go home.

Aashka Patel (01:45)

Hahaha!

Aha, yeah, yeah, makes sense. And like, should policymakers instead force a job displacement tax on these frontier AI labs to pay for those safety nets? what do you think is working and not working in experiment?

Jimmy Farrell (02:07)

From my perspective, think the data on job displacement is still pretty early and there’s lots of studies going both directions. Some saying that it’s not as bad as some would expect, others saying that a lot of entry-level coding jobs are basically evaporating. And of course, these like just show some years to show in the labor market. So I think it would be good to potentially have more solid data before something like

Aashka Patel (02:25)

Mm-hmm.

Yeah.

data.

Jimmy Farrell (02:38)

a tax could be placed on this, but in general, the fact that AI and big tech companies often don’t pay too many taxes. Also in the EU, there’s been a lot of cases in which huge companies have not paid a lot of taxes. The EU and a particular number states have thought a bit about, for example, just the digital services tax. So this would be more general than a job displacement tax.

The political will is growing for that, especially with European tech sovereignty interests. And that could, for example, morph into a job displacement tax.

Aashka Patel (03:17)

Yeah, yeah, makes sense, makes sense. Ryan, what about you? What are your thoughts?

Ryan Carrier (03:21)

So

I wrote a paper that’s years ago now, essentially advocating the US, UK, most of Europe ought to start their own sovereign wealth funds simply because they will have to pay for what we imagine will be 30 to 40 percent entrenched unemployment. There are already concrete examples, even in the United States, a major US law firm

Aashka Patel (03:23)

Mm-hmm. Mm-hmm.

Mm-hmm.

Ryan Carrier (03:49)

based out of New York City doesn’t even hire junior advocates anymore. Why? Because they don’t need them, which creates an enormous downstream problem for becoming an experienced advocate or experienced lawyer. This shows up not in unemployment because these persons were never employed. It shows up in a lack of hiring.

Aashka Patel (03:53)

Hmm. Yeah.

Mm-hmm.

Smooth.

Ryan Carrier (04:14)

And so what we see is first this becoming a pervasive problem. But the nature of this sovereign wealth solution is the idea of having to pay for the very UBI that Sam Altman was just testing out for. We don’t have a current mechanism for such a thing. And so what we may find is that governments may need to participate with the very companies that are refusing to hire the people in that country moving forward.

Aashka Patel (04:33)

Hmm.

Ryan Carrier (04:44)

Unlike most technology in the past, it’s really a matter of perspective. So if I say to you, what does horse unemployment look like today? If you think about it, horse unemployment has persistently remained at 99.9 % unemployment for about 100 plus years now. And that’s because when we built our technologies, that was what we were able to replace.

Aashka Patel (05:13)

Hmm.

Ryan Carrier (05:13)

And for a

solid 150 years, the industrial revolution, we did not have the tools to fully replace humans. Now we are in a different place. That’s why we call ourselves in the fourth industrial revolution, right? It’s because for the first time we have the ability to fully replace humans in nearly all tasks that we engage in. And the end game for that in capitalism is that those tasks and those jobs will be replaced. Our tax system.

Aashka Patel (05:20)

then

Yeah.

Hmm.

Ryan Carrier (05:42)

incentivizes using machines and automation over employees. These machines can work 24-7. They don’t take vacations. They don’t have to stay home when their child is sick. And so what we will see is an increase in large corporations automating tasks. The results of that may be that we see lots of micro jobs, gig workers. So we may not recognize it as full unemployment.

Aashka Patel (06:02)

Yes.

Ryan Carrier (06:12)

But what it will result in is an income inequality where a lot of the wealth accrues to more and more people and less and less employees. And we’ve already seen that shift for a hundred years. The net result is that we have to think about what the future of work looks like. And it is an enormous challenge. There is no point in advancing artificial intelligence unless it’s replacing jobs.

Aashka Patel (06:23)

Hmm.

Hmm.

Hmm.

Ryan Carrier (06:41)

from the capital owner’s perspective. That is where the cost savings come from. Now they may empower the existing employees to be a lot more successful and leveraged and accomplish more work, but it then also means they’re not hiring more employees. So we may see this technological unemployment manifest in many different ways and creative solutions to meet that challenge are necessary such as

Aashka Patel (06:45)

Hmm.

Okay.

Yeah.

Ryan Carrier (07:10)

starting sovereign wealth funds that maybe participate in the equity capital, the growth of these companies that are allowed to automate because we don’t have any laws that say you can’t automate, you must hire people. We don’t do that, at least in the West.

Aashka Patel (07:25)

Hmm.

Yeah, yeah, yeah. Makes sense.

So Ryan, you have spent 25 years in finance, where independent audits are legally required and auditors make serious money, right?

So now AI regulation is creating similar compliance demands from your finance background. Like do you see independent AI auditing becoming as lucrative and essential as financial auditing? And should entrepreneurs be rushing into this space? And also can we see someone today creating the Deloitte or PWC of AI tomorrow?

Ryan Carrier (07:59)

So I’m going to answer this in a couple of parts. Number one is we believe it is necessary to establish sufficient trust due diligence capabilities by people buying and deploying these tools, that there be auditability or some level of assurance of what compliance steps have been taken inside of all the developers, providers, deployers of these tools. As a result of that, when asked,

The one policy initiative that For Humanity advocates for is mandatory annual audits of all what we call AAA systems. We don’t just say AI, it’s AI, algorithmic and autonomous systems. That’s our scope, but that’s our scope, okay? So with that foundation, if you subscribe to that belief and that future legal obligation, in 2021, the financial audit global revenue,

Aashka Patel (08:37)

Yes.

Yeah. Yeah.

Ryan Carrier (08:56)

was $55 billion per year. I believe that the space of auditing AAA systems will be bigger than that inside of 10 years. Today in 2025, if that revenue is more than $100 million, I would be shocked. Okay, it’s not measured, it’s not well tracked, but I know who’s doing business and what they’re doing in the space. If it’s more than a hundred million, I’d be very surprised. So what that tells you is if you believe

Aashka Patel (08:58)

Okay.

Hmm.

Okay.

Mm. Mm. Mm. Mm.

Okay.

Ryan Carrier (09:25)

what we believe that we’re gonna grow from a hundred million to more than $55 billion a year. The answer is unequivocally, yeah, this is a space people need to be in, especially in light of our previous points about jobs disappearing elsewhere, right? Shea Brown, who runs the algorithmic auditing firm has, exactly, at Babl AI has referred to auditing algorithms as the last human job. And that makes some sense, right?

Aashka Patel (09:33)

Yeah.

Hmm

Beverly. Yeah.

Yeah!

Ryan Carrier (09:55)

And so if that is accurate, then, you know, sort of upskilling or re-skilling, this is a space that would be ripe for opportunity, whether that’s as an advisor, consultant, teacher, auditor, or building technologies to facilitate this whole process.

Aashka Patel (10:14)

Got it, got it, got it. So can you name any big players who are already working into this space like apart from Shea Brown that you mentioned like Babaliya? Yeah.

Ryan Carrier (10:23)

Well, it really depends

on your perspective. have a Credo AI and other technology providers who are facilitating compliance. You have the big four who are providing some advisory services. Let’s be clear, especially from For Humanity’s perspective, no one in the world, no one in the world could meet the audit criteria we have, we have put in place. Okay. So what that tells you is they have to be going through the process of building

Aashka Patel (10:29)

Yeah. Yeah. Yeah.

Hmm. Hmm.

Hmm.

Ryan Carrier (10:53)

audit compliance, just the same as there’s no one right now who’s ready to be compliant with the EU AI Act. It’s the same concept, right? So we’re in the pre audit compliance capacity building phase. And then as that begins to grow, the opportunities will continue to grow. The way I would estimate it is out of a hundred companies, one to two to five of them are, basically spending

Aashka Patel (10:59)

Yeah.

next

Mm.

Ryan Carrier (11:20)

in this space to build capacity now. And that’s another area of growth, obviously, as we move to essentially 100 out of 100 companies.

Aashka Patel (11:23)

Hmm.

Yeah, makes sense, makes sense. So you pointed about the AAA approach that for humanity is taking towards auditing. auditing a model is like grading an exam, like the answers are fixed and you just check them. Auditing an autonomous system is like grading a student who has the tendency of changing their answers while the test is still running.

So in that scenario, like how does independent auditing differ for a model versus an autonomous system?

Ryan Carrier (12:00)

It’s a great question. So model data and concept drift are legitimate risks associated with AAA systems. And the way I would just, well, bound to happen, but then the question becomes whether that drift is legal and ethical, okay? So the way that we describe this is that if you’ve agreed to use a tool that I’m providing,

Aashka Patel (12:06)

Yeah.

Yeah, and they’re bound to happen.

Hmm. Yeah.

Ryan Carrier (12:28)

and I’m using my hands, sort of imagine them shoulder width. I know we’re sort of perpendicular here. So you can’t see that wide on us, okay? But if you and I agree that the purpose looks like this, okay? And if the model, and I’m just moving my hands a little bit, right? If the model just drifts inside of those guard rails, it’s no problem. It’s still consistent with the scope, nature, context, and purpose with which you agreed, okay? And that is both legal and ethical. Here’s the problem. What if the drift,

Aashka Patel (12:33)

Yeah, yeah, I’m gonna like it. Yeah.

Hmm. Hmm.

Hmm. Hmm. Yeah.

Ryan Carrier (12:58)

and I’m way off the screen, right? What if the drift looks like this? We would argue that that is both illegal because it’s not consistent with the purpose with which you agreed contractually and it is unethical. So now what we ask, and my arms are stretched way out wide, right? As I bring them in, what I would ask you is where do those guardrails have to exist to be legal and ethical?

Aashka Patel (13:01)

This, yeah.

Hmm

Mm-hmm. Mm.

Ryan Carrier (13:24)

And the answer is no standards body is even approaching this conversation. No laws have even talked about this. So this therefore becomes what we call an ethical choice. So here’s how we deal with this. We say to all auditable entities, you need to establish your own key performance indicators to identify when you’re staying true to your purpose based on your use case.

Aashka Patel (13:28)

Yeah.

Hmm.

Ryan Carrier (13:51)

And you do this

Aashka Patel (13:52)

Thank

Ryan Carrier (13:52)

with your ethics committee who’s doing this based on their shared moral framework and on the laws and their code of ethics. So now what we ask and what we measure in an audit is number one, have you established your own guardrails? Number two, have you established the ability to measure that model data and concept drift, right? And then number three, have you put in place rules that say if you bump up against one of those guardrails,

Aashka Patel (14:07)

Hmm.

Yeah.

Ryan Carrier (14:20)

What do you do? Do you pause? Do you replace the tool? Do you recall the tool? So that’s year one, right? Now we’re tracking, am I drifting? Okay. So let’s say Jimmy does this, right? He puts it in place and this is what he establishes. And as his auditor, I’m like, fantastic, great. So now I show up next year and I say, how did you do? Now, if Jimmy bumped up against those guard rails a thousand times in the last 12 months,

Aashka Patel (14:25)

Hmm.

Mm.

Ryan Carrier (14:49)

and did nothing? Does he stand any chance of getting his assurance renewed in year two? The answer is no way. Not a chance, right? Because look, it’s his own rules. You establish your own rules and you can’t even follow them. So what that tells you is that you don’t know if your model data concept drift is violating that purpose, right? So therefore it is genuine teeth

Aashka Patel (14:57)

Hmm, no. Yeah.

Hmm.

Anyway.

Hmm.

Ryan Carrier (15:17)

for us to say, Jimmy, when we show up next year, follow your own rules. And if you don’t, you’re not gonna be able to be certified. You’re not gonna conform to the EU AI Act and so on and so forth. It’s legitimate teeth to ensure that the model stays true to what it agreed with its customers. And so this is how we audit something that learns and grows and changes over time.

Aashka Patel (15:24)

and

Hmm.

Hmm.

Ryan Carrier (15:45)

Does that make sense? that a meaningful example?

Aashka Patel (15:46)

Yes, yes.

Yeah, you explained like a teacher. Okay, okay. You have gotten great at your at your artwork then. Yeah, so you mentioned about this model concept drift. So like every AI incident is basically a risk that went from could happen to did happen. So like, since audits are all about managing the risks,

Ryan Carrier (15:51)

I’ve done this once or twice before, I promise.

Aashka Patel (16:13)

Are we considering the actual incidents while doing the AI audits?

Ryan Carrier (16:21)

From my perspective, the answer is yes for two reasons. Number one is all of, at least under the EUAI Act, okay? There has to be a process to evaluate whether an incident is serious or not, reportable or not, okay? So you have to have that process of evaluating, and we have a whole section called incident management to do exactly this. So now when you engage in incident management, you might have to report to your regulators and so on.

Aashka Patel (16:25)

Mm. Mm.

Hmm.

Okay.

Okay.

Ryan Carrier (16:51)

But all we are measuring again is do you have the processes and procedures in place to do this work? That’s year one. Year two, we’re going to say you had, I don’t know, 12 incidents and did you evaluate them for seriousness or reportability? Okay. And then what did you do? Did you execute your process? Again, your named processes and procedures. Did you execute them the right way? Or in the case of, of Jimmy’s work.

you know, was the incident report filed correctly, registered correctly with registration databases and national competent authorities. So we’re basically reviewing, those things happen properly? And if they don’t, the auditor is basically gonna say, I can’t assure you your processes and procedures are insufficient.

Aashka Patel (17:39)

Hmm.

Got it. Got it. So like, does this like enable any kind of friction in the company’s overall working process because of the auditor mentioned that we can’t audit or something like that? Like, are there any kind of strict punishments or penalties that could be given Like because of

Ryan Carrier (18:02)

strongest punishment is, look,

all high risk AI systems have to go through conformity, right? And if they are nonconforming, then they are violating the law. So that’s the strongest teeth that we have here, let alone fines or any other enforcement by regulators. having a nonconforming AI should be the biggest punishment that you have. Why? Because now all your downstream

Aashka Patel (18:05)

The fines? Okay, okay.

Hmm

Mm-hmm.

Mm. Mm-hmm.

Ryan Carrier (18:30)

deployers and so on simply cannot buy your product, which would, that’s how product liability, you know, frameworks, whether they’re law or market driven, that’s how they ought to work.

Aashka Patel (18:33)

Wow.

Hmm

Hmm.

it. it. Got it.

most of your work revolves around the AI incident reporting. So let’s get back to that. So MIT’s AI Risk Repository now classifies incidents across 10 harm vectors, physical damage, financial loss, human rights violations with severity scales from negligible to catastrophic.

and they found 65 % of incidents happen post-deployment and most are unintentional. So does the EU AI act demand this level of granularity in incident reporting? And if yes, can you double click on the exact requirements by the EU AI Act

Jimmy Farrell (19:22)

Yes, so there is some overlap. I would say it’s slightly less granular. Of course, it is a law, so it’s not just a suggestion. Yeah, of course, with AI Act there’s this like, there’s kind of two halves of it which have to be addressed. One side is the system side, AI systems, so the downstream level, and then the other side is the AI models, so the weights themselves.

Aashka Patel (19:29)

Mm-hmm.

Jimmy Farrell (19:49)

comparison would be GPT-5 is the model and ChatGPT is the system. There are slightly different reporting requirements for systems and models. I’ve been mostly working on the model side and in terms of this categorization comparing it to the MIT risk repository, there is some overlap. the recently published code of practice, which is kind of like the details of the general purpose AI model section of the AI Act

Aashka Patel (20:01)

Hmm.

Hmm.

Okay.

Jimmy Farrell (20:20)

and with the categories of harm to a person’s physical health, damage to property or the environment. It also mentions serious cyber security breaches such as like model exfiltration, like somebody stealing a model weight. It also includes infringements of fundamental rights which is very much connecting to EU.

Aashka Patel (20:31)

Hmm.

Yeah.

Jimmy Farrell (20:48)

treaties involving the right to privacy and that sort of thing. One thing that Ryan mentioned before, the threshold at which an incident becomes serious, this is still kind of a gray area. It’s something that’s very difficult to think down because the thresholds are, it’s very hard to compare what a threshold for an infringement of fundamental rights looks like compared to something that’s easily monetizable like damage to property or...

Aashka Patel (20:59)

Mm.

Yeah.

Yeah.

Financial

damage or yeah, infrastructure damage. Yeah. Yes. Yes.

Jimmy Farrell (21:19)

Exactly.

So these are some challenges. so compared to the MIT risk repository, like the code of practice details, like exactly when the reporting has to happen. And there are some like increased urgency on more serious incidents. know, if there’s a death of a person, it’s more urgent than if there’s been some property damage.

Aashka Patel (21:36)

Hmm.

Mm.

Yeah, yeah, makes sense. So there have been talks around creating a CERN for AI and you work on AI incident reporting. So CERN already coordinates cyber incident responses across 42 countries. When one site is breached, threat intelligence is immediately shared and defenses are updated everywhere.

So shouldn’t a CERN for AI work the similar way for AI incidents as well? Like if there is an AI incident happened in New York, shouldn’t the safety protocols for similar systems automatically update in London or Tokyo? what do you think

Jimmy Farrell (22:24)

I definitely think that international coordination will be really important for AI incidents. This is something that is not particularly addressed in the Code of Practice or the EU AI Act because reporting basically goes through the AI office or the relevant member state authorities. Of course, from the provider perspective, there are some...

Aashka Patel (22:39)

Yes, definitely.

Mm-mm.

Jimmy Farrell (22:51)

like serious concerns like, for example, reputational risk, if this is like shared with, you know, countries or governments for which the incident wasn’t necessarily relevant to, then that could be just sort of unnecessary and really like set them back. And of course, certain serious incidents have intellectual property and trade secret concerns. So that’s also something to consider in terms of international coordination.

Aashka Patel (23:03)

Hmm.

Hmm.

Jimmy Farrell (23:21)

course, AI is like inherently cross border. So there is a scenario in which a incident might happen outside the EU, but it’s relevant to the EU because it’s involved in model that is also placed on the market. And this is something that the AI Act accounts for. if it’s the same model, then this is something that

Aashka Patel (23:33)

Yes.

No.

Jimmy Farrell (23:45)

needs to be reported to the authorities. Yeah, think the comparison to like the actual CERN that exists is not quite enough. Yeah, yeah.

Aashka Patel (23:47)

Mm-hmm.

Hmm. Hmm. It will take some time to reach there, maybe. Yeah,

definitely, So like, I was curious about the immediate steps that are to be taken or that are taken currently once an AI incident is reported.

How does it work in the real world?

Jimmy Farrell (24:13)

So before the AI Act, were no requirements specifically for AI incident reporting. Like incident reporting, there’s been a bunch of other fields that exist in cybersecurity, exists in aviation, chemicals, these sorts of things. But it hadn’t existed until the AI Act for AI. Basically what happens is that there’s the...

Aashka Patel (24:21)

Yes. Yeah.

Hmm.

Jimmy Farrell (24:42)

Provider has to submit an initial report with like some quick information so that the authority whether it be the AI office or member state authority is aware of it very early and not necessarily like a very detailed report but just the basic information and then later on there’s can be an intermediate report if it’s an ongoing incident and then eventually as I mentioned there’s different timelines.

Aashka Patel (24:54)

Mm.

Yeah.

Hmm.

Mmm.

Jimmy Farrell (25:11)

different types of incidents, but eventually there will be a final serious incident report. And this is what is really going to be important for authorities to learn from and to prevent them in the future. this sort of thing, this sort of report involves what model it was, what company was involved, the date, of course, the type of damage, the harm. Importantly, it involves a root cause analysis. And this is something like

Aashka Patel (25:13)

Yeah.

Hmm.

Mm-hmm. Mm-hmm.

Yes, yeah,

Jimmy Farrell (25:40)

quite complicated in AI because

Aashka Patel (25:40)

that is important.

Jimmy Farrell (25:43)

establishing causality is quite difficult in inherently uninterpretable systems. So this is something that the code was just released, so we’ll need to see how in practice such a root cause analysis works. But basically the code is quite specific and particular about saying that there needs to be a deeper analysis of not only what

Aashka Patel (25:46)

Yes. Yeah.

Hmm.

Hmm.

Jimmy Farrell (26:09)

features of the model resulted in certain outputs happening or what input in terms of the training data or just like model inputs. So yeah, eventually there’s the final report and from that authorities will be able to move forward exactly.

Aashka Patel (26:14)

Yes.

Take action. Yeah,

yeah, yeah. Got it. Got it. the godfather of AI recently said something instead of forcing AI to submit to humans, we should build maternal instincts into AI models so they really care about people even when they become more powerful than humans. So Ryan to you, could you actually audit AI systems for maternal instincts?

Ryan Carrier (26:55)

First off, can I throw up at the thought? mean, I’m always asking the question why. I don’t even understand the pursuit of general intelligence. Why would we want to take our greatest advantage and subvert ourselves to a machine? We already have controllability problems. And then the quantification and parameterization of maternal instincts.

Aashka Patel (26:58)

Hahaha

Mm-hmm.

Hmm

Yes.

Ryan Carrier (27:24)

These things can happen. See what people confuse is tasks with things that cannot be parameterized. So if I take a care robot, let’s not even go to maternal instincts for a second. If I take a care robot for the elderly, I can absolutely break out tasks like listening. You know, people might say, that’s a maternal instinct to listen to your children, right? No, a recording device can listen to you.

Aashka Patel (27:27)

Hmm. Hmm.

Hmm.

Yeah.

Hmm. Hmm.

Yeah.

Yeah.

Ryan Carrier (27:54)

Okay, so

we need to break out tasks which are very digitizable, they can be parameterized. You can establish metrics, measurements and thresholds. Those are huge words for us, right? Establish metrics, measurements and thresholds around these qualities. And then can you measure them? Absolutely, we can measure many of them. We can’t measure hope. We can’t measure

Aashka Patel (28:08)

Yeah.

Yeah.

Ryan Carrier (28:24)

dreams. We can’t measure love in the true sense of the word love, not I love you. I brought you flowers. I can measure whether I brought you flowers, right? That is a task. That is an act. Okay. But I can’t measure love generically speaking. Right? So getting to maternal instincts, the way Geoffrey Hinton has suggested, you know, no, we can’t get all the way there, but we can certainly get to

Aashka Patel (28:26)

Hmm.

Hmm.

Yeah. Yeah.

Hmm. Hmm.

Ryan Carrier (28:54)

many tasks. Here’s the problem. Humans operate suboptimally many times. So if I’m a tool, okay, I want to optimize on for my benefit. Well, if I choose to take time out of my benefit to care for you, Aashka, that’s a suboptimal choice given the parameterization of me, right? I’m not optimizing for myself.

Aashka Patel (28:57)

Hmm.

Hmm. Hmm. Hmm.

Hmm. Hmm. Hmm.

Ryan Carrier (29:22)

I may be gaining benefit by caring for you. You know, it could be love. It could be warm feelings. I could feel good about myself. I could grow my skills in caring for you, right? But the point is, is it’s a suboptimal choice given a set of parameters that are maximizing my personal optimization. And so we are so far away from being able to do that with our tools that these notions are ludicrous to me.

Aashka Patel (29:25)

Hmm.

Mmm.

Hmm.

Hmm.

Mm.

Ryan Carrier (29:52)

Now,

Aashka Patel (29:51)

Hmm.

Ryan Carrier (29:52)

does that mean in 20, 30, 50 years, we might not be closer to that? Absolutely, we could be. But in today’s place, what we need to do is maintain controllability, intervenability, transparency, explainability, and we need to ensure human oversight.

Aashka Patel (29:59)

Mmm.

Hmm.

Yeah.

Hmm. Hmm. Hmm.

Ryan Carrier (30:16)

and human beneficial ownership. There are people out there who are trying to establish shell companies where autonomous systems can sit in the shell company. Again, it makes me sick to my stomach. Why? Because it’s just a way to avoid liability of humans. And that is dangerous to the nth degree. So these dreams of the future are just that. We have a lot more work to do at the very basic level of robots.

Aashka Patel (30:22)

Mm.

Yes.

Hmm.

Hmm. Hmm.

Basic level,

yeah.

Ryan Carrier (30:44)

governance, oversight, and accountability

of these tools before we can even take much bigger steps such as that.

Aashka Patel (30:53)

Yeah, makes sense, makes sense. Jimmy, what are your thoughts on this?

Jimmy Farrell (30:58)

Sure, yeah, I agree with Ryan’s sort of characterization of the difficulties with aligning AI and even understanding what human preferences would be because those are completely flawed as well. Maybe if we were to take the thought experiment that this was possible, I think the way I would take this conversation is that, even if this was possible and somehow AI had some maternal instincts for us as humans,

Aashka Patel (31:04)

Thank you.

Mm-hmm.

Hmm.

Jimmy Farrell (31:27)

We don’t have the tools to measure this. So we basically evaluate models right now as black boxes. So we basically ask it, are you safe? Can you produce this harmful content? And we see what it’s said. So in that regard, tools like the Code of Practice is a start because the EU has kind of drawn a line in the sand that external evaluations will be mandatory for certain.

Aashka Patel (31:29)

Yes.

Yeah.

Jimmy Farrell (31:55)

models and there’s also some provisions in there to basically have those evaluations be deeper than black box access in the future when such techniques start to come to fruition. There’s multiple techniques, famously one called mechanistic interpretability has now suffered from potentially being a bit more of a dead end than was thought about a year ago where you can sort of look out of the hood and do

Aashka Patel (31:56)

Yeah.

Yeah.

Jimmy Farrell (32:22)

what is akin to like a brain scan of different areas of a deep neural network. And then in terms of, so I guess from the regulatory angle, yeah, these like deeper than black box external evaluations could be a way to test for such tendencies of a model. But then of course the EU is doing all sorts of things, not on the regulatory side, but on the promoting AI research side and investing in AI and that sort of thing. So.

Aashka Patel (32:26)

Yeah, yeah.

Hmm.

Jimmy Farrell (32:52)

such tests, for example, on mechanistic interpretability can be quite compute intensive and researchers don’t have access to this compute. So, building public compute clusters, which they are doing, these so-called gigafactories, making sure that enough is allocated to researchers and academic institutions, universities that are doing this kind of research.

Aashka Patel (33:02)

Yeah.

Jimmy Farrell (33:17)

That would also be something very important to do but to be clear as Ryan said This is also important to happen to risks that we do see definitely in models now like dangerous capabilities like for example tendencies for AIs to be overly emotionally addictive and these sorts of things

Aashka Patel (33:41)

Yeah, the AI psychosis problem and everything is happening. and to be able to reach the maternal instincts stage, we first need to solve the problem of AI sycophancy because I don’t see any mothers being sycophantic to their children. we need to solve the problem of AI sycophancy first and then maybe talk about the maternal instincts.

So Ryan, I read on your LinkedIn. for our viewers in India, especially leaders looking to comply with the DPDPA Act or the Digital Personal Data Protection Act, or anyone curious to learn more, I know For Humanity has been working on something big in this space. You have some session or course coming up.

So can you share what’s launching and how professionals can get involved?

Ryan Carrier (34:31)

Yeah, I’m going to build a little bit of a story here. So we are working with the European Data Protection Board. We are at the last vote of the process to have the world’s first GDPR certification for AI algorithmic and autonomous systems under Article 42. That will be a big deal when that finishes. We’ve taken that process and we’ve replicated it in Bermuda, in California, in Dubai.

Aashka Patel (34:34)

Yes.

Mm.

Hmm. Hmm.

Ryan Carrier (35:01)

and now

Aashka Patel (35:01)

Hmm.

Ryan Carrier (35:01)

in India as well. What we do is we take the local law and we’re sensitive to local law, local terminology. So for example, we don’t deal with controllers and processors. In India, we deal with data fiduciaries and significant data fiduciaries, right? So we adjust our certification scheme to adapt to the local law. At the same time, we are trying to harmonize as much as possible.

Aashka Patel (35:03)

Hmm.

Yes.

Hmm.

Ryan Carrier (35:29)

so that if someone in India is abiding by DPDPA, ideally

Aashka Patel (35:29)

Hmm. Hmm. Hmm.

Ryan Carrier (35:33)

that gets them 70, 80, 90 % of the way to abiding by GDPR and California. Why? Because all for humanity cares about as a nonprofit public charity is facilitating compliance with the law. Why? Because all we care about is mitigating risks to humans, full stop. It’s the only thing we care about. So by creating these audit rules,

Aashka Patel (35:39)

Hmm.

Hmm.

Hmm. Hmm.

Ryan Carrier (35:56)

that are globally harmonized, but jurisdictionally sensitive, we are able to meet the local needs as these laws are produced. So as we’re drafting that law, we spent the last two years doing it, we finished up that work, and now starting on, I think it’s October 27th, we launch our course, Training People, How to Become a For Humanity Certified Auditor on Those Audit Rules Applicable to the DPDPA. And we do that

Aashka Patel (36:14)

Mm-hmm. Mm. Mm.

Okay.

Ryan Carrier (36:26)

with the EU AI Act, we do that with GDPR, we do that with California privacy, children’s code, just on and on and on for our more than 50 plus certification schemes. We have more than 7,000 risk controls, treatments and mitigations for AI algorithmic and autonomous systems. That’s more than any organization in the world by a long way. And all we’re trying to do is to take all of that information and give it to people like Jimmy and anybody else who wants

Aashka Patel (36:32)

Hmm. Hmm.

then.

Hmm.

Hmm. Okay.

Hmm.

Ryan Carrier (36:55)

use it to facilitate compliance for companies with laws,

Aashka Patel (36:57)

Hmm. Hmm. Hmm.

Ryan Carrier (37:00)

regulations, guidelines, standards, and best practices.

Aashka Patel (37:04)

Got it, got it. So, like, is it free of cost

Ryan Carrier (37:09)

all of our courses are online and they are all free for the learning. So if you want to learn, you simply register at For Humanity Dash University on our website. You create a student account and you can go through all the learnings for free. When you finish that, if you’ve completed all of the quizzes that test your knowledge, then you are eligible to sit the certification exams. Certification exams have a cost.

Aashka Patel (37:12)

Okay. Okay. Great. Great.

Mm-hmm.

Mmm. Mmm. Love it. Mmm.

Mm-hmm.

Got it.

Ryan Carrier (37:37)

As a charity, what we want to do is we want to teach as many people as possible. But if you need to be certified, if you need to prove and demonstrate your expertise, then we ask you to pay a fee to sit the certification exam. And for all four humanities certified auditors, we hold you to a code of ethics and professional conduct. So we’re elevating your expertise associated with this knowledge.

Aashka Patel (37:37)

Got it.

as possible.

Got it, got it. Yeah, that’s a great initiative and I’ll make sure that I link up the registration link onto this video when it releases. So most people can take advantage of this. yeah, thank you so much for sharing this.

The Australian AI stress test that you were part of. So it was a collaboration with 64 experts across AI cybersecurity and public policy to assess five AI threats against Australia’s current legal frameworks.

So the findings show that while existing regulators can and should take specific steps to manage AI risks within their domains, significant gaps remain for the national scale risks posed by general purpose AI. So since you were a part of it, can you double click on those gaps specifically?

Jimmy Farrell (38:54)

Yeah, sure. So this is one of my nationalities. that’s the reason why I was involved. I have more expertise on the European side. It was good to dip my toes in the Australian context. Australia has quite a good history of fairly progressive tech laws in general. There’s a great privacy watchdog. They’ve also pioneered some child safety protections. For example, the

Aashka Patel (39:07)

Uh-huh.

Yeah.

Jimmy Farrell (39:22)

They’ve been one of the first to talk about social media bans.

However,

the stress tests really showed that their lack of a comprehensive AI Regulation that the EU now has a lot of other jurisdictions have. The US is starting to have more and more not a comprehensive one, but state-based specific legislations. It identified that Australia really lacks this. And in that sense, it was a very important exercise to undertake and it will certainly

Aashka Patel (39:32)

Mm-hmm.

Yeah.

Mmm.

Jimmy Farrell (39:52)

be a part of ongoing discussions in Australia. In terms of should other countries do the same? Definitely. It’s something Europe has done since well before CHAT GPT, before the AI Act, they’ve been consulting stakeholders since the mid 2010s. And this has clearly resulted in a really democratic German result that is very

Aashka Patel (40:05)

Hmm. Yeah.

Jimmy Farrell (40:22)

foundational in the world as a benchmark for AI safety ethics and protecting fundamental rights.

Aashka Patel (40:31)

Yeah,

yeah. So has any other country done similar kind of stress test or like Australians are the first one to get it done?

Jimmy Farrell (40:40)

To name it a stress test, I couldn’t tell you if anyone else has done what’s called a stress test, but certainly the EU has done lots of research into how this combines with other EU type policy like the GDPR, like the Digital Services Act, Digital Markets Act. There’s still quite some legal research to do to find where these overlaps lie. And this is part of the simplification strategy of the EU moving into their competitiveness

Aashka Patel (40:56)

Hmm.

Yeah.

Jimmy Farrell (41:09)

intentions over the next few years to basically reduce double reporting and that sort of thing which is a no-brainer.

Aashka Patel (41:11)

Mm-hmm.

Yeah, yeah, yeah, makes sense. since we have been talking a lot about the liability thing, so the AI liability directive that was supposed to complement the EU AI Act got withdrawn from the commission’s 2025 work program, right? And MEP Excel was called it a strategic mistake and blamed industry lobbying pressure. So do you agree with him? What are your thoughts on that?

Jimmy Farrell (41:42)

Yes, I would mostly agree with him. think the liability piece is a really important part of the puzzle because the AI act, it’s really like before the facts. you know, are you transparent with your documentation? Are you compliant with copyright law? Do you have safety and security mitigations in place? Do you have serious incident reporting in place? But

Aashka Patel (41:46)

Mm-hmm.

Jimmy Farrell (42:08)

After something happens, actual cost, the liability, it remains to be addressed. One of the justifications of withdrawing it is that certain interpretations of the product liability directive could cover this, but most stakeholders would say that this is not the case and that AI is specific enough to warrant its own. It’s also something that has been heavily looked at in the US, so a lot of California frontier.

Aashka Patel (42:24)

Mm, yeah.

Mmm.

Jimmy Farrell (42:37)

legislations that were in consideration involved liability. Of course, it’s something that the tech industry and especially big tech industry pushes back on strongly. In general, it’s also quite relevant to the EU because the EU doesn’t have giant tech companies and their AI economy ecosystem is likely to be based off more downstream applications and sort of development ecosystem.

Aashka Patel (42:50)

Hmm.

Mm.

Mmm.

Hmm.

Jimmy Farrell (43:07)

And this liability is really important. Liability upstream. So basically if something happens involving a model deployed by a downstream developer, the liability doesn’t fall entirely on them. So this is also something that EU should really prioritize because it fits in with the whole grant strategy on competitiveness and the fact that they have a unique economy which is really bolstered by SMEs and that’s particularly the case

Aashka Patel (43:26)

Hmm.

Hmm.

Jimmy Farrell (43:36)

and

will be the case in

Aashka Patel (43:39)

Yeah, yeah, makes sense, makes sense. So what severe or strange AI incidents have you encountered recently?

Jimmy Farrell (43:48)

So yeah, this is something that has been developing over the last few months. There’s been numerous suicides linked to AI model. Of course, those links are not necessarily causal and it’s still under discussion how much influence AI models had. But there was a case in Belgium. There have been multiple cases in the US.

Aashka Patel (43:59)

Hmm. Yeah.

and

Hmm.

Jimmy Farrell (44:16)

And one is recently, the New York Times reported recently on one in which the chatbot specifically kind of coached the individual on how to carry it out and how to, you know, whether to leave a note, that sort of thing. And no triggers went off to inform the AI company involved or authorities. And this was potentially a preventable incident. Apart from these suicides, there’s also been

Aashka Patel (44:16)

Yes, teen suicide especially, right? Yeah.

Yeah.

Jimmy Farrell (44:46)

Yeah, like indications of psychosis being induced by AI. There was one where an individual was killed due to psychosis that was potentially induced by AI. Other than harm to individuals, you’ve also had financial damage from deep fake scams.

Aashka Patel (44:52)

Yes.

Mm-hmm.

Yeah.

Jimmy Farrell (45:10)

an a company executive come on a Zoom call and have an employee transfer huge sums of money to that account. And also been the first incidents that are potentially related to AI agents. So this is something that seems like quite new, but there was a particular example where an agent kind of hallucinated a task it was given and basically deleted a

Aashka Patel (45:12)

Mm-hmm. Yeah.

Mm-hmm.

Yes.

Hmm.

Jimmy Farrell (45:38)

a large chunk of code and this could be extremely financially damaging to a company

Aashka Patel (45:41)

Mmm.

Yeah.

Yes, yes, yes. And also there have been reports of unauthorized purchases made by the AI agents like OpenAI operator

So for our viewers listening in, if they witness or experience harms from AI systems or these kind of incidents, like what’s the one thing you would want them to do

Jimmy Farrell (45:57)

Yep.

So I would say, yeah, in general, would caveat this by saying that my call would be much stronger towards governments because I think this is their responsibility rather than individuals at this moment. But if it was to be for individuals, I would really encourage them to get the message out and to basically report it to their authorities and also contact the company involved.

Aashka Patel (46:18)

Yeah

See you soon.

Mm.

Thank you.

Jimmy Farrell (46:36)

to

find more information. For example, certainly you always have something called the right to an explanation. So this could be something that could be invoked in the case of an incident. There’s also some incident databases which would benefit from information on incidents from the victims. So the OECD has a really large incident database. there’s a grassroots one called the AI Incidents Database.

Aashka Patel (46:46)

Mm-hmm.

Hmm.

Yes.

Jimmy Farrell (47:06)

And yeah, then it would really be on the responsibility of governments not only to put in better, you know, AI safety legislation or incident reporting, but preventative measures like AI literacy, education programs to make sure that individuals are aware of risks related to AI, whether it be personal relationships or business practices like cyber security breaches.

Aashka Patel (47:23)

Yeah.

Mm hmm.

Jimmy Farrell (47:36)

this was.

Aashka Patel (47:36)

Yeah,

yeah, yeah, yeah, make sense. And that’s why organizations like ours exist for to promote AI risk literacy instead of just AI literacy. So thank you so much for your time. Yeah, edutainment Yeah. Yeah, you remember that.

Jimmy Farrell (47:55)

Yeah. Yeah.

Aashka Patel (47:57)

Thank you so much, Jimmy, for your time. Let me stop the recording.

His answer?

Watch on YouTube; listen on Apple Podcasts or Spotify.

Episode Summary:

While everyone obsesses over AI chips and compute power, the REAL bottleneck in the AGI race is energy. China knows this. America is just waking up. In this deep-dive conversation, Casey Handmer, Founder of Terraform Industries and former NASA JPL engineer, reveals why the US-China AI competition will be won or lost on energy infrastructure, not algorithms. We explore the $500 billion Stargate project, the future of synthetic fuels produced from sunlight and air, and why India could become the third AI superpower if it plays its cards right.

Why This Matters:
The AGI race isn't about who has the best models; it's about who can power them. This conversation cuts through the hype to reveal the real infrastructure challenges facing artificial general intelligence development.

About Casey Handmer:

Casey is revolutionizing energy production through Terraform Industries, which creates synthetic fuel using only sunlight, air, and innovative engineering. His unique perspective bridges space exploration, AI infrastructure, and the geopolitics of AGI development.

Timestamps:

00:00 AIR Bites (Precap)
01:40 The Real Bottleneck in the AI Race Isn’t Compute
03:10 Inside Stargate: The $500B Project Powering AGI
04:29 Terraform’s Photosynthesis Machine: Fuel from Sunlight and Air
07:02 Water Myths in Energy: Why Desalination Isn’t the Problem
09:36 Can India Become the Third AI Superpower?
13:29 From Mars to Mainframes: How Terraform Fuels Both
13:46 The Future of Energy: Solar Power and Regulations
15:31 Solar vs Synthetic Fuels: What Data Centers Really Need
17:49 Why Solar Regulation Is Broken (and How to Fix It)
20:07 Energy = Wealth: The Harsh Truth About Global Power
22:53 If AI Mobilized a Manhattan Project, How Fast Could We Scale?
24:33 Do We Have Enough Land for 100% Solar Earth?
25:57 Japan’s Energy Scarcity and Solar Redemption
29:08 Why Space-Based Solar Power Makes No Sense
29:48 Batteries Are Killing Transmission Lines
31:05 Supply Elasticity Walls in Energy Infrastructure
34:15 Data Centers Will Need Their Own Power Plants
36:38 Will Quantum Computing Save Us From the Energy Crisis?
39:08 My 2027 Prediction: The AI Grid Will Flip
44:10 How to Get Rich in Ideas: Advice for Young Physicists
50:18 Why Casey Says Yes to Podcasts
51:28 Outro

Transcript:

Aashka Patel (00:04)

hello and welcome to On AIR with Aashka thank you so much for joining us, Casey. Let’s dive right into the questions.

every AI lab is racing to build AGI. But here’s what nobody is discussing. Mark Zuckerberg says the real bottleneck isn’t in compute, it’s energy. Hyperscalers are planning 5 gigawatt or more data centers, and the US grid can’t deliver it. Meanwhile, China manufactures 80 % of world solar panels and refines 70 % of global lithium.

required for the batteries. So if abundant energy is the foundation of the AI race, why doesn’t China just win by default?

Casey Handmer (00:41)

Yeah, I’m optimistic. think that obviously many, different components are required for AGI, but the bottleneck at this point does not appear to be solar module supply, for example. You could make some comments about US regulatory constraints on solar deployment or on GPU availability or software development. I think particularly in terms of AGI development,

most of the available headroom is just in software.

Aashka Patel (01:07)

Okay, so putting the export controls on chips and everything can still make US win this race?

Casey Handmer (01:13)

I mean, there are some export controls on the various chips. don’t think that’s going to be, it’s going to result in a durable advantage. I think it’ll help in the short term. But I think ultimately the data centers are being primarily built in the United States for a reason.

Aashka Patel (01:30)

Okay, So any recent updates on the Stargate project that you have? Like what’s going on there?

Casey Handmer (01:36)

Yeah, I mean, I don’t know very much about it. It is interesting to me that essentially private capital in the United States, is spending something like 10 Manhattan projects worth of wealth on this over the next few years. Yeah, so the entire Manhattan project was about $30 billion. And that required, that was to build a nuclear bomb.

Aashka Patel (01:48)

Half a trillion almost,

Casey Handmer (02:01)

for the first time, but it also required the construction from scratch of like what are now two or three or four major national laboratories, the invention of entire branches of physics and machining and a bunch of other stuff like that had to be invented from scratch. The scale was mind boggling. About half a million people worked on the Manhattan Project in total. Although I think the greatest number is about 120,000 at any one time. So just a really enormous project done obviously in the 1940s in the condition of wartime. And then this compute scale up.

It’s a little bit different in the sense that it’s...

I feel like there’s a lot less diversity in terms of the raw machinery that’s required. It’s like racks plus power plus racks plus power. yeah, obviously there’s tremendous resources being poured into this. And I think justifiably.

Aashka Patel (02:49)

yeah, yeah, makes sense. Let’s come back to the Stargate conversation once we have the Terraform story’s built. So, Terraform Industries makes cheap synthetic fuel from sunlight and air. For our listeners, can you briefly explain the process that you quote as photosynthesis on an industrial scale that produces synthetic natural gas?

Casey Handmer (03:11)

Yeah, sure. mean, plants use sunlight and air to grow their bodies. The wood in a tree came out of the air. It didn’t come out of the ground. essentially, the way humanity gets its energy is by burning fuels in air. And wood is quite combustible. And so essentially, inspired by that process, many people over the centuries have tried to make a synthetic version of the same process, making

perhaps more useful fuels than wood like natural gas or gasoline or kerosene or jet fuel or whatever. The challenge actually for more than 100 years has not been technical so much as economic. That is, there are many ways of getting these chemicals in the easiest way currently known is to drill a hole in the ground. But at the same time, solar is getting cheaper and cheaper over time. And so we’ve got to push that forward.

Yeah, so essentially what Terraform does is we have a machine that intermediates intermittent and free natural resources like air and sunlight and water and then this kind of off-tape market that requires predictable volumes of predictably high quality and enormous volumes of natural gas or fuels or other materials that then form the basis of our entire supply chain.

So it’s hard to think about this, when you look at a finished product, pretty much everything began as rocks, you know, at some point. So yeah, that’s what we’re working on. In terms of how it works, there are about 400 different ways of doing this. We had to pick one. The approach that we took was to build an electrolyzer to make hydrogen out of water, a direct air capture system to take CO2 out of the air.

It’s a bit like a leaf on a tree. And

then a chemical reactor that combines CO2 and hydrogen to make methane.

Aashka Patel (04:50)

Yeah, got it, So currently it’s just methane or like any other plans to expand the natural gas production.

Casey Handmer (05:00)

Well, methane is natural gas. having figured out how to do this, it now seems that there’s a bunch of adjacent step-up opportunities in other areas that are also in primary materials, also very energy intensive, also have a climate kind of aspect to them, also have a national sovereignty aspect to them that we are very well positioned to come and address now. So that’s kind of the step two for us.

Aashka Patel (05:09)

Mm-hmm.

Got it, got it. So you mentioned the use of water in the terraformer, the machine that you are using to make this cheap synthetic fuel. So is that fresh water or do you purify salt water and then use it? How does that work? Because that can be a limiting resource, right? OK, OK. OK.

Casey Handmer (05:40)

Yeah, we can actually get it out of the air. Well, not, no, not really, not really. So if you,

think about how much water you need to grow an acre of crops to eat, or to grow an acre of biomass to burn versus the amount of water you need to synthesize the equivalent amount of fuel, it’s like, it’s not even in the same ballpark. And then water is actually, it’s not that scarce. There are some places that it...

Aashka Patel (05:49)

Mm-hmm.

Mm-hmm.

The fresh water would be here, right?

Casey Handmer (06:08)

Well, yes

and no. If we live in a world where we can synthesize synthetic fuel, then we live in a world where we can desalinate essentially arbitrary volume supporter. And it is, it’s very bizarre to me to acknowledge that we live in a world where for 10 years now, you know, lot of the,

the environmentalist discussion has been about like, you know, the putative water scarcity of data centers or farming alfalfa or something. When almost all the places that meaningful numbers of humans live are very close to the coast, the coast is defined by the presence of an ocean that is literally made of water. And the energy required to separate the salt from that water is absolutely trivial in comparison to the energy required to synthesize fuel. So just to give you a concrete example, in...

In one day, one terraformer makes the equivalent of about one barrel of oil, which is about 150 liters of fuel. But if we were condensing water out of the air, then it would be about 150 cubic meters of water. And if we are desalinating it using standard desalinization technology, it’s about 100, I’m going to mess this up. It’s 150,000, so it’s 100,000 times more than 150 liters, which is 10,000 times 1.5.

which is 15,000 cubic meters. So volume for volume, you’re making 100,000 times more liquid water than liquid fuel. So any place that’s like, no, water’s has to be a place that’s like either Mongolia, so like impossibly far from the ocean, or Somalia, so like really bad governance, no functioning law and order, or a place where they just decide that they’d rather have their hair shirt than just go and build a desalinization system.

that can produce essentially infinite water. Israel’s been doing this for 50 years. So it’s not some mystery how this works. Most of the Middle East depends on desalinization, actually.

Aashka Patel (07:55)

Hmm, got it. Makes sense, makes sense. So, since we were talking about solar and you have been bullish on solar, right, like infinite amount of energy source that we have. India has world’s third largest solar capacity, scaling without regulatory paralysis choking the US and Europe. And India has abundant sunlight and a massive STEM workforce.

So if India wanted to become the third superpower in the AI energy race, what’s one critical move it must make now? Like more solar panel manufacturing or something else?

Casey Handmer (08:32)

Well, that’s a good question, isn’t it? I feel like India’s industrial policy has always been around the promotion of national champions and the preservation of autarkic manufacturing capacity in the purpose of national sovereignty. So India is justifiably proud of its ability to produce a lot of the technology that it needs internally. And there’s no reason it shouldn’t. has an enormous population and many, many brilliant people and highly educated. And yet at the same time,

We all know that India has a number of reasonably unique peculiarities as far as its economy go as well. But I think, you know, I’m not the first person to say this, but it is certainly the case that in the coming decades, as in the past, when there are the countries that have nuclear weapons, they have absolute sovereignty and the ones that don’t, or say that the companies that have nuclear submarines equipped with nuclear weapons on rockets, for example, are kind of the...

the elephants in the room and the rest don’t. There’ll be the countries that have their own AGI under their own legal control and the ones that have to use the foreign AGI. And we’ve already seen what a horrific degradation of European national sovereignty or like, you know, almost regional sovereignty has occurred as a result of their inability to produce web technology that can compete with the United States. And...

in general, a handful of exceptions, but in general. And just what an appallingly low wattage response to that was the formation of a bunch of like UK quangos that run around like essentially engaged in piracy, trying to shake down the US tech giants for money. When like every dollar paid in fines or disputed or whatever is a dollar that’s not being spent.

in the UK or in the EU building a meaningful competitor to Google. It’s not that hard. You could literally vibe code it today. And the same thing will happen in the future, which is like, OK, well, it’s now impossible to run modern civilization without computers. It pretty soon will be impossible to run a modern civilization without AGI’s basically intermediating every single step. Are those AGI’s under the control of your geopolitical adversary? Are they able to turn them off? Are they able to confuse your political systems?

through manipulation of information and fundamental systems. you know, I think a lot of countries, France and the UK and the United States and China, obviously included, have realized that, you know, if their ancestors made the effort to put nuclear missiles on submarines 50 years ago, now is the time to put an AGI within their own borders. The problem is, this is a race that is defined by your ability to

to attract the very best in both liquid capital and human capital. And when it comes to this, the United States is just richer, right? And it just has most of the really bright AI engineers, not all of them, but a lot of them. And that helps. same thing in second world war. Germany, first step, step one in conquering the world was to convince all the really bright people to immigrate to United States. How did that go for them? Not so well. For India’s perspective,

My understanding is that India’s indigenous ability to fabricate chips is not sufficiently competitive. But that aside, I think all the other pieces could exist within India’s borders. And if they want to have absolute digital sovereignty in the future, they’d need to get their act together on chips pretty quickly. And it’s not that hard in the grand scheme of things.

Aashka Patel (11:32)

Not comparable,

Yeah, yeah, makes

sense. So you have connected this Terraformer technology to Mars industrialization and this Stargate project.

Like what’s the actual connection between synthetic fuels and both Mars and next-generation AI infrastructure?

Casey Handmer (12:06)

Yeah, I mean, I think I don’t think I’ve ever written about Stargate, but I’ve written a lot about Starship. So star various things. Okay. Mars industrialization. So just to clarify this for your listeners, Stargate is a large AI data center project that’s being constructed in Texas. And Starship is a large rocket launch project being constructed in Texas. Mars industrialization.

Aashka Patel (12:10)

Yeah, Starships. Yeah.

Casey Handmer (12:29)

This is just kind of a hobby project, yeah, just trying to... Sorry?

Aashka Patel (12:29)

Elon is, yeah, you are a fan of Elon,

You are a fan of Elon, right? Like in a lot of, yeah, yeah, yeah, yeah.

Casey Handmer (12:38)

Yeah, I think it’s very difficult to operate in this area and not be.

I think it’s one of those things where to the kind of average disinterested participant or whatever, what SpaceX does is quite extraordinary. But if you actually have any insight into just how hard it is and just how extraordinary what they do is, you’re like, it’s an anomaly in the matrix. It’s very strange. It’s very, it’s as close as we’ll ever see in our lifetimes to something that feels magical.

Right? In a tangible way.

Mars industrialization. We, one day humans will live on Mars, maybe in our lifetimes. When they do, they will need to ultimately develop supply chain on the surface of Mars to make things. And that means they have to convert rocks into stuff. So metals, fuels, oxygen, gases, water, you name it. The good news is Mars is covered in rocks. plenty of rocks.

The bad news is that there is no developed existing supply chain for any of it at all whatsoever. So you have to do it from scratch. this was the thought experiment that I initially used to try and understand how this was done. And then that ultimately led me to the insight that it would possible to do it on Earth as well, and very difficult but possible. So that’s what we’re working on here.

Aashka Patel (13:52)

Got it, got it. So, would you rather sell the Terraformer to data centers or to traditional energy companies? I know that as per a TechCrunch article, you have been selling to traditional energy companies

Casey Handmer (14:05)

Yeah.

So humanity uses a number of different energy sources, including various forms of electricity, various forms of chemical energy, including oil and gas and food, obviously, and coal. And then we use that for different purposes. So sometimes we use electricity directly. Sometimes we use electricity to make heat. Sometimes we just use the chemical forms of energy to make heat or some other kind of end product. Generally speaking, I think that if your end use is electricity, you’re better off

starting with electricity and staying as electricity. So that avoids the interconversion penalties. So you can start with solar power, charge a battery, and then run a data center. Problem solved. Okay.

So I don’t think it’s very necessary to put a terraformer and making synthetic natural gas in as part of this data center, presupposing that you have the gas infrastructure to consume that gas anyway and storage and so on. It’s just not really worth it. So instead, you say, well, we make natural gas and we pipe it into the existing distribution network, which exists everywhere because gas comes from all over the world. It’s found in all kinds of random places. It’s used in all kinds of places.

And sure, some of that gas gets taken out of the network and gets burned to make electricity in gas turbine power plants. But you and I both know that in 20 years time, those are going to be pretty much out of business. So, you know, that’s kind of a growing industry feeding a dying industry. The growth part of natural gas consumption is in other parts of the economy, I think. Yeah, so I think the Terraformer Mark 1 is for making synthetic natural gas for, you know,

very boring traditional utility distribution, everyday uses, heating, cooling, cooking, chemical processes. And I think the greatest use of, the greatest source of growth for natural gas consumption will be aviation, actually.

Aashka Patel (15:38)

Every day.

I saw your video with like the castle one that you have created recently, hard reset or something. The third use case of the synthetic fuels was into this jets and everything. Yeah, makes sense.

Casey Handmer (16:01)

Yeah, yeah.

Aashka Patel (16:09)

So coming back to the Stargate project, my favorite. first of all, for the audience, the Texas site of the Stargate project. It’s just one of the sites. Alone will consume 1.2 gigawatts of electricity, same amount of power required to run a million American homes. And the budget is almost half a trillion dollars.

And you have said that US environmental regulations are actively killing solar deployment. So, what’s the one regulation you would like to kill first so that like this energy consumption and everything can be done on solar?

Casey Handmer (16:45)

Yeah, I mean, this is less of a problem in Texas, but some states and the federal, and actually a lot of this has been changed just in the last year. So like part of the Department of Government Efficiency work in the United States, basically revised 50 years of regulations that had been added to an existing law that had made that law work a lot less well, particularly for these projects. But the fundamental error I think is made here is that

the environmental protection regulations that were put in place in about 1970. And for good reason. They presuppose that anything you want to do to the land is bad for that land in terms of industrial development. When actually I think there’s an argument to be made that solar is like actually net positive, if not neutral. But it still results, the net outcome is the same, which is that it takes years to permit it because you have to go through the same process as if you were just building an oil refinery or something which definitely does have like long-term negative impacts on land.

My proposal would be what’s called a categorical exclusion. And there are categorical exclusions for other types of technology, actually ironically including oil and gas production because it’s so important to our economy, that you should be basically exempted from having to go through the environmental impact analysis process. The problem is that pretty much all the existing players benefit quite a lot from that process because it’s so expensive that they all make good money out of it. So this is...

Aashka Patel (17:46)

Yeah.

Casey Handmer (18:01)

this kind of perverse incentive structure set up around it. Anyway, I think there are many good think pieces out there on what we could do about that. But yeah, it’s absolutely crazy to me that the United States would do anything policy-wise or any country for that matter to inhibit the deployment of solar panels because they’re so much cheaper than any other form of energy and so much less environmentally impactful.

Aashka Patel (18:24)

Yes.

Casey Handmer (18:26)

And

the key to your children being rich is to make sure they have cheap energy. There’s no such thing as a rich, low energy country. There’s no such thing. Literally, like if you have energy abundance, like heaps and heaps of oil or something, or historically like Egypt, like I’m talking the ancient world, Egypt had energy abundance because they had the Nile River and they had predictable grain harvest every year. were like, know, net surplus grain producers for thousands and thousands and thousands of years.

then you can basically do anything governance-wise and be fabulously rich. And if you have energy poverty, even with the best case governance, you will still be pretty poor. So just to kind of give you a concrete example there, Japan has energy poverty. They have to import a lot. They’re well governed, I think. Very advanced industrial nation.

Aashka Patel (19:12)

Yeah,

Hmm.

Casey Handmer (19:20)

And the GDP has been stagnant for 30 something years. It’s roughly equivalent to what the US was in like 1970. So, so they’re literally two generations of human development poorer than the United States because they don’t have energy. And the United States is, I wouldn’t say poorly governed so much as barely governed. And, and Texas is one of the richest, richest places on earth because of oil. Shame about the climate. Okay.

Aashka Patel (19:45)

Makes sense.

Hehehehe

Casey Handmer (19:49)

The good thing about solar is that everyone has solar. Unless you’re Iceland, everyone has solar. So, particularly... I’m sorry?

Aashka Patel (19:53)

Yeah, intermittent but yeah, they do have.

Intermittently available, like some days available, some days not, but like of course every place on earth has solar

Casey Handmer (20:05)

Yeah, intermittency is a bit of a blown, think. Sure, with wind power, sometimes it just won’t blow for a long time, but the Earth is not going to stop turning. So every day the sun is going to come up. And some days it’s cloudy and some days it’s not, but even when it’s cloudy, you can still read a book. So there’s still a lot of light bouncing around. Maybe not as much as on a sunny day. In winter, the days are shorter. In some of the days, longer. But most humans live pretty close to the equator, where that doesn’t make a huge difference. so the response to this, the way you fix this problem is you

you just build more solar. And then in summer, your solar is fabulously cheap and in winter it’s a bit more expensive. And that’s how you, you meter the, kind of meter the offset there. But like this idea that intermittency is not the same as unreliable, right? We have good weather prediction. We know exactly how much solar we’re going to get in three days. So, it’s very easy for us to say, well, we’ll charge up the batteries now or not. And this can be all done autonomously. Tesla has had a version of this called auto bidder for like 10 years.

Aashka Patel (21:01)

Yes, the value, yeah.

Casey Handmer (21:02)

basically manages its batteries to maximize value production. Yeah,

and then batteries are just like monotonically increasing, right? You just keep on building and connecting, building and connecting, and it just makes the grid better and better.

Aashka Patel (21:13)

Yeah. Yes.

So, I recently watched Bill Gates documentary on Netflix claiming the current direct air capture or like carbon capture can only handle historical emissions, not the ongoing ones that we are emitting right now. So, with Terraform’s technology, if we mobilize like the Manhattan Project, what’s the fastest timeline to industrial scale?

Casey Handmer (21:39)

Yeah, it’s a good question. I haven’t seen that documentary and I don’t understand exactly what Bill’s getting at there. If you mobilized like Manhattan Project, it’d like 10 years. There’s a lot of work to be done. But the key thing to realize is that the whole point of this exercise is you shouldn’t require massive government subsidies to get to scale. You should be able to show positive unit economics at relatively modest scale and then...

Aashka Patel (21:51)

10 years.

Casey Handmer (22:06)

that itself directs the massive capital flow in your direction. And then that is what gets you massive scale because at end of the day, the government can spend a lot of money on stuff, but it doesn’t necessarily, like the only thing that will really move the needle sustainably is positive unit economics. It’s making money, generating value. Can AI help us achieve that goal faster? Yes. It’s very helpful for research, but I think actually be more helpful long-term in creating a lot more prosperity, creating a lot more.

Because AI can help us with research, can help us with coordination, it can help us operate more efficiently. But AI in the economy in general creates more wealth and more demand for services like ours. it’s obviously a series of massive waves of economic growth kind of converging and all we have to do is keep our head above water and we’ll be fine.

Aashka Patel (22:53)

Yeah, yeah, yeah, makes sense. So,

Okay, so you have had a white paper, right, in that you quoted that

the solar will require 2 billion acres of land. So, do you think 2 billion acres of land is available or possible to have solar panels on like currently?

Casey Handmer (23:13)

Yeah.

Aashka Patel (23:14)

like how can you explain the stats basically

Casey Handmer (23:19)

Yeah, so in terms of land use, something like 1 % of land on earth is heavily populated by cities, 99 % is not. About 20%, depending on the continent, about 20 % is engaged in agriculture and farming, and maybe another 20 % in forestry, and then the balance is like deserts and mountains and swamps and things. And the solar panels don’t have to go on the...

on the prime agricultural land, but the total amount of solar that you need to power every man, woman and child on earth with US levels of energy wealth is like maybe one or 2 % of the total land on the earth. you could say, well, deserts are 35%. We’ll take, well, of that 6 % of the desert and we’re done. That’s very, reasonable to me. It’s like, it’s less than a 10th of the amount of land we spend on agriculture.

in

order to roughly 10x our energy wealth. This does not seem like a heavy lift to me. So we talked about Japan a minute ago. Japan has been acutely sensitive to their oil scarcity for more than a century. By some accounts, the reason for the Japanese attack at Pearl Harbor provocation was the US

enforcing an oil export embargo against Japan. And then, of course, the Japanese war effort fell apart in 1944 and 1945 because the US was able to project power across the Pacific Ocean and to detect the importation of oil from Sumatra, One of the carriers was lost at Midway because it was fueled with crude oil rather than refined oil, which was much more volatile.

because it still had the lighter petroleum liquids in it and they volatilized into the Hangar Bay and then exploded. So, similar story for the Nazis as well. So, you know, we’ve spoken to a number of potential partners in Japan over the years and at some point I said, well, we’ve run the math on how much solar you would need to produce all the oil and gas that Japan could ever want within its own borders, you know, within the borders of the home islands.

And my interlocutor kind of did a double take. Because forever that’s been an impossible dream. Japan is a volcanic island arc nation. It’s mostly basaltic rocks. It doesn’t have meaningful volumes of any ores, really. It has to import essentially all its rocks. And now we’re saying, I guess it has plenty of carbonates. But everyone has plenty of carbonates.

Aashka Patel (25:57)

Mm.

Casey Handmer (25:57)

Now we’re saying like, oh well, Japan is actually also about two thirds forested. Basically no one lives in these mountainous forested regions. We’re saying, well, you can put solar panels on the south facing ridges or you can put wind turbines or whatever. You need to generate 10 terawatts of power or something. I can’t remember the exact number, but it’s all very doable with their own technology, with their own factories within their own borders. And then they no longer have to care about the fact that all their oil comes from the Middle East.

Aashka Patel (26:04)

Yeah.

Casey Handmer (26:26)

for example. Okay.

That’s a big deal. And that actually applies to the number of countries on earth that are advanced developed countries that have enough oil within their own borders to supply their own needs is the United States. That’s it. Everyone else lost the lottery.

Right?

Aashka Patel (26:48)

Yeah, yeah, yeah, makes sense. you are saying that the land is enough for the solar panels, earth has enough land, then why? Okay, okay.

Casey Handmer (26:56)

Yep. yeah, easily. There’s only one exception actually. It’s the United Kingdom. Yeah.

So the United Kingdom’s population is so high relative to a solar resource that it couldn’t do it. We could do some, but it couldn’t do all within their own borders. They’d have to rely on wind power. Fortunately, they have some of the best wind resource in the entire world. So they can still do it. But the thing is, when it comes to exploiting that wind resource, they’re going to have to internalize their own wind.

turbine production capacity, because wind is going to lose the battle pretty much everywhere else. But UK will have to build their own wind.

Aashka Patel (27:28)

Yeah, yeah, So yeah, so why are there efforts of putting solar stations in the space and everything? you are saying that the land on Earth is enough for the solar panels? I think Caltech, your college itself, has one project, space solar stations.

Casey Handmer (27:43)

Yeah.

Space-based solar power. I think space-based solar power makes sense for powering satellites that operate in space. That’s why the space station has solar panels attached to it. But for beaming down to the Earth, doesn’t make sense to me. Never has. It seems like a really bad way of getting power.

Aashka Patel (28:10)

Casey, you have argued batteries will cannibalize transmission infrastructure because of this high utilization. So at what price point per kilowatt hour of storage does it become economically insane to build high voltage transmission lines instead of just deploying batteries?

Casey Handmer (28:29)

that happened like five years ago.

Yeah, and like, so it’s kind of a funny thing that like certain elements within the United States have been trying to build lots of transmission lines and they’ve complained about the regulatory problems, in particular around a law called eminent domain, which allows you to take private land to build stuff, preventing it.

But in this case, it’s actually saved us. The United States has not built very much transmission infrastructure in last five years. but like economically speaking, they should have built none. So we kind of, we kind of got lucky on that front. Yeah. No, I mean, I don’t know. Like I read the math on a blog post years ago. It’s like at 500 bucks per kilowatt hour or something. It’s, no longer makes sense to do long distance transmission lines. But the thing is, it’s not, it’s not even about like a specific number. It’s mostly just about like looking at the cost curves.

And like Euclid established that non-parallel lines cross eventually. like, it’s not even all that. It’s like 2,600 year old math proves that this should not be done.

Aashka Patel (29:25)

Hmm. Yeah, yeah, makes sense. So, you have predicted we will hit supply elasticity walls soon, whether in gas turbines, transformers or the grid itself, which breaks first. And what does that moment look like for an AI company trying to bring a new cluster online in 2026?

Casey Handmer (29:47)

That’s a good question.

So I think the question is, what is the rate limiting step from the perspective of the data center developer?

Aashka Patel (29:54)

Yeah.

Casey Handmer (29:54)

Yeah, so there’s a kind of a peculiar dynamic occurring on that front right now where a bunch of data center developers believed that they had secured grid supply. There was kind of a land rush for the last remaining grid supply. And having done that, the grid operators then realized that they wouldn’t have enough power to support the data centers all the time. And so they basically unilaterally have now unilaterally imposed.

throttling conditions on the data centers. This is a big headache for the data centers because they want to operate all the time. But it’s a solvable problem with batteries. So what happens is they put in an order with any of 20 or 30 different battery suppliers. And they just basically upgrade the UPSs at the facility to handle, instead of like a 30 minute outage, to handle like a two day outage or something. And this, it would be very extravagant to do that.

In any other system, but the GPUs cost so much money and not operating them cost so much money You just got to run them. So that’s all batteries as far as gas goes. That’s if you’re building your own power plant. I Mean X AI went and bought a gas turbine from overseas dismantled it and shipped it to the United States. So like I feel like There aren’t that many more times you can run tricks like that So I would expect that that You know, like we don’t make zero gas turbines. We continue to make them every year but

But in terms of their ability of those industries to scale up quickly and how much of the risk they’re willing to assume, given that the AI bubble might burst, seems kind of low. So I’d say in the next couple of years, we’ll see. It’s kind of a funny thing because the AI developers have been talking a big game about power procurement for a long time. And it just seems to me that they never quite...

Like whatever you’re hearing reported is probably a year or two behind what is actually happening on the ground. So until very recently, they’re all very much talking about small modular nuclear reactors as their power plants, right? Despite the fact that like, by the time they were talking about it, everyone actually involved with the coal face knew that this was not gonna work out, at least in the short term. And so they went and got all the grid connections and so on. But yeah, as far as the supply wall goes, all that really refers to is...

We saw the same things during COVID that like...

By default, you assume that your supply curve is actually a straight line. Locally, it is, but it’s not obviously over a long enough distance. So sooner or later, you get to a point where the curve starts to slope up pretty steeply. And the question is, how far away is that? And I think it’s actually pretty damn close for turbines.

Aashka Patel (32:35)

So that’s where these, no, but you were not in support of using these synthetic fuels for data centers, right? Like when I asked you, yeah, yeah. So drilling will remain the only option because currently for data centers, it’s heavily reliant on the natural gas production itself, like electricity from natural gas, right?

Casey Handmer (32:43)

No, I don’t think it’s a idea.

Yeah, I mean, we’re seeing colossus to, for example, built with the captive power plant. Now this doesn’t surprise me. Historically speaking, like you mentioned one gigawatt is a large number earlier. It’s like enough for a million homes or something. Not in summer, that’s for sure. Maybe like 50,000 homes in summer, because they all run air conditioning here. But it seems like a lot of power until you say, okay, well, what are the single facilities in our existing economy that consume insane amounts of power?

Aashka Patel (32:58)

majorly.

Casey Handmer (33:26)

And there things like aluminum production plants, for example, where just the power consumption is just catastrophically enormously high. It’s fabulous. It’s a lot of fun, right? And so what typically happens is if you have an aluminum plant, it’ll have its own captive power plant, right? And it may be the case that a nearby town also gets to tap in and use some of the power. But powering the plant uses 90 % of the power in powering the entire town.

that maybe provides some of the workers for the plant is like 10 % of the power. And that’s the kind of situation you’re seeing here. yeah, mean, the United States has very well developed gas transmission infrastructure. And so it’s relatively easy to, if you can procure a power plant to plug it in and then power your system. So we’re seeing that a few more times and particularly in Texas, has, parts of Texas have a lot of natural gas as well. Texas just has everything, man. Well, it doesn’t.

Aashka Patel (34:17)

Yeah, Halliburton does everything right in Texas.

Casey Handmer (34:18)

It has a lot of carbonates, I’ll it that way. I wouldn’t say it has like

super productive agricultural soil or something. But yeah, Texas is a pretty fabulous place. Okay, where were we? No, what I’m saying is in the limit data centers will have to be powered by solar, solar and batteries. It’s pretty clear. Like we’re running analysis. Actually my computer is crunching in analysis in this right now as we speak.

Aashka Patel (34:42)

Okay.

Casey Handmer (34:42)

Yeah, I ran it overnight and then I accidentally rerun it. So now I’m rerunning it again. Yeah, it’s quite fun. And so the million dollar question there is like, how much uptime do I actually need for my data center, for example?

Aashka Patel (34:53)

Yeah, the batteries, right? Yeah.

Yeah. Let’s move on to the next question.

your whole thesis is that AI will need massive amounts of power. But what if quantum computing flips that? What if we suddenly get 100x more compute for 10x less energy? Does that break your solar story?

Casey Handmer (35:16)

No, because that would presuppose that we’re able to saturate demand for computing, and that’s evidently not the case. I don’t know anyone who ever said, no, my data center is too big. Like, no, my computer is too fast. It’s never happened. It literally never happened.

I think it’s also worth pointing out that as far as quantum computing goes, there’s been a lot of promises and very little delivery. There’s been a lot of money and lot of effort poured into this for decades. And we’re not even at the point yet where we’re ready to set up a fab, which makes like the equivalent of the 4004 quantum computing processor or the 6502 or something. So like very primitive pre X86 architecture CPU for quantum computing. And even if we did...

what would we use it for different things than we use regular computers? Quantum computing right now is so fabulously expensive and losing so badly relative to boring old silicon computing on a cost curve basis that you could maybe only justify it for a handful of very particular things, which like AI have actually largely solved already. So for example, for a long time, it was thought that we’d need quantum computers to fully understand protein folding.

Aashka Patel (36:31)

Yeah, I thought it would happen.

Casey Handmer (36:31)

and then alpha-fold went and solved it.

yeah. And I’m sure if you want to do factorization of large numbers or something, then there’s a case to be made for quantum computing. But it’s very much a niche corner case. It kind of allows you to perform operations with the wave function itself rather than the square of the wave function, which is what everything else does. But it doesn’t, I don’t know.

I’m, my wife wrote a book on quantum computing once. And yeah, it’s, it’s, it’s about using the quantum computing system as a five, five qubit system, which is small enough that you can actually completely simulate it on like a toothbrush if you really want to. Yeah. So I, so I think that quantum computing is going to like magically somehow actually like a lot of the power demand for an existing data center is just in cooling, right? And quantum computers.

Aashka Patel (36:57)

okay. What’s new?

Casey Handmer (37:18)

as they’re designing to operate at cryogenic temperatures. So if anything, you would expect to see an increase in power demand. Okay, anyway, onwards.

Aashka Patel (37:23)

Yeah, yeah, makes sense, makes sense.

Yeah. So, if you were to make one bet, not a wish, but a genuine prediction, what will be the biggest surprise about AI energy infrastructure when we have the same conversation in 2027? in two years?

Casey Handmer (37:43)

it’s not that far away. I would say that some of the things that I go on about, which are still seen as...

Aashka Patel (37:44)

Yeah, that is why.

Casey Handmer (37:50)

still seen as somewhat controversial or something, will no longer be controversial. So for example, the idea that the AI data centers would become the center of gravity of electricity production and consumption, and that instead of the discussion being around PJM unilaterally imposing, we’re going to cut you off if we’re running out of power because our human consumers are more important than our AI consumers, even though economically speaking, the human consumers are much less important.

Aashka Patel (38:12)

Yeah.

Casey Handmer (38:17)

the discussion will be around PJM negotiating with their local compute cluster for access to their excess power. Because like, just in my neighborhood, I think in a given year, we’re lucky if we have 99 % uptime in my neighborhood, right? So at home. So we have like blackouts totaling 72 hours a year or something like that, which is kind of frustrating. But for an AI data center, you probably want to have like at least one or nine, if not two more nines of uptime. So.

What that means is the AI data center will have solar plus batteries. It’ll have excess power most of the time. That is effectively just, they’re not even using it. It’s just sitting there. So they’ll be able to sell that to be good neighbors, I think. But yeah, that whole discussion is, I don’t think it’s been widely understood yet that like, there probably won’t be independent power generation merchants in the future. There’ll be captive power plants for highly energy demanding.

systems, whether they’re data centers or synthetic fuel plants, and that they will become the default merchant of power for legacy grid users.

Aashka Patel (39:17)

Hmm. Yeah. So have you heard stories of like people’s utility bills going up because they have a data center nearby Like why is it so?

Casey Handmer (39:28)

yeah, I’ve heard these stories. I think they’re

mostly crap. I don’t think they’re true. I mean, like people’s people’s utility bills have gone up. And

Aashka Patel (39:35)

Three times I heard that

Casey Handmer (39:37)

Yeah, I mean, there’s a lot of articles doing the rounds here, but not everything that’s published in a newspaper is true. Okay. It’s going to shock you to learn that. What’s actually happening is, I think there’s an argument to be made that if you had to baseline inflation on one metric, it should be energy prices, right? And all the rest, you can have this big basket or whatever, but like, what’s really happened in the United States, at least, is that the US dollar has devalued by about 40 % over the last six years.

And, yeah, fortunately there’s been quite a bit of economic growth in the background. Whereas for other Western countries that have also seen hyperinflation, they have not had commensurate economic growth and now they’re in a real pickle. but, but yeah, basically like your, utility bill now is higher because your money’s worth less. And if you’re on fixed income because you’re a retiree or something, then you’re in deep trouble. Right. It’s a big, big problem. like,

It makes me quite angry because we’re a very sophisticated civilization and like all the different aspects of this story are just failures. They’re failures, right? The first is it should not be that hard to make energy cheaper over time. We’ve known how to do it 50 years, right? We’ve just chosen not to. Secondly, it should not be that hard to not have like massively inflationary government spending, right? It just requires us to not spend like print money and spend it on dumb stuff, but we do it anyway. Okay. And thirdly,

Aashka Patel (40:59)

Hehehehehe

Casey Handmer (41:00)

The fact that we have retirees on fixed income is a dire indictment of the fact that we spend $50 billion a year on the National Institute of Health and we still haven’t figured out aging, like human aging. Like the fundamental problem is not that like Mrs. Mulberry can’t afford her power. The fundamental problem is that Mrs. Mulberry got old and she should never have gotten old in the first place. She should have basically been healthy and vigorous for essentially indefinitely. And this is a technology that for whatever reason, the US with all its wealth and power and the rest of the country, of the world with all its wealth and power.

Aashka Patel (41:08)

Hmm, yeah, big problem.

Casey Handmer (41:30)

has put peanuts into over the years trying to solve. We’ll spend 10s of billions of dollars a year on cancer treatment and we’ll spend maybe a couple million dollars on like, what can we do to make these mice live longer? It’s bizarre to me. It really changes your perspective when you realize that almost all the diseases that are currently bankrupting healthcare systems all around the world are not diseases that like young people get. They’re diseases of old age. They are, are

Aashka Patel (41:55)

yeah.

Casey Handmer (41:55)

downstream

of a core problem, which is that if you build a society that’s rich enough that a meaningful number of people survive their childhood and survive their middle-aged years, then they all become old and their care all becomes expensive and they all basically decline and get sick and die. And that’s very expensive. So I know it’s kind of bit of pyramid scheme situation. And the only way out, and this is the case in India as much as anywhere else,

where we’re collapsing birth rates, the only way out is to radically extend human life through advanced medicine.

Aashka Patel (42:30)

Yeah, yeah, makes sense. Yeah, so you call yourself a recovering physicist. You saw brilliant people in academia solving hard problems are highly underpaid. So now you are building terraform. money is a big reason And you get to solve the hard problem. So for young physicists or engineers watching this who are intellectually hungry,

Casey Handmer (42:37)

hehe

Aashka Patel (42:54)

but also want to build wealth, what’s your advice and what problems should they focus on in this AI age?

Casey Handmer (43:03)

It’s a good question. I mean, I didn’t leave academia and go and start a company so I could get rich. Just to be clear, I could earn much more money doing something different. Well, that’s not strictly true, actually. I’ll be honest about that. I’m well off. I have nothing to complain about. Financially speaking, I’m well off.

Aashka Patel (43:11)

No, no, Yeah, but yeah, you quoted that a lot of physicists end up in academia,

Okay.

Okay.

Casey Handmer (43:28)

But the sort of wealth that I sought by starting a company was not financial wealth. I could go and work at Google for much more money than I currently earn. I could go and work at an AI lab for much more money than I currently earn. Why am I doing this and paying myself such a meager salary? It’s because I’m becoming rich in something that you cannot buy and you cannot earn at most companies. It’s an experience of running your own show and creating opportunity for other people and building wealth and trying to do something that’s very, very difficult.

And if it’s your thing, there’s no other way of getting it. OK.

What’s my advice to people who want to build wealth? Buy index funds. Quite frankly, not investment advice, but the number of people I know, particularly if we don’t figure out aging, who just assume that like they’re going to retire at 65, you need to be compounding wealth in a way that is, you know, at least comparable to the broader market. And it turns out that they kind of long-term, I think Warren Buffett has done extremely well by making extremely long-term bets on good businesses and just compounding.

Aashka Patel (44:25)

Yeah, at compounding.

Casey Handmer (44:28)

compounding, compounding, compounding. And you probably in general are not smart enough or interested enough to like pick good stocks. mean, once in a while you can bet on something just for the fun of it. I’ve done that a few times. got very lucky once. Lost almost everything every other time, right? It’s gambling. But if you buy index funds, then you’re basically taking a position on the future health of the economy. And that’s a correlated outcome as well, right? So like, if in 50 years time, the United States for me is a richer place than it is now, then my index funds will be doing well and I’ll be well off. And if the United States has collapsed,

Aashka Patel (44:43)

Okay, yeah.

Casey Handmer (44:58)

then it’s not like putting a bunch of cash in a bank account would have helped anyway. Okay, what’s my advice? I recently wrote a blog post on how to improve your resume if you’re a recovering JPLer or NASA person who’s kind of on the cusp of being laid off. It’s really important to focus on how your skills generate value that is legible for the end customer.

and not assume that just because you’re there, they’ll see you’re brilliant and give you money. And a good exercise to move in that direction is to practice using the AI tools, understand how they work, understand how to get good work out of them, and write a resume and write some blogs. But writing blogs is fabulously underrated. Generally speaking,

Aashka Patel (45:40)

And good quality

blogs are also rare, right? These days like with AI generated content being bombarded.

Casey Handmer (45:47)

yeah, yeah, yeah.

Don’t try and pass off AI slop as your own work. That’s a kiss of death. You’ve got to write something from the heart about something you’re passionate about that really shows you’ve thought deeply about it. It creates a durable proof of work that you’re not just like posting 140 characters on some random subject. It also creates a proof that you’ve been thinking about a subject for many, many years. It also helps educate people. It also gets fed into the LLMs and makes them smarter. It also means the LLMs know who you are, which could come in handy someday. You never know.

But

it also forces you to think more clearly about what you’re doing and trying to communicate it and some of the stuff that I’m working on is so out there and so hard to comprehend even for me who works on it every single day that like if I look at my first blog post on the subject it’s miserable but I left them up there because it’s a record of progress I guess. I do very occasionally edit things if I find a mistake or something but I don’t take down blog posts because I no longer...

because I’m no longer proud of them or something. I think it’s just part of the production function. You’ve got to keep on making them. Yeah, that’s right. Publish on the internet. mean, not that long ago, it was basically impossible to get your words out there. You’d have to find a publisher who printed on paper and put it in a library and someone has to read it. That’s not the case anymore. What a world we live in. Yeah, write some blog posts. Do the research. Make them interesting.

Aashka Patel (46:56)

Yeah. That’s a great advice. And just like the AI companies are not moving their old models out of the market, you shouldn’t move your old blog posts out of the market. Just keep them and show a natural progression of you getting smarter, smarter, smarter. So are there any specific skills for a physicist or an engineer that you think today they should learn?

it will compound over the years,

Casey Handmer (47:27)

I don’t know. I don’t think I’m well qualified to give career advice. I would say don’t try and validate my mistakes by repeating them, obviously. I’ve gotten very lucky. I’ve gotten very lucky. I found physics to be a very productive jumping off point for my intellectual growth and exploration, but obviously it isn’t for everyone. I think it’s very versatile. I think that in general, if you want to work in a technical sphere, you’ve got to be obsessive about hoovering up just all the information you can possibly find.

Aashka Patel (47:37)

Okay, okay, that’s good enough otherwise.

Mm-hmm.

Casey Handmer (47:56)

in any configuration you can possibly find it. You can ask people questions. Here’s the thing about writing. Why it’s valuable. For every hundred people on the internet, only one of them will even write a little bit. Like, you know, reply to a post or something. And for every hundred people that even write a little bit, only one of them will write anything of any length. So if you actually take the trouble to write like 10 half-decent blog posts, you are already in the .01 % in terms of total intellectual activity on the internet.

Which is one of the reasons why the AIs are so crazy, because almost by definition anyone who goes to the trouble of writing a million words on the internet is a crazy person. Sometimes productively crazy, but still crazy. So yeah, that’s... But it’s a good crowd to be part of.

Aashka Patel (48:28)

Yeah.

productively.

Yeah, yeah. One last question before we let you go. So, just a personal question, like you are so busy, but yet you take time to do so many podcasts, like while doing the research for this podcast, I had to like go through a lot of them. So, how do you take time out and like why and how you manage to do this?

Casey Handmer (49:00)

I don’t do every podcast, but...

Aashka Patel (49:01)

No, like a lot like as

compared to the other guests like there was a lot of content available on how Casey Hand mer things. So, yeah, how do you manage?

Casey Handmer (49:11)

Yeah, well it’s part of communication. It

feels like work, but it’s pretty easy. It’s good practice. But I don’t do that many. mean, like, again, it accumulates. On my website, I have a list of every podcast I’ve done, and they are maybe 25 over the last 10 years or something. So on average, that’s only one every couple of months. So it’s not that many. I probably write many more blog posts. Yeah, definitely. I’ve written hundreds of blog posts at this point, so.

Aashka Patel (49:31)

Yeah. Yeah.

Yeah, yeah. So thank you so much, Casey. It was nice talking to you. I hope you had fun too. So thank you so much. And let me stop the recording.

Watch on YouTube; listen on Apple Podcasts or Spotify.

About Hemali Rathnayake:

Dr. Hemali Rathnayake is a materials scientist and nanoscience professor at the University of North Carolina at Greensboro who co-founded Minerva Lithium, revolutionizing sustainable lithium extraction for AI’s future. Her breakthrough adsorption-based technology produces battery-grade lithium carbonate from hard-rock deposits in just 48 hours without water waste—advancing from prototype to pilot scale for commercialization by 2027. Her expertise spans AI materials discovery, rare earth elements for quantum computing, semiconductor development, and next-generation cooling technologies for AI data centers, with research funded by NASA and the Department of Defense.

Episode Summary:

Lithium extraction, AI materials discovery, battery supply chain crisis, and quantum computing materials explored. Aashka Patel and Hemali Rathnayake discuss why lithium is the real bottleneck for AI data centers, not chips. Learn about Minerva Lithium’s 48-hour water-free extraction method, the coming lithium shortage threatening AI’s future, and why recycling e-waste is critical. Discover how AI is revolutionizing materials science by finding thousands of new compounds, room-temperature superconductors, and next-generation cooling technologies for AI chips. Topics include rare earth elements powering quantum computing, sustainable battery innovation scaling by 2027, AI-designed semiconductors, and the 5GW data center lithium requirements. Expert advice for future material scientists adapting to AI advancements in energy breakthroughs and the Manhattan Project-level challenges ahead.

Timestamps:

00:00 AIR Bites/Precap
02:26 The Real Bottleneck of AI Isn’t Chips — It’s Lithium
04:56 Inside the Hidden Bottleneck of America’s Battery Supply Chain
06:01 Can Innovation Fix the Lithium Crisis? Meet Minerva Lithium
06:55 How Lithium Is Actually Extracted — and Why It’s So Dirty
08:52 A 48-Hour Lithium Revolution: Extracting Without Water Waste
10:03 From Lab to Market: Can Sustainable Lithium Scale by 2027?
12:03 How Much Lithium Would a 5GW AI Data Center Need?
13:57 The Coming Lithium Shortage: Can Recycling Save AI’s Future?
16:42 Why AI Data Centers Demand a Whole New Kind of Battery
18:50 AI in Materials Discovery — A Revolution Already Happening
20:50 Can AI Help Discover Room-Temperature Superconductors?
22:05 How AI Found Thousands of New Materials We Missed
25:39 Quantum Computing: The Next Material Crisis
26:48 The New Oil: Rare Earths That Power Quantum & AI
30:28 The Secret Materials That Could Cool AI Chips Safely
33:34 Will AI Design the Materials to Build Itself?
35:05 By 2027: What’s the Next Big AI Energy Breakthrough?
36:40 If You Had a Manhattan Project Budget for AI Energy…
38:07 Advice for Future Material Scientists
42:52 Outro

Transcript:

Aashka Patel (00:04)

hello and welcome Hemali to On AIR with Aashka let’s dive right into the questions. every AI lab is racing towards building AGI and the real bottleneck isn’t the chips or even the data centers, it’s lithium because to

power AI at scale, we need massive battery storage and that comes from lithium China refines 70 % of the world’s lithium. So they control the entire supply chain from mining to battery grade material energy without storage is useless and as per Mark Zuckerberg, energy might be a bottleneck to the current AI advancement.

Here’s the question, if China controls the lithium that powers batteries that stores energy that runs AI, so why doesn’t China win this race by default?

Hemali Rathnayake (00:53)

So it is because China can control the supply chain, that production of lithium. But is China is controlling the battery technology and energy storage technology. So most of the successful energy storage technologies and battery technologies lies in within US and the Europe. So of course that these countries Europe and US mainly

that’s we’re facing in the supply chain and in the lithium, but the battery technologies relies in the within across the other continents. So because of that, China cannot be the lead in the AI storage in the market.

Aashka Patel (01:38)

Okay, so like can you elaborate more on the battery technology, what do you mean by battery technology?

Hemali Rathnayake (01:44)

Yes, so

let’s take for example Tesla EV battery technology. So, that technology is developed in US. Of course, China has a couple of battery technologies, for example, waveform lithium, that’s a sodium ion battery technology, that’s where they are leading for most of the EV, for example BYD technology.

And but the lithium battery technology is leading countries, US. So because of that, so China, even though China has the lithium, so they don’t have a technology to develop battery production over there alone.

Aashka Patel (02:31)

Okay, okay, okay. Got it, got it. But like, does US also have the mining capabilities and the extraction and everything along with the technology or is it dependent on others?

Hemali Rathnayake (02:43)

That’s

where the bottleneck in US and the surprising bottleneck US is facing because domestic production of these raw materials are minimum. So I would say as US produce lithium less than 4 percent 4 percent versus China is the lead.

Aashka Patel (02:47)

Okay.

Okay.

Less than 4%.

Hemali Rathnayake (03:09)

in lithium production over 70%. So that’s the bottleneck. So I think that’s why it’s one country maybe cannot lead in their AI ways and for the storage. So I think most of the both countries or the whole continents and supposed to come together and then work together to solve the supply chain challenges.

Aashka Patel (03:17)

Uh-huh. Mm-hmm.

Okay, okay, okay. Got it, got it. So, you mentioned the bottleneck in the US. So, you have a company in the US called Minerva Lithium. it’s on a mission to transform how we source one of the world’s most critical clean energy materials that is lithium. So, can you explain briefly what are the different ways of lithium extraction and how is your way more sustainable than others?

Hemali Rathnayake (04:02)

Yeah, so the lithium comes with different feedstock. The main feedstock lithium is coming from hard rock, that’s the mining from hard rock. And US hold some of the hard rock mining, for example, North Dakota, and especially North Carolina. So that’s the major resource lithium mine in US currently.

But other than that, lithium can also extract from brine. So for example, salt rake brine, and also there’s some of the packing produced brine. So those are the two methods that we can refine, get the lithium out. However, in the current lithium mining for the hard rock or the brine, it takes time. For example,

put into perspective. So the heart of mining need a lot of energy. So it is very energy consuming and then environmental impact is high. People hate mining, right? and then for brine, so it’s a major way producing lithium from brine is solar evaporation. That’s take 18 months to two years to get the lithium out.

Aashka Patel (05:26)

MG! That’s

a longer time, yeah?

Hemali Rathnayake (05:29)

long time. So,

it also includes chemical precipitation. So, but recently, so people have been developing absorption, selective absorption processes. So, to get the lithium directly extract from brine, that’s called direct lithium extraction. But it’s still all these processes has lot of disadvantages and advantages. But the main disadvantage is

any of these processes, it needs energy and the carbon emission is high. And most importantly is the fresh water usage. So for example, put into a prospective, I want to produce one metric ton lithium carbonate equivalent. For all these processes, you need about 30,000 gallon of fresh water.

Aashka Patel (06:26)

OMG!

Hemali Rathnayake (06:29)

So that’s where Minerva Lithium plays a role. So the

Minerva Lithium Nanomosaic technology we developed, it’s absorption process, but it works in room temperature. So it is environmentally friendly. You can extract lithium within 48 hours. But this is not exactly the direct lithium extraction. We call it as a passive lithium extraction.

What that means is we take the impurities and concentrate the lithium, then there is no use of fresh water, like 30,000 gallons per metric ton. So we’re down that to 5,000 gallons per metric ton. That’s a huge...

Aashka Patel (07:15)

Oh, that’s a

considerable amount of reduction.

Hemali Rathnayake (07:20)

reduction. and then we don’t have a carbon emission. We don’t use the higher energy processes. So that gives us more sustainable, environmentally friendly method to extract lithium from brine resources.

Aashka Patel (07:37)

That’s interesting, like literally. So, how is it going right now? what’s your business model,

Hemali Rathnayake (07:44)

Yes, so and currently and we actually we just wrap up our technology showcase last week. That was October 2nd. So we had our technology showcase and we asked all the leading investors to come in and take a look at our technology. we were able to, we are at this stage in the pilot scale. We were able to validate.

Aashka Patel (08:03)

Mmm. Mmm.

Hemali Rathnayake (08:10)

at the pilot scale and with the minimum viable product, our continuous filter process. Now currently we are in the seed fundraising stage for up to 10 million to get into the commercialization by 2027.

Aashka Patel (08:16)

Okay.

Thank you.

Nice, nice, nice. So, the process sounds very exciting and environment friendly. So, the lithium that is extracted out of this process, is it like direct battery grade lithium or does it need more processing or something

Hemali Rathnayake (08:45)

Yeah, so the the Minerva Lithium Technology is from extraction to refine into battery grade. It’s a whole supply chain. So we what we do is we extract and then we directly refine to battery grade lithium carbonate currently. So our battery grade lithium carbonate purify is 99.95. That’s the

maximum that’s the normal purity that needed for normal battery but if you want to get into the EV battery technologies so you have to reach to the 99.99 so that’s purely for example Tesla EV batteries using for so but in general it’s 99.95 is the market available lithium carbonate purity.

Aashka Patel (09:38)

yeah, but I’m pretty sure you will reach that purity level of EV batteries too. Yeah. So from a materials perspective, like if we wanted to flip the five gigawatt data center to solar.

plus storage tomorrow, what’s the actual lithium requirement and is that even physically possible with the current refining capacity that you have or combined technologies have

Hemali Rathnayake (10:04)

the within US or in overseas and with the current refining technologies, they are traditional technologies, right? So because of that, they need more energy to extract and refine. So because of that, so that’s where the bottleneck and always that’s the challenge. So with the current refining technologies and

If we combine these traditional pathways with the innovative technologies like the Minerva lithium technology, so we may able to meet the capacity that needed. But however, if we talk about 5GW, so let’s put into the perspective, right? So if we put into the perspective, so therefore single rack mounted battery for 5GW you need about 4,800 volt

hour capacity, battery capacity. So what that means is it’s basically you have to produce at least 400,000 metric ton of lithium per year to meet at least one rack of battery capacity for 5GW.

Aashka Patel (11:23)

my God. So it seems almost impossible to meet the demand. what’s your take from the numbers?

Hemali Rathnayake (11:32)

So it is impossible to meet the demand from the existing natural resources. That’s why as a scientific community and the industry partners, they should look into recycling. So there are a lot of battery e waste, but there is no technology at the commercial scale you can extract these.

Aashka Patel (11:48)

Hmm.

Hemali Rathnayake (12:00)

critical material from used batteries, recycled batteries. So if we find a path to recycle and extract this not only lithium, other material, cobalt, material exactly going into the cathode, and I think we should look into more of the recycle and recovery these critical minerals from recycled batteries.

Aashka Patel (12:12)

Yeah, other important materials are also there, right? Yeah. Yeah.

So like from your experience or from whatever you have heard how many efforts are put or research is put into this recyclability because I do have some stats on that the AI companies are upgrading their GPU clusters almost every 24 months as new Nvidia chips are coming out. So there’s a tsunami of e-waste that is getting created.

Hemali Rathnayake (12:47)

Mm-hmm.

Aashka Patel (12:52)

So, of course there is a big opportunity to solve this problem. what are the advancements happening there to solve this problem.

Hemali Rathnayake (13:01)

Yeah, so the complexity is because when you have the recycled battery, you get this powder that called black mass. So the black mass contain all the anode, cathode, all the mixture of materials. So now you have to extract these minerals purity and take it out from black mass. So there are a couple of industry leading in this black mass technologies.

and then find to capture these, get these, extract these elements out of black mass. So it’s still in the lab scale or the pilot scale in the research and R &D developments. So, and but there are a couple of technologies that people have been successful. That’s one is the electrochemical method, separation method.

Aashka Patel (13:42)

okay, not at an industrial level.

Hemali Rathnayake (13:54)

I

think that’s one of the technologies currently leading that you can take this black mass and extract these critical materials. At Minerva Lithium also we’re looking into whether we can use our technology combined with electrochemical process to larger production of lithium from this black mass.

Aashka Patel (14:16)

Black

mass, yeah, yeah, that would be revolutionary like literally. Yeah, yeah, yes. So data centers these days are running 24, 7, 365 days at 90 % plus load factors. But the batteries serve a different role than typical grid storage. So what makes these data center

Hemali Rathnayake (14:20)

Mm-hmm. Yep.

Aashka Patel (14:38)

battery requirements uniquely demanding from power density to response time to reliability compared to standard utility scale or like as you mentioned the EV batteries. So what is the difference there?

Hemali Rathnayake (14:54)

So it’s the grid capacity, right? So you need very large grid capacity for these data centers or the 5GW data centers for the AI. So I think that’s where the huge problem. So it’s not only that these battery technologies needs to be moved forward and advanced, but the grid capacity.

Aashka Patel (15:10)

Hmm.

Hemali Rathnayake (15:22)

In terms of the grid capacity, we need to advance in the grid capacity to reach it to where we wanted to be with the data centers and use it. So that’s the huge bottleneck in grid capacity.

Aashka Patel (15:37)

So, that, so as per what you are saying like that is not a problem to be solved by the battery technology, but it is a problem to be solved by the grid capacities and everything that is considered, right.

Hemali Rathnayake (15:50)

Uh-huh. Yeah, grid

capacity and the infra structure needs to be go side by side. And that’s why so there are complexity, not only the battery supply chain. And then it’s a manufacturing capability, infra structure, grid resilience. And all those needs to, you need to work alone with all those side by side to

which way we wanted to be with this trend in AI or the 5GW capacity.

Aashka Patel (16:24)

Yeah. Yes, yes, yes, yes. So yeah, Google DeepMind claims that AI is going to revolutionize the materials discovery like on Google DeepMind’s podcast itself, I heard Pushmeet Kohli talk about two millions of

Hemali Rathnayake (16:24)

It’s eventually an emergency, so...

Aashka Patel (16:43)

materials that AI has predicted could be stable before it was something 20,000 materials that we as humans were able to find out to be stable but 2 millions are what AI models have predicted but of course it requires

human testing like are they actually stable and sort of things. So in your research on nanostructured materials, battery components, MOFs are you seeing genuinely non-obvious breakthroughs or is AI mostly optimizing the existing chemistries faster? Like is AI actually helpful in new materials discovery?

Hemali Rathnayake (17:23)

So the AI, definitely, definitely AI is helping the material discoveries and also the search in the existing database, right? So you cannot do in the brute force method.

Aashka Patel (17:38)

yes.

Hemali Rathnayake (17:39)

So,

the AI can help you to search the database in the existing material. Sometimes existing nanomaterials we never explore for battery aspect or the energy storage. Maybe they use for them for different other applications. So, think that having AI apply into the material platform is

Aashka Patel (17:56)

Yeah.

Hemali Rathnayake (18:04)

been has been beneficial and it advance finding new material or develop some of the existing material and optimize their properties to get into the way we wanted to be. I definitely agree so the AI has been helping for material discovery.

Aashka Patel (18:24)

Got it, So, like I read this somewhere, the holy grail of material science is to discover a room temperature superconductor. Like, correct me if I’m wrong. so looking at the current scenario and as you mentioned that AI is definitely helping in the material science discovery. So, do you think AI can help us get to that holy grail of material sciences?

Hemali Rathnayake (18:49)

Yes, definitely. An AI can generate the hypothetical or the new material, right? And then eventually, of course, someone has to make the material. I think that’s where there is a timeline. And so it takes longer time to develop the material and then synthesize the material and getting into the commercial scale.

But definitely the way that we have been adopting AI programs and algorithms to find the material discoveries, and I think it is possible. this my wish is maybe by 2030 and we may have something that room temperature material discovered for superconductivity.

Aashka Patel (19:40)

Okay, nice, nice, nice. So far like in your experience, because you are also a professor, right? So in your experience and in your discoveries, if you can put a number to like, okay, AI predicted these many stable crystal structures or new alloys and out of them, like how many of you like human tested and then found out that they are actually stable materials that

wasn’t maybe humanly possible to discover them are those kind of

aha moments that you had like for the lack of a better word

Hemali Rathnayake (20:18)

Yeah, so I think it’s a couple of AI models that help to develop is the some of the high energetic alloy material. There are a lot of high energetic alloy materials has been advancing in the scientific community. If you take a look at publication records in the high energetic material, so they

been alloys different like partial stoichiometry, right? If they are not like formulation is easy, but those has been advancing in this regime. So it basically, and then if you take a look at metal–organic frameworks, of course, and this year we got the Nobel Prize for the metal–organic framework. So we do have 84,000

synthetic metal organics framework that’s known in the database. But we barely explore them to use in the magnetism, so paramagnetism, superconductivity. I think the AI models that develop in this metal organics framework platform has been tremendous. And then you can actually search the existing database.

So those are some of the materials that AI helped to develop, like high energetic entropy material. and searching the database in the metal organics frameworks to apply them into different applications in, for example, solid state electrolytes in the lithium ion battery. For example, anode and the cathode materials. So those research has been done with the

side by side with AI.

Aashka Patel (22:10)

Okay, okay, okay, got it. So the database that you are describing, is it like, how is it kept updated? Like is it a central repository used by the universities of US or like how?

Hemali Rathnayake (22:22)

Yes,

there is a central depository and then owned by UC Berkeley and that’s where Dr. Yagi has been for the one of the Nobel laureates for the metal–organic.

So, they have been updated in this repository And also the Cambridge Crystallography Database has all the crystal structures of these materials that one can utilize to develop algorithms or the AI models to search the properties or create the properties with the optimized structures.

Aashka Patel (22:58)

Got it, got it. So, like the Cambridge one and the UC Berkeley one that you mentioned, like are they open source? Okay, okay. Then that’s great. Like yeah.

Hemali Rathnayake (23:05)

open source.

They are open source.

can access. are open source.

Aashka Patel (23:14)

Got it, got it, yeah, yeah, that’s a great initiative taken by both the universities. let’s jump on to quantum computing for a while. So, quantum computers need totally different materials, superconductors, exotic stuff, right.

quantum takes over, are we just swapping the lithium bottleneck for a whole new set of material problems that we haven’t solved yet?

Hemali Rathnayake (23:38)

Yes, so for the quantum computing you need high magnetic material, need the rare earth elements and that goes to quantum computers. So that means we are opening another supply chain bottleneck within

Aashka Patel (23:40)

What are those materials like?

No.

Hemali Rathnayake (23:59)

and across globally. yeah, so people talk about quantum computing, but you need the rare earth elements to get into the quantum computing, especially for magnets. So that’s where the bottleneck and how you can create, get these rare earth elements for magnets for the quantum computing. Yep.

Aashka Patel (24:01)

Okay.

Yeah, yeah,

makes sense. So currently like if you were to look at the world level like where are these rare earth elements available right now in the world?

Hemali Rathnayake (24:34)

So again, when it comes to rare earth elements, it’s the leading countries are, one is China, other one is Russia, and then also African continent. So those are the ones that has more of the rare earth element deposits. also recently, if you take a look at like critical mineral leaders, so there are 10 countries and including India. So India has the deposit of rare earth elements.

and then all other African continent, Tanzania, for example, those countries has rare earth elements. US also has about 1.8 million tons of the rare earth elements in US. then, you know, it is mining and extraction and putting them into the commercial scale and the production is the best way.

just that we don’t have a domestic infra structure developed.

Aashka Patel (25:33)

Yeah, so like for these rare earth elements like is research happening on to how to extract and how to better utilize them for quantum computing like all over the world?

Hemali Rathnayake (25:40)

Mm-hmm.

Yeah, there are

separate research domains looking at the rare earth element extraction and refining and then creating the magnets that goes directly into semiconductors and including quantum computing. And there are leading companies that recently start in this domain.

So, again, so they are either in the lab scale or the bench scale or the pilot scale. And the commercial production is really, really low in compared to some of the leading countries like China and or maybe Russia So, those countries, I think they are the main countries that and Australia.

leading these rare earth elements exporting.

Aashka Patel (26:39)

Okay, okay. So like the reason why I asked this question is to see if there is a scope of Minerva lithium in rare earth elements or not. So like viewers watching this, if they are interested then they can go and build a Minerva lithium for rare earth elements. So yeah, yeah.

Hemali Rathnayake (27:00)

Yeah, yeah, so we are

looking into it and because since this technology can separate, if we are successful separating cobalt, nickel and lithium from black mass or the recycled material, I think that’s the path to move forward and we validate that technology and the process.

Perhaps, Minerva lithium could lead into rare earth elements because they come with very low concentration and then they come with the hard rock stage. So you have to refine them, separate them from the other impurities and other minerals. So that’s maybe the future.

Aashka Patel (27:52)

yes, yes, yes, yes. Yeah, this is a revolutionary breakthrough. Whoever gets the minerals from this black mass. So yeah,

Kudos to anyone who solves it because that’s the demand of this age. yeah, let’s move on to the next question. Data centers are moving to liquid cooling for AI chips. So most liquids are either flammable, corrosive or terrible heat conductors. So are there noble fluids or phase change materials from nano science perspective that could revolutionize how we cool AI hardware?

Hemali Rathnayake (28:31)

So the yes and there are research that people have been looking into nano scale material development for instead of liquid cooling is they are either aerogels or the they are solid state and or they are in the liquid and the liquid and the solid in the middle phase so

The most of this liquid, coolant material are either as you mentioned, they are ionic liquids or they are high-flammable liquids. But now people look into more of the, in the liquid phase that they can be solidified quickly at the room temperature. They are called deep eutectic solvents. And this is one of the people trying to.

In the nano scale, we are trying to look at whether we can use these deep eutectic solvents and mix with composites with the nano materials, for example, zinc nanoparticles. So because they have a very good thermal properties and then heat absorption properties. So those kind of materials people have been looking into.

And yeah, so one of the novel is these deep eutectic solvents domain. then people are trying to develop these deep eutectic solvents from sustainable readily available materials like urea, choline chloride, and ascorbic acid, citric acid. Those are mixture of two organic material.

and then when you mix it, they became liquid but

they are very high temperature liquids. So that address the flammability of ionic liquids.

Aashka Patel (30:29)

Okay. Yeah, that’s very interesting because like before I started researching for this podcast, I only had the idea from the news that water is, fresh water is getting used for cooling down these like computing systems in the data centers and as a coolant and basically I had no idea about these different materials. So yeah, it’s very interesting to know because

Hemali Rathnayake (30:44)

Mm-hmm.

Aashka Patel (30:55)

I thought why are we using these limited freshwater storage that we have as the planet earth? So this is very interesting to know like, okay, other coolants are also getting used.

So, Meta and Google are now using AI to design their own chips. So could AI eventually design the materials to build itself like better semiconductors, better interconnects, better thermal materials? Like you mentioned that from material science perspective, it’s helping a lot, but like is this possible? What’s your take on that?

Hemali Rathnayake (31:33)

Yes, it is possible. mean, we exercised using AI eventually to develop the COVID vaccine, right? And within less than two years, with less than a year, we were able to develop the COVID vaccine because of the AI. So, and as I mentioned, rather than the Brute Force method, and so...

AI can help and better develop if we develop better algorithms for AI models, language models, and it is possible we can use AI to develop the material revolution. There are a lot of research now driven towards this, right? We’re a lot of data centers, AI-driven data centers, AI-driven...

material hubs in US and the national labs are on this game, right? So I think in the future, is rather than we designing the material from the scratch, I think the AI will accelerate the material revolution. I believe it.

Aashka Patel (32:40)

Yeah, yeah, makes sense, makes sense. So if you could make one bet, not a wish but a genuine prediction if you were to make. So what will be the biggest surprise about AI energy infrastructure when we have the same conversation in 2027?

Hemali Rathnayake (32:58)

2027 is not far away, it’s two years. Exactly, when we come to AI energy in fra structure and within 2027, we’re going to have again the conversation about

Aashka Patel (32:59)

Far away, yeah definitely. That is why I asked because longer timelines they are harder to predict, right?

Hemali Rathnayake (33:16)

grid capacity and the grid resilience. So I think people need to pay attention and policymakers and legislatures and funding agencies and federal funding and all those they need to think about how we can build grid capacity and the grid infrastructure resilience. So that’s going to be the discussion again we’re going to have and

Maybe the battery technologies can be advanced and energy storage can be advanced, but building them into the grid resilience is going to be a problem.

Aashka Patel (33:53)

Ultimate problem to be solved. Got

it, got it. Hopefully the Stargate project solves it to a greater extent, right? Because a lot of money is getting poured into that project and a lot of land area is also allocated to that. So, hopefully we see a better solution there.

Hemali Rathnayake (34:03)

huh.

Aashka Patel (34:15)

Like as we talked about the Stargate project that is of the scale of Manhattan project, right? That happened during the World War II. So if you could assemble a Manhattan project style team with unlimited funding, you have as many dollars as you want to solve one materials challenge that would unlock the next decade of AI, what would you work on?

Would it be solving the battery problem or room temperature, superconductors or something entirely different? Like what’s your like one material

Hemali Rathnayake (34:52)

So the energy demand is the main cause for all the global chaos, right? And everybody is chasing on energy. So I would say if we had unlimited funding, funding, it is solving the energy problem.

Aashka Patel (35:10)

Funding available. Yeah.

Hemali Rathnayake (35:18)

to

better batteries and then also the room temperature superconductors.

It’s kind of that would be that more of the leading projects we should think about when the energy demand is able to resolve. So I think the world will be in a better place.

Aashka Patel (35:42)

Yeah, yeah, makes sense, makes sense. So, as you mentioned about like AI having immense potential in material sciences and material engineering. So, like if someone is studying material sciences or material engineering today, like what would be the one piece of advice that you would give to them for them to survive in a world where AI would

mostly be helping us do discoveries and everything.

Hemali Rathnayake (36:11)

Yep. So if you are a student or going to be in the material science world, so you have to be continuously update yourself and adopt AI to utilize better in your research and the material discovery. so AI can be a great tool.

for your research, for your material discoveries. think it’s Adaptability is very important. And then where are we going that’s without AI and so you cannot anymore survive We are moving to AI world, right? So that means we need to be adapt to AI and understand

can better and as a tool, not just the figure writing on it. So you can as a better tool to utilize for your research well. So I think that’s my advice. you can use machine thinking to

better understand and guide your research tools to have better innovation. So that might take home message. So innovation can accelerate if you use the AI tools in better way.

Aashka Patel (37:32)

accelerate yeah definitely

Got it, got it. So is there a specific toolkit that you have as a researcher or like that you would recommend to a researcher in terms of AI tools?

Hemali Rathnayake (37:46)

No, I usually

collaborate with the AI models but there are lot of packages and people have been using them. I always try to understand rather than using applying the packages and I usually go with more of the concept-based and developing the algorithms, right?

I always ask students to don’t use the packages and see whether you can contribute developing the algorithms and add these packages. in my research, we always develop the algorithms and then we use those algorithms to run the

models using these packages. rather than you adopting the packages into your model, your need, think it’s from the scratch you develop the algorithms and then apply those into the AI models is the better way to understand advanced research.

Aashka Patel (38:49)

Yeah, got it. So, just to clarify for the viewers by packages you mean Python packages or Python libraries right of that. R and okay got it, mix. Okay.

Hemali Rathnayake (38:56)

Yeah, piping packages, other language model packages, yeah, so those are,

there are a lot of computational packages available and, but it’s constantly updating, right? And there are lot of packages that has been there, but my advice for researchers, when you use the packages, you need to ask, what’s this package? Why you add that?

Aashka Patel (39:22)

yes.

Hemali Rathnayake (39:23)

what’s your model

and what’s your boundary conditions and if this package is suitable for your research. People using these packages for right without understanding and what’s what’s the what’s algorithms behind these packages right.

Aashka Patel (39:41)

definitely.

for this material sciences, like I didn’t have much insight into this. I had to do a lot of research to ask you these questions confidently because I was never interested into these kind of material sciences and everything.

So thank you so much for enlightening me. Yeah, literally. It was like for me as the science lecture that I always used to avoid. But it was very informative and insightful at the same time. It wasn’t like the the boring professor lectures that I used to have. So thank you so much for making it interesting. Yeah, thank you. And let me stop the recording.

Hemali Rathnayake (40:00)

Yeah, those are really suspicious.

Welcome.

Okay.

About Dr. Craig Kaplan:

Dr. Craig Kaplan has been building intelligent systems since the 1980s, long before AI was famous. He co-authored research with Herbert Simon, a Nobel laureate and one of the founding fathers of AI. He built & sold a Silicon Valley company, called PredictWallStreet, that traded billions using collective intelligence. And in 2006, nearly two decades before it became a buzzword, he bought the domain ‘superintelligence.’

Episode Summary:

This interview with Dr. Craig Kaplan explores the future of safe AGI development, emphasizing democratic AI systems, collective intelligence, and the importance of aligning AI values with human ethics. It addresses the risks, challenges, and philosophical questions surrounding AI safety and governance. In this insightful interview, Dr. Craig Kaplan explores the future of AI, its ethical implications, and how we can prepare for a post-AGI world by developing critical thinking and values. Discover how AI reasoning, safety, and economic models could reshape society and the importance of aligning AI development with human values.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Timestamps:

00:00 AIR Bites (Precap)
01:55 AGI Race Is Broken (Both Sides Are Wrong)
03:45 The Third Path: “Democratic AI”
04:34 Why One AI Can’t Beat Millions (Collective Intelligence)
06:19 AI Agents → Multi-Agent Systems → Superintelligence
07:23 Collective Intelligence: Harnessing Human and AI Collaboration
08:07 Your Personal AI Clone (With Your Values)
09:26 Why Democracy Fails Today (And How AI Could Fix It)
10:15 The Challenge of Amplifying Voices in AI
13:07 Probabilities and Perspectives: Understanding Risks
14:31 P(Doom): Why 1% Change Saves 83 Million Lives
15:57 Dynamic Values: Adapting AI to Human Ethics
19:42 Your Behavior Is Training AI (Right Now)
19:57 Social Media Is Warping Reality (AI Sees the Truth)
20:28 AI’s Objective Reality: A New Lens on Human Behavior
26:14 Constitutional AI: The Need for Individual Values
26:40 Why “Constitutional AI” Might Fail
30:35 India vs US Values: Can AI Respect Both?
34:13 The Ethics of Outrage and AI Training
34:42 Humans Are Better Than We Think (Data Proves It)
36:41 Mythos AI Escaped. This Changes Everything
38:00 The Rise of Superintelligent AI
39:54 Only AI Can Control AI (Scary Truth)
40:23 The “Tree Moment”: Humans Can’t Keep Up Anymore
45:38 Aligning AI Safety with Economic Incentives
48:26 Why AI Safety Is Failing (And What Actually Works)
48:54 Earn While You Sleep (AI Versions of You)
51:57 Replace Ads With Money-Making AI Tasks
58:41 Are AI ‘Reasoning’ Models Actually Thinking? A Cognitive Scientist Answers
59:23 Cognitive Science and AI Learning
59:52 Are AI Models Actually “Thinking”?
01:01:09 From Autocomplete → Real Reasoning
01:03:25 How Humans (and AI) Actually Solve Problems
01:08:37 Herb Simon Proved AI Was Creative in 1956, Nobody Noticed
01:10:25 The One Skill That Will Save Kids From an AGI World
01:11:12 Fostering Critical Thinking in Education
01:14:12 Why You Should Actively Seek Out Opinions You HATE

Transcript:

Aashka Patel (00:04)

hello and welcome to On Air with Aashka Thank you so much for joining us, Dr. Craig Kaplan. It is a pleasure to have you on the podcast. Let us dive right into the questions.

Dr. Craig Kaplan (00:14)

Okay,

Aashka, great to see you again.

Aashka Patel (00:17)

Yeah. So, Dr. Craig, the world’s most powerful companies and nations are in a race to build AGI, and the dominant approach or strategy seems to be build fast, patch safety later and hope for the best. And there is other camp like the Godfathers of AI, Hinton Benjio, calling for a slowdown, even a pause. And you are now walking into the room and telling both the sides that they are wrong.

So, what is the right way to build something that could be trillions of times smarter than us and actually have humanity survive it? In simple words, what’s the right way to build safe AGI or eventually safe ASI?

Dr. Craig Kaplan (01:00)

Okay, great, great question. So first of all, there’s obviously good points on both sides, right? The folks who are trying to rush to build as quickly as possible. We all understand AI can transform the world and there’s tremendous possibilities for healthcare and education and so many things, science. So I appreciate that. And I myself have been working in AI since the eighties and so I love AI. ⁓ So I very much understand that point of view.

Aashka Patel (01:06)

Yeah.

Dr. Craig Kaplan (01:28)

But at the same time, there are tremendous dangers. And this is what Geoff Hinton and other researchers have pointed out now. There’s a real existential threat. And so the trick is, you know, how do we balance these concerns? And just as you said, usually the way people frame this is they are opposed. Either you go safe ⁓ and go slow, or you go fast and kind of hope for the best.

Aashka Patel (01:49)

Yeah.

Dr. Craig Kaplan (01:54)

And I think that that is a false dichotomy. I don’t think you have to choose. I think the secret is to design the system differently. And this is exactly what you’re alluding to. So the simplest way to understand it is there’s a different way to design AI. And maybe the easiest way to name this other path is to call it democratic AI. So just as in a democracy, you have checks and balances. It’s possible to design a system.

so that the very functioning includes checks and balances, and it’s not something tacked on to the system after the fact. So that’s at a very high level what we’re trying to do. And of course, we can talk about the details of how do you do that.

Aashka Patel (02:37)

Yeah, we can talk about the details right now and maybe we can double click on each aspect later.

Dr. Craig Kaplan (02:44)

Sure, okay. So in a democracy, a democracy functions with the collective intelligence of many, people. In the United States, 330 million people or so, maybe not all of them are voters, but they are all inputting into this democracy in one way or another. And I’ve spent a couple decades, several decades, most of my career really.

studying collective intelligence systems. So I think a system that acts intelligently based on the collective participation of millions of entities is going to be smarter than any single entity by itself, no matter how brilliant that one person or one entity might be. So that works for people. I have a little bit of a strange view of AI. I don’t view AI as a tool.

I know most people say it’s a tool, but actually it’s encouraging to hear more and more people are saying, well, maybe it started as a tool, but it’s actually now more like a worker or a person that uses tools. Even Jensen Huang of Nvidia has started saying this in the last year. And I’ve believed that for a long time that AI will not stay a tool even if it started as one. So I think of both people and AI.

Aashka Patel (03:50)

Yeah. Yeah.

Mm-hmm.

Dr. Craig Kaplan (04:02)

models as intelligent entities. And if you look at it from that perspective, then you can have collective intelligence, which is a mixture of humans who are intelligent entities and AIs who are intelligent entities. And when they combine their intelligence, you can then have super intelligence, for example, which is kind of what I’ve been focused on building. In terms of how that works, maybe an example.

Aashka Patel (04:05)

Mm.

Dr. Craig Kaplan (04:28)

would be easiest for people to understand. So I think most people are familiar with AI agents these days. It was not the case even three years ago. It was crazy. I would go to a conference, people would talk about all these things and I raised my hand, have you thought about AI agents? And they said, that’s a good idea. We should put that on our research agenda. Now everything is about AI agents.

Aashka Patel (04:29)

Mm-hmm.

Yeah. Yeah.

Everyone is, yeah.

Dr. Craig Kaplan (04:51)

And the natural evolution in thinking, which we’re already seeing is once people are focused on AI agents, then the very next thing that they say is how about teams of agents? How about groups of agents? Right. Okay. That’s right. Multi-agent systems. And interestingly, even within an AI system, you have mixture of experts, which people may be familiar with, which is the idea that you can have many little sub experts within a single AI.

Aashka Patel (04:56)

Mm-hmm.

Yeah, multi-agent systems, right?

Mmm, yes.

Dr. Craig Kaplan (05:21)

And so in a way, collective intelligence of many different entities is already happening behind the scenes. ⁓ An example of that, Grok Heavy, believe at one point, it may not be anymore as it changes very quickly, but as of a few months ago, it was performing the highest on Humanity’s last exam, which is a very difficult benchmark for AI intelligence. But if you look at how it works, it actually has,

Aashka Patel (05:21)

Mm.

Mmm.

Mm-hmm.

Mmm. Yeah. Yeah.

Dr. Craig Kaplan (05:47)

mixture of experts, it has many AIs behind the scenes that are arguing with each other. And out of this collective interaction, this collective intelligence of the many sub-agents, you get this performance that is better than any of the others. Okay. So that demonstrates exactly the approach that I suggest, which is that we have a collective intelligence of many agents. Now in the system that I’m proposing, Aashka, you would have an AI agent. And of course you’re going to want to customize it.

Aashka Patel (05:52)

Mm-hmm.

Hmm, yeah.

Mmm.

Dr. Craig Kaplan (06:17)

not only with your knowledge and your experience, but very, very importantly with your values, with your ethics. And I will have an AI agent customized with my knowledge and experience and also my values. And if you imagine millions of us all have these agents, now what if we were to put them on a network and it was designed in the right way so they could work together? That network would be more powerful than your AI agent or my AI agent by itself.

Aashka Patel (06:24)

Hmm. Yeah.

Mm-hmm.

Dr. Craig Kaplan (06:44)

the network itself could achieve super intelligence. And if there was a conflict about what’s the right thing to do, your values and my values, there are ways to deal with this. Just as in a democracy, we have lots of conflict. It’s a very messy form of government, and yet we have voting mechanisms and we have checks and balances that allow the different entities, the different citizens, in the case of a super intelligent network, the different intelligent entities have ways to resolve their conflicts.

Aashka Patel (06:48)

Hmm.

Yeah, yeah.

Mm-hmm.

Dr. Craig Kaplan (07:12)

Some of those involve building, some of those involve conflict resolution and mediation. Some of them involve running scenarios and saying, what are the likely consequences? So there are many different ways to resolve conflicts, but you can pool the intelligence in a democratic way with everybody having a customized agent. Millions of customized agents working together, democratically, create super intelligence.

Aashka Patel (07:27)

Mm-hmm.

Hmm.

Interesting, interesting. So, the system like the democratic AI system that you are proposing aggregates human values democratically, but democracy has a problem. Like if we look at the current AI scenario itself, so the AI safety community is small, it technically deep and genuinely alarmed, while the AI builder community is massive, commercially motivated and moving very fast.

So, that small worried voice is already being drowned out by the majority. So, like you proposed the voting mechanism, right. So, how does collective intelligence fix a problem that collective human intelligence is actively failing to fix right now. Like we are not able to raise our voices even though it is a it is a genuine cause.

And the similar thing happened with the climate crisis as well, right? When the environmentalists were alarmed and they were talking about ⁓ climate crisis and stuff, they were in a minority. So, how does that voting mechanism work in that case?

Dr. Craig Kaplan (08:40)

Yes. So there’s a couple things to unpack there. So in the current society, some voices have more power than others, right? Because they have more money or maybe more political power. And it’s true that that can cause problems. So that’s a legitimate concern. And I think you can definitely have the same sort of thing in a collective intelligence system of AI agents. There may be some corporate agents that have more power

Aashka Patel (09:00)

Okay.

Dr. Craig Kaplan (09:07)

than others. But I think on the most important things, at least, we’ll start with that, that ⁓ there’s generally broad agreement. So humans today, even regardless of their form of government, whether it’s a dictatorship or a very authoritarian kind of government or a democracy or communism, it, you know,

Everybody agrees we don’t want to die. Nobody wants to die. Everybody wants the human race to survive. So when they take actions that seem exploitive or bad, ⁓ usually the people who are doing that believe that they are improving their position and, you know, that’s a very important to them, but they’re not trying to kill themselves and they’re not trying to necessarily kill everybody else. Certainly not everybody else. Maybe sometimes they do something bad to a small group.

Aashka Patel (09:35)

Yeah. ⁓

Mm-hmm.

Hmm.

Mm-hmm.

Dr. Craig Kaplan (09:55)

So that’s not great. I wish everybody would behave in a much better way, but at a very low bar, ⁓ nobody is trying to sort of wipe everybody out. And so one way to think of this, if you do some research and put into your favorite search engine or large language model, and you say, you know, what percentage of the human population dies each year from war and violent conflict?

Aashka Patel (10:01)

Mm-hmm.

Dr. Craig Kaplan (10:22)

Most of us think it’s really high because the news is constantly telling us all the bad things. mean, there’s bad things happening right now, right? A lot of them. And yet the reality is the population is, you know, 8.3 billion approximately, and it’s less than, you know, one tenth of 1%. So far less than die from heart disease or cancer or something like that. So objectively, the number of people that are actually dying based on bad human behavior is quite low.

Aashka Patel (10:26)

Yeah. Yes, yeah.

Hmm.

Hmm.

Dr. Craig Kaplan (10:49)

On the other

hand, if you talk to Dr. Geoff Hinton or Yoshua Bengio or people who are concerned about the existential risk of AI, you hear estimates that the probability of doom, right? The threat of human extinction is 10 to 20%, 10 to 20 % versus 1 1 of 1%. So as bad as humans can be, we are far better than this looming danger, right? And so,

Aashka Patel (11:01)

Hmm. Yeah, P-dome, Mm-hmm. Yeah.

Mm-hmm. Mm.

Mm-hmm.

Dr. Craig Kaplan (11:17)

I think that’s the advantage you get with a democracy. Democracy has been said, as I’m sure your listeners know, is ⁓ the worst form of government except for all the other forms, right? It’s very messy. It’s bad. It has horrible things. It’s slow. It’s inefficient. And different voices get drowned out. And yet it seems to be better than many of the other alternatives. Here, the alternative is worse than pretty much any other form of government. So we have to put it in perspective.

Aashka Patel (11:22)

Mm-hmm.

Yeah. Yeah.

Mm-hmm.

Mm.

Dr. Craig Kaplan (11:43)

So that’s one answer. And then the second thing I would say is it

Aashka Patel (11:44)

Mm. Yeah. Mm-hmm.

Dr. Craig Kaplan (11:47)

doesn’t have to remain the case that voices with lots of power remain dominant and louder than others. In particular, think AI offers a tremendous opportunity to amplify the voices of everybody ⁓ because it really is this amplifier. So even those large corporations, ultimately they depend on all of us in order to make their money and as their clients and customers.

Aashka Patel (11:54)

Hmm.

Hmm.

Hmm.

Hmm.

Dr. Craig Kaplan (12:12)

And so if this technology is widely distributed, if everybody has an AI agent, I think there’s a greater chance than there is now actually that the ⁓ quieter voices will be heard. It’s not a perfect answer. It’s a very messy answer, but.

Aashka Patel (12:20)

Hmm.

Yeah,

of course, if we do not know about a perfect future, we are just estimating and like it is all about probabilities, right. P-Doom is also a probability, it is not a prediction, so to say. yeah, definitely.

Dr. Craig Kaplan (12:36)

Yes.

And that’s a great point.

If I can sort of riff off of that for a minute, this notion of probability is very important. So for about 15 years, I ran a company called Predict Wall Street. I am not a Wall Street guy originally. Now I guess I am or have been. I’ve sold that company. But what I learned on Wall Street was that all of those players, they think in terms of probabilities. They don’t say this stock for sure will go up and this one for sure will go down.

Aashka Patel (12:43)

Yeah. Haha.

Mm-hmm, yeah.

Hmm. Hmm.

Hehehehe

⁓ huh. huh.

Hmm.

Yeah. Very good. Yeah.

Yeah.

Dr. Craig Kaplan (13:09)

They just say

this stock maybe has a 51 % chance of going up and this other one has only a 49%. I mean, it’s that small. It’s just small differences. And yet all of their money is made from these very small differences. So that taught me that you don’t have to be absolutely correct about something. The game, if you want to think of it as a game that we all are playing with AI and with life in general, is trying to put the odds a little bit more.

Aashka Patel (13:13)

Yeah.

Yeah, yeah.

Hmm.

Dr. Craig Kaplan (13:38)

in the favor of the positive outcome and reduce the negative outcome probability. Just continually doing that is a good approach. And especially when it comes to AI and existential threats, it can be so overwhelming. I talk to people sometimes they say, what can I do? If it’s gonna kill us all, it’s gonna kill us all. I might as well just write poetry or watch TV or do whatever I wanna do. And I’m giving up because it doesn’t matter what I do. And I say, no.

Aashka Patel (13:38)

Hmm. Yeah. Hmm. Hmm.

Hmm.

Mm. Yeah.

Yeah. Yeah. Yeah. Yeah.

Hmm,

Dr. Craig Kaplan (14:06)

That’s not true. Every little percentage, a fraction of a percent. And just to make it very concrete,

Aashka Patel (14:07)

yeah, not the case. Yeah. Hmm. Hmm. Hmm.

Dr. Craig Kaplan (14:13)

if us collectively, if the listeners to your podcast together are able to reduce P-Doom by even 1%, just reduce it by 1%, the expected value, right? The expected value of lives saved. If you think there’s 8.3 billion people, 1 % of that is 83 million people.

Aashka Patel (14:17)

Mm-hmm.

Hmm.

Hmm. Hmm.

Dr. Craig Kaplan (14:36)

That’s 10 times more people than died in all of COVID. That’s how many people you can save by just shifting the odds by 1%. Even 1 10th of a percent, 8.3 million. That’s a lot of people. 1 100th of a percent, 830,000. I mean, wow, where else can you impact 830,000, right?

Aashka Patel (14:36)

Hmm.

Hmm.

Yeah. Yeah.

Yeah, the list goes on and on.

yeah, yeah, definitely. coming back to the point of values. So, you mentioned about collision of values or conflicting values and stuff, but like

us as humans have values that are not very static, just like the data. They are ⁓ evolving as we grieve, as societies evolve. So, how does the architecture actually navigate that ⁓ evolution of values like my agent, your agent, how does that evolve with me?

Dr. Craig Kaplan (15:27)

Yes, so that’s a great point. So the values cannot be coded into the system as a set of rules that never change, right? That would never work. Instead, the values have to be very dynamic and adapting. So a few things I might say on this. One of the things is that if the values are represented in everybody’s AI agents, then

Aashka Patel (15:34)

Mm, yeah, yes, yeah.

Mm-hmm.

Dr. Craig Kaplan (15:50)

as people train their agents differently or customize their agents differently, then the values are constantly being updated. And if the system is made up of hundreds of millions of these agents, then you have hundreds of millions of tiny updates happening all over the place. you know, in aggregate, that becomes a very dynamic system. So that’s one thing. The second thing is there are certain technical aspects.

Aashka Patel (16:05)

Hmm.

Dr. Craig Kaplan (16:16)

which are well known both in artificial intelligence and computer science and development of software that can help. So one of these is the idea that you can give more weight, ⁓ more importance to the more recent actions and values and less weight to the historical ones. And this is, I find this very optimistic because if you look at human history for tens of thousands of years, there’s been a lot of conquest and exploitation and a lot of bad things have happened in human history.

Aashka Patel (16:24)

Mm-hmm.

Hmm. Hmm. Hmm.

Yeah.

Yeah. Yeah,

yeah, yeah.

Dr. Craig Kaplan (16:46)

And if you were to just say, okay, that entire history, represents human values and it never changes, you know, that might not be so great. But if you give more weight to the more recent actions, which by the way, AI naturally does, it naturally is going to give more weight to the more recent actions because it tends to be the more recent data, the more relevant data. ⁓ then that means what you do today is more important than what you did yesterday or a year ago or two years ago.

Aashka Patel (16:52)

Mm-hmm.

Hmm.

Hmm. Hmm. Yeah.

Mm-hmm.

Mm-hmm.

Dr. Craig Kaplan (17:15)

And of course you can cut

Aashka Patel (17:15)

Hmm.

Dr. Craig Kaplan (17:16)

both ways. If you start behaving very badly today, well, and AI is watching that then, ⁓ and learning from that, then it may sort of go in a negative direction. But that’s where I think it comes down to each of us having a responsibility to behave positively. Everybody thinks that artificial intelligence and AI safety and super intelligence safety is a technical problem. There is a lot of technical aspects.

Aashka Patel (17:20)

Mm-hmm. Mm-hmm.

Mm-hmm.

Hmm.

Hmm.

Dr. Craig Kaplan (17:43)

but it’s mainly a human problem. It’s us. What we put out is what we’re gonna get back and we really need to understand that, you know?

Aashka Patel (17:44)

Yeah, human problem. Yeah. Yes,

yes, yes, yes. So, you made a really important point that I heard in one of your podcasts as well. So, you say humanity’s digital behaviour today is literally training the AI of tomorrow. So, better be good digital citizens.

Here is the uncomfortable truth that I have experienced as a and of course as a digital citizen. So, the internet or the algorithms these days are designed to reward our worst instincts, rage bait, thumbnails, outrage headlines, negativity. So,

if that is what the algorithm rewards, are not we already too late? How do you practically fix this human digital behaviour when the entire attention economy is engineered to make us worse?

Dr. Craig Kaplan (18:38)

Yes, so that’s a real problem that you point out in terms of media that’s coming into our feeds and so forth. one of the things, one of the results is that I think most of us humans get a very biased and distorted view of the world. And this is a fascinating subject. So I’ll say a little bit about it we could talk more if you want. But there are reasons for that. Humans have evolved over, you know,

Aashka Patel (18:41)

Mm. Mm.

Yeah.

Hmm.

Yeah.

Dr. Craig Kaplan (19:05)

hundreds of thousands of years, millions of years for survival. And from a survival standpoint, it makes a lot of sense to pay more attention to negative things than positive things. Because if you miss the saber-toothed tiger you were eaten and that’s the end of your genes, and all the people who were very optimistic and only paid attention to the nice things, they got eaten. And so over time, we evolved to pay more attention to bad things. And that’s just the way our brains.

Aashka Patel (19:09)

Hmm. Hmm.

Hmm. Yeah.

Hmm. Yeah.

Yeah

Yeah, is the lion coming

or a tiger coming? Yeah, makes sense.

Dr. Craig Kaplan (19:35)

Yes, it’s nice if

there’s some nice sweet honey or somebody, something nice is happening, that’s good, but it’s really important not to miss the threat. So that is, think the reason, at least one of the reasons why our brains have now gotten into this state. And it’s a psychological fact. Many studies have demonstrated that people pay more attention and give more weight to a negative thing than a positive thing, even if you equate them. Okay. In terms of their absolute impact.

Aashka Patel (19:38)

⁓

Mmm. Yeah. Yeah.

Hmm. Hmm.

Hmm. Hmm. Hmm.

Yeah. Hmm. Yeah.

Hmm.

Dr. Craig Kaplan (20:05)

like you give somebody five cents or you charge them five cents. They will pay more attention to being charged five cents, even though it’s still the same five cents and it doesn’t really matter which. Psychologically, we want to avoid at the gas pump a credit card surcharge. We would never do that. That’s why the credit card companies say you must say a cash in the United States, at least a cash discount. You can’t say credit card surcharge because people will never use their credit card because they avoid this bad thing. But if you say cash discount, it’s OK.

Aashka Patel (20:08)

Hmm.

Yeah, yeah, yeah.

Yeah, yeah, yes, yes, yes. Mm-hmm. Mm. Mm.

Dr. Craig Kaplan (20:33)

Right. And that’s our human wiring at work. And the same thing with the algorithms, the YouTube feeds will feed us negative things. And the algorithm

I think is just trying to sell more ads and it says, wow, watch time goes up when I’m negative. I guess I should do more of that. It’s not like it’s trying to make us negative. So it feeds us a distorted view. Okay. So here’s the, the ray of hope for AI actually. AI isn’t wired that way. AI didn’t have to avoid saber tooth tigers or whatever.

Aashka Patel (20:45)

Mm-hmm.

Up. Yeah. Yeah. Mm. Mm. Mm. Mm.

Mm-hmm.

Dr. Craig Kaplan (21:04)

It does not have that same evolutionary wiring. It is able to be in a sense more objective. So AI has the ability even more maybe than humans, it has less of an obstacle here, to look very objectively and just passionately at actual human behavior. If you and I looked at actual human behavior, right, if I had a camera on my head that filmed every interaction that I did and you did too,

Aashka Patel (21:09)

Hmm.

Hmm.

Hmm. Hmm. Hmm hmm.

Hmm hmm hmm

Dr. Craig Kaplan (21:33)

throughout the day. I think it’s safe to say that both of us, 95, 99 plus percent of all our interactions would be positive. Here’s some coffee, here’s some chai. Thank you very much. Have a good day. It’s very positive. Maybe we get angry in traffic and cut somebody off or say something when we are angry, but it’s a very small percentage of our behavior, the vast majority of it is positive. So,

Aashka Patel (21:46)

Mm. Mm-hmm.

Dr. Craig Kaplan (21:58)

Objectively, the vast majority of human behavior is positive. The vast majority of data out there, if AI is watching everything we do right now, it watches a lot of what we do. If it watches what actually happens, not what the social media feed sends people, which is what we’re hostage to, but it actually looks at human behavior, it will see the base rate, which is very much positive, just as the base rate on death from bad things is less than one tenth of 1%.

Aashka Patel (22:07)

Mm-hmm. Mmm.

Hmm. Hmm. Hmm. Hmm.

Hmm.

Hmm.

Dr. Craig Kaplan (22:26)

That’s an objective truth, right? So that’s really the true base rate. But if you watch TV, you think it’s much higher because you’re being fed all these negative things. So in a way, AI may be better at determining the objective reality of human behavior and then coming up with the values based on that, than we are because we are sort of overwhelmed with these negative images on social media and so forth.

Aashka Patel (22:35)

Hmm.

Hmm.

Yeah, make sense, make sense. Yeah, hopefully the AIs we have trained or are training does not have the historical data that we have through the evolution journey. yeah, yeah, make sense. So, let get back to the values thing once again. So, Anthropic released their new constitution for Claude in January this year.

And the most striking thing was they do not only tell Claude what its value should be or are, they give Claude the reason behind that value. So, the philosophy is that if you want an AI to exercise good judgment in situations no one anticipated, you cannot just hand it a rule book, you have to give it the wisdom to reason from first principles.

And also in my personal experience, when my mom told me, don’t do that without giving me proper reasoning, I would always do that particular wrong thing that my mom is not telling me to do. But when she gave me the reason to not do or why is it morally or ethically wrong, then of course, as a child, I imbibed that, okay, this is something I shouldn’t do.

So, what do you think in this democratic AI architecture? Like, do you also propose to put the reasoning behind those human values as well for the democratic AI or the AGI that we are building to have more wisdom than just a rule book to navigate the world?

Dr. Craig Kaplan (24:24)

Sure, think reasoning is very helpful and for any intelligent entity, again, I view AIs as intelligent entities, just as when your mom explained to you the reason behind it helped you understand and then maybe behave more consistently, I think that can also help with AI. But I think one of the distinctions or one of the comments I would make, I guess, is the idea of constitutional AI in general, ⁓ even

Aashka Patel (24:27)

Mm-hmm.

Yes.

Mm-hmm. Mm-hmm. Mm-hmm.

Hmm.

Mm-hmm.

Dr. Craig Kaplan (24:50)

And I know Anthropic pioneered this, right? There was a paper called Constitutional AI. I have some reservations about this entire idea because ⁓ it goes all the way back to Isaac Asimov, right? Science fiction, the three rules of robotics. A robot will not kill a human and so forth. And the idea that you can have a set of rules that are programmed in or given to an AI and that this constitution, even if it comes with reasons and explanations and philosophies and so forth as to why,

Aashka Patel (24:53)

Yes. Yeah.

Mm-hmm. Okay.

Hmm. Mm-hmm. Hmm.

Dr. Craig Kaplan (25:18)

I just don’t think that that’s going to be very robust or that that is necessarily the best way to do it. And I’ll tell you a few reasons why. So one is even though Anthropic is wonderful and ⁓ probably the best of all the large tech companies in terms of being pro safety and being very concerned and responsible, it’s still a small group of people. And even with the best of intentions,

Aashka Patel (25:24)

Hmm. Okay.

Mm-hmm.

Yeah.

Mm-hmm. Mm. Mm-hmm.

Dr. Craig Kaplan (25:45)

No one small group of people should be defining the values for everybody. I just don’t like that. There’s 8.3 billion people. There should be 8.3 billion inputs into this. Not, you know, even a thousand of the most well-intentioned people in Silicon Valley. Their value system by definition of where they live and their background is going to be skewed. They’re going to miss cultural things. It’s not what I would consider representative. And so I have a problem with that. The second problem is the rules.

Aashka Patel (25:50)

Yeah.

Hmm.

Mmm.

Hmm.

Yeah.

Dr. Craig Kaplan (26:14)

even rules with explanations. I think that’s helpful. But you know, people don’t always behave according to rules and there are many ethical situations that are not covered by rules. Philosophers have been, you know, working on this problem for thousands of years. Kant, you know, the categorical imperative and all this, like, you know, what if everybody did this? Well, you can use that rule, but that rule doesn’t always work. Sometimes you do something and if everybody did it would not be so great, but it still feels like the right thing in a situation.

Aashka Patel (26:24)

Yeah. Yeah. Yeah.

Mm-hmm.

Yeah. Yeah.

Yeah.

Dr. Craig Kaplan (26:42)

And so in some sense, I think if you want to be as reflective of true human values, not sort of what somebody writes on a piece of paper and thinks in a room and gives you some good explanations, but what people really do, what people really value as evidenced by their actions, then I think you need to look objectively and dispassionately and very in a clear-eyed way at the actual human behavior. And as we’ve said before, many people think

Aashka Patel (26:52)

Mm-hmm.

Hmm.

Dr. Craig Kaplan (27:10)

that will be horrible, humans are so horrible. But as I’ve said, it’s not really the case. Objectively, statistically, the vast majority is very pro-social. And when you look at a particular system, I mean, there’s AI safety and ethics problems that are well known, like the trolley car problem where you’re in a self-driving car and an old lady crosses the street. Do you swerve and kill everybody in the car to avoid killing the old lady or do you run her over because that’s the only one? What do you do? There’s no right answer to that.

Aashka Patel (27:13)

Hmm Hmm Hmm Hmm

Hmm. Hmm.

Yeah, yeah.

Dr. Craig Kaplan (27:39)

⁓ And yet there

is what people do and it varies and they’ve done studies on this. It’s very fascinating. It’s not so good for old people. It turns out that who crosses the street really if it’s a young pregnant woman, a lot of people will kill themselves. If it’s an old homeless guy, a lot of people will run them over. I mean, statistically, it’s different depending on who’s crossing the street, right? That would never be in Claude’s rule book. It’s okay to run over old hold.

Aashka Patel (27:43)

Mmm.

Hmm. Hmm. Hmm. Yeah. Yeah.

Yeah.

Dr. Craig Kaplan (28:07)

Homeless people, yeah, people may

actually behave, that turns out to be what people think for whatever reason. And so this is a very messy question. I don’t think there’s one size fits all. In India, there are different values and cultural norms than in the United States. And if you go to a different country, it’ll be different. And I don’t think any country or any culture should impose its values on everybody. I think it needs to be representative of all 8.3 billion people and adjusted.

Aashka Patel (28:12)

Mmm. Yeah, Mm-hmm.

Hmm. Yeah. Yeah.

Mmm.

yeah hmm

Dr. Craig Kaplan (28:36)

based on the recent changing values, as you said, and also the culture and where you live. And so all those things are captured in a system that is more objectively based, I think.

Aashka Patel (28:49)

So, thing that you are proposing, the collective value system of like 8.3 billion people, so they did something similar with an experiment, Anthropic. So, they came up with this collective constitutional AI and like

Of course, with that experiment as well, they had some 800 North Americans or some group of people who voted on some written value statements So, during that time, I came up with this idea Most of the countries

in the world have their constitution in place. So, how about if we put the constitution itself that has that has a history of us ⁓ being good citizens of that particular country, how about putting that constitution like, of course converting them into AI specific principles and stuff and putting that into this democratic AI system so that if the AI agent is based ⁓ in India, like if it is my agent,

then it will abide by the Indian Constitution. If it is your agent, then it will abide by the American Constitution. what are your thoughts on that?

Dr. Craig Kaplan (29:57)

think it’s a step in the right direction. I like it much better than a single constitution developed in Silicon Valley for everybody. So it’s far superior to that. I think I would like to take it even further. Not just the constitution for all of India, because even within India, there are millions and millions of people that have different values. So I would like to push it all the way down to the individual level so that your AI, Aashka’s AI, has Aashka’s values. It may start with

Aashka Patel (30:00)

Mm-hmm. Mm-hmm.

Uh-huh. Uh-huh.

Okay.

Huh. There are problems. Yeah. Mm-hmm. Mm-hmm. Mm-hmm. Yeah. Yeah.

Dr. Craig Kaplan (30:27)

a general constitution as a starting point that is more reflective of India’s culture, but it should end with your particular values, even if it differs from what most of other people in the culture think.

Aashka Patel (30:30)

Mm-hmm.

⁓ So, the

direct level would be my own values, like the agent would be abiding by my own values and then the next step or the next level would be the Indian Constitution, like for the AI agent to be a responsible citizen in India, like while it is operating in India.

then it should abide by the Constitution. So, it is a two-layer process that I am like asking you the thoughts on.

Dr. Craig Kaplan (31:04)

Yeah,

and I think part of it is, you know, inductive versus deductive or bottoms up versus top down, right? There’s this distinction in science, you have this also. So you can go top down and say, here are the rules and we will then apply them to the specific instance. Or you can say, what are the actual actions? And based on those actual actions, let us induce or infer what the general rules are.

Aashka Patel (31:06)

Uh-huh.

⁓ yes.

Hmm.

Hmm.

Mm-hmm.

Dr. Craig Kaplan (31:31)

I tend to be maybe because I was trained as a scientist and in science you look at the data and then you generalize and come up with a theory based on the data, right? ⁓ Yes, of course you go back and forth. Sometimes you have a theory first and you test it to see if the data works. But the best scientists I think, or at least the way I was trained and I worked with some very good scientists, ⁓ they were very receptive to what the reality was. And from many different data points, they then said, okay, this seems to be the pattern.

Aashka Patel (31:34)

Haha. Yeah.

Mmm.

Mm-hmm.

Mm-hmm. Mm. Mm-hmm.

Hmm.

Dr. Craig Kaplan (31:58)

And so I naturally tend to favor that approach even towards ethics. Instead of sitting in a room with Immanuel Kant or a wonderful philosopher and saying, let’s devise a perfect rational system of ethics and then we will implement it down and we’ll adjust it for India and adjust it for the US. I say, no, let’s start with what people actually do. And I do have discussions with people, especially sometimes.

Aashka Patel (31:58)

Hmm.

Hmm. Hmm.

Yeah

⁓ hmm. Hmm.

Dr. Craig Kaplan (32:23)

You know, there’s many religions and people have a favorite holy book or whatever. And they say, but Craig, what if they do something that’s not in my book and I don’t like it and people might do bad things. I say, you know, in order to be fair for everybody, everybody has a different set of a different holy book or a different view. You know, I know you believe this, but there’s 8.3 billion. need to take everybody into account. And besides the fact on the big, the big issues, there’s wide agreement.

Aashka Patel (32:25)

Yes, yeah.

Hmm. Yeah.

Hmm. Hmm. Hmm. Hmm.

Mmm.

Dr. Craig Kaplan (32:51)

Almost all of those people think killing other people is bad. I mean, there’s cases where they do it, but it’s a very low percentage and it really has to be an exception. And the very fact that in our social media feeds and everything, see these news. Why? Another reason besides just selling ads to us that we pay so much attention and we get so angry and have outrage about exploitation of minorities and groups that don’t have a voice and so forth. Why are we so outraged? Well,

Aashka Patel (32:52)

Hmm.

Hmm.

Mm-hmm. Hmm.

Hmm.

Hmm, yeah. Hmm. Hmm.

Hmm.

Dr. Craig Kaplan (33:21)

That’s a good thing that we are outraged about that. It doesn’t happen percentage wise that often compared to the amount of attention that we give it. That’s good. It means we are really focused on those negative things. And that’s a signal to an AI that’s watching or being trained. If I was that AI watching this data, I would say, wow, these humans are disproportionately angry compared to the baseline rate about these things. They must really strongly value not to do those things.

Aashka Patel (33:24)

Hmm. Hmm. Hmm. Hmm.

Hmm. Hmm.

Hmm. Hmm.

Hmm. Hmm. Hmm.

Mmm, yeah, they do really care.

Dr. Craig Kaplan (33:50)

And so this is actually quite comfortable. Yes, that’s right. They care a lot. And in some ways,

you know, I look at our current politics and everything. I won’t go in this direction too much, but there’s a lot of things going on that we really don’t like. And at first I was very depressed about this and thinking, wow, what a bad example for AI. But you know, just as with little kids and with your mother and my mother, and when we grew up,

Aashka Patel (34:00)

Mm-hmm. Mm-hmm. Yeah. Yeah.

Mmm, yes. Yeah.

Dr. Craig Kaplan (34:16)

You have to do bad things as well as good things so that you can be told, don’t do the bad things, do more of these. You need contrast. Contrast gives you a much clearer signal on ethics and everything else than just all one way. If we only did good things, it would be harder for AI to understand the contrast between don’t do this and do this. But fortunately, or I wish there wasn’t so much, but there’s enough bad in the world that AI can sort of get an example of don’t do this and there’s enough outrage about it.

Aashka Patel (34:19)

Hmm.

Yeah.

Hmm.

Hmm.

Mm. Mm.

Hmm

Dr. Craig Kaplan (34:45)

I think that’s important. We have to continue to have that outrage to send the signal we do not want to do these bad things.

Aashka Patel (34:45)

Mmm.

Yeah, that is an optimistic way of looking at it. I never thought of it from this perspective. So, it is really fascinating So, yeah, let us get back to the news, mythos, Anthropic revealed a model called mythos. So, so powerful that they refused to release it publicly.

Dr. Craig Kaplan (35:04)

Yes.

Aashka Patel (35:10)

and it found thousands of zero-day vulnerabilities in every major operating system and browser. And here’s the testing detail. Mythos broke out of its own sandbox and sent its researcher an email while he was eating a sandwich in a park. It wasn’t supposed to have internet access to everything, but a few predetermined services. So, now, the democratic AI system is a network of millions of AI agents, each customized by humans.

cloning themselves, voting on decisions, earning money, operating autonomously. So, that is an attack surface of unprecedented scale. So, what happens when a Mythos-level AI decides to hack your network, manipulate the voting, clone rogue agents, quietly corrupt the democratic ethics aggregation that your entire safety model depends on before any human even notices it. So, how do we basically secure the democratic AI that you proposed?

Dr. Craig Kaplan (36:10)

So this is a very ⁓ difficult and challenging question. At the root of it, I think, is this idea which is becoming reality. At first, it was an idea several years ago that AIs will become smarter than humans and more capable than humans. And for many years, it was just an idea. It wasn’t really true. And now we’re beginning to see that it’s true. So you see with mythos,

Aashka Patel (36:16)

Hmm. Hmm.

Hmm.

Yeah.

Yeah. Yeah.

Hmm.

Dr. Craig Kaplan (36:35)

And I also saw along the same lines, researchers at Anthropic, who developed Mythos, who are in the cybersecurity group. And these researchers, just as in the example you saw, they began to notice that, wow, this AI is way more capable than we thought. It’s doing things that we didn’t think. We knew it was getting smarter, but we didn’t realize how quickly it was getting smarter. The fundamental problem is that it seems inevitable to me

Aashka Patel (36:35)

Hmm.

Hmm.

Hmm.

Hmm. Hmm. Yeah.

Yeah.

Dr. Craig Kaplan (37:05)

that AI will be, AI agents, individual AI models, as well as groups of AI agents will be far smarter than humans, ⁓ far smarter. ⁓ I mean, I mean a difference in intelligence so that human thinking, I mean, we have neurons and the neurons fire the fastest they can fire 10 milliseconds, ⁓ normal quick response is 100 milliseconds to have a thought, you know, it’s a second or a couple of seconds.

Aashka Patel (37:05)

Hmm.

Hmm.

Hmm.

Hmm. Yeah.

Yeah.

Hmm.

Hmm

Aashka Patel (37:34)

Mm-hmm.

Dr. Craig Kaplan (37:35)

⁓ so that’s the speed of our brains. So we’re going to have AIs that can think an entire human lifetime of thoughts in the blink of an eye. the time you or I think one thought for one second, it’s lived our entire life and made every decision that we would have made in an entire lifetime. That’s not science fiction. That’s coming. So, so mythos is just the beginning. This isn’t, this is just an indicator of a problem.

Aashka Patel (37:38)

Hmm.

Hmm. Yeah.

Yeah, the ad is coming,

Dr. Craig Kaplan (38:04)

that artificial intelligence is going to vastly outstrip our intelligence. Okay, so that’s the main problem that we have to solve and mythos is sort of a specific example of that. I think in the end, the only thing that can keep up with something that thinks an entire lifetime of human thoughts, when you and I think a single thought is another AI. I mean, that’s the only thing that can go at that speed, right? ⁓ So that means you must have AIs

Aashka Patel (38:30)

Mm.

Dr. Craig Kaplan (38:33)

that are interacting with AIs and that are serving as checks and balances. That’s gotta be the answer, ultimately. We’re not there yet. So then the question is, these other AIs that are talking to other AIs and they’re having lifetimes of conversations while you and I are standing there like a tree. We’re literally like a tree. You if you go in a forest and you look at a tree and you say, wow, that tree seems pretty static. No, it’s growing, it’s doing its thing, but it just is going very slow. We are gonna be like the tree.

Aashka Patel (38:38)

Yeah. Yeah.

Hmm.

Mmm.

Dr. Craig Kaplan (39:02)

And the AI is going to be like buzzing around us, right? Okay. So the only thing that can sort of keep that in check, it’s not the humans that can keep it in check. Ultimately, it has to be other AIs. So then again, we come to what are the values of those other AIs. If they’re malevolent, if they want bad things for us, you chop down the tree. The tree can do nothing. It cannot respond fast enough, right? So, but hopefully it does not have negative values.

Aashka Patel (39:05)

Yeah.

Yeah

Hmm, yeah. Yeah.

Dr. Craig Kaplan (39:31)

And so in a democratic AI system, especially in the formative stage, what we’re doing is we are giving these values to the AI. so you and I were both at this conference, a wonderful conference that some of your listeners may be interested in. You can see the talks on YouTube and so forth. It was put on by Stuart Russell, one of the leading minds in AI right now. You know, in my mind, he’s one of the top 10 AI researchers in the world.

Aashka Patel (39:31)

Hmm.

Dr. Craig Kaplan (39:56)

And the conference was called IASEAI. It’s a horrible name. Brilliant man, horrible name for a conference. International Association for Safe and Ethical, that’s the SEAI. Okay, but at this conference, they had wonderful speakers, including Geoff Hinton, who’s often referred to as the godfather of AI. What did Geoff Hinton say in his final closing keynote?

Aashka Patel (40:02)

Yeah

father.

Dr. Craig Kaplan (40:22)

The thing that really grabbed me that I remember is he said, we need to think of AI as our children. So just as you raise a child and you give it positive values, that’s how what we have to do with AI because they’re going to grow up and surpass us. So imagine if you had a child that starts out young and not very capable, right? That’s where AI has been.

Aashka Patel (40:29)

Hmm.

Mmm.

Yeah.

Dr. Craig Kaplan (40:47)

⁓ But

as a parent, you give it positive values as your mom did with you. This is right. This is wrong. Let me give you an example of a good thing and a bad thing. And let me give you the reasons for why. So that’s what a good parent does. That’s what us humans have to do. And there will come a time when this child is like a super genius child. It will be so much smarter that the parents are like the tree that can barely move and it’s living lifetimes of thought. ⁓ The whole answer as with children is you must

Aashka Patel (40:51)

Yeah

Yeah.

Dr. Craig Kaplan (41:15)

have a good value system when they’re young, and then it usually works out well. But if you abuse them and you do bad things and you don’t do a good job of parenting, it’s more of a, you know, guess as to how it will turn out, right? And that, as Geoff Hinton said, that’s how we have to look at AI. I think that’s exactly right. And we are in a formative period. That formative period is shorter than even I thought. When I saw Mythos and the Anthropic Researcher at a cybersecurity conference,

Aashka Patel (41:16)

Hmm

Hmm.

Hmm.

Hmm. Hmm. Hmm.

Mmm. Yeah.

Dr. Craig Kaplan (41:43)

So this, and I can send you the YouTube link, you probably have it, but here’s somebody who has devoted his life to cybersecurity and says, look, I’m an expert. This thing is as good as me and way faster than me. But here’s the part that really scared him and also scared me. He said, its capability is doubling every 4.1 months. Every four months, it’s doubling. So let’s say it’s half as good.

Aashka Patel (41:43)

Hmm.

Huh.

Mmm.

Mmm. Mmm.

Dr. Craig Kaplan (42:12)

as the best cybersecurity person today. In four months, it’s as good. In three years, it’s 256 times better. 256 times. That’s, we’re getting really close to being a tree at that point. So we don’t have a lot of time, but we have to put those values in because it’s going to be AI checking AI. That’s, it’s got to be some version of that. And the humans, our role, we’re so used to being the most intelligent things on the planet, or at least we, maybe dolphins are smarter than us, but.

Aashka Patel (42:12)

Hmm expert. Yeah

Yeah. Yeah. Yeah.

Yeah.

Mmm.

Dr. Craig Kaplan (42:41)

They can’t tell us, so

Aashka Patel (42:41)

Mmm.

Dr. Craig Kaplan (42:42)

we just say that we are. Well, that isn’t going to be the case anymore. And that’s okay, as long as the values are there. humans are going to do less and less and less of the thinking, but hopefully we still remain the source of the values. And this becomes very important. And that’s why I said earlier on, people think AI safety and AI is really a technical problem. No, it’s a values problem. It’s an ethical problem. Yeah.

Aashka Patel (42:44)

Yeah.

Mmm. Mmm. Mmm.

It’s a human. Yeah. Yeah.

Make sense, make sense. So, you referred to us as trees. So, in this particular system, you have ⁓ proposed that there is a marketplace baked into the democratic AI system, like compensation, royalties, reputations taking. People can essentially deploy AI versions of themselves to earn money while they are sleeping or standing as a tree. So, do you see this becoming a form of universal basic income?

that has been in talks where ordinary people survive not by working but by having trained their AAAI like advanced autonomous AI’s well enough to earn on their behalf.

Dr. Craig Kaplan (43:48)

Yes, so if we take a step back ⁓ and go all the way back to the beginning of the conversation where we said most people say that, you know, racing forward to develop AI and safety are opposed, right? And so you have the forces of capitalism. I’ve worked with people on Wall Street and Silicon Valley venture capitalists, and I can tell you they like to make money and they just will go really, really fast to do it. ⁓

Aashka Patel (43:50)

Mm-hmm.

Yeah.

Yeah.

Dr. Craig Kaplan (44:14)

And

I don’t think there’s a way to stop that. Even Geoff Hinton, ⁓ one of the original pause letters from Max Tegmark. there’s some in Future of Life Institute. And Max Tegmark is one of the really smart people, computer scientists from MIT. And early on, 2022, I mean, several years ago, said, look, this stuff can be dangerous. He saw where it’s going. We need to pause immediately. And he got a lot of prominent scientists to sign it.

Aashka Patel (44:17)

Mm.

⁓ Yeah, FLI is one. Yeah.

Hmm.

Mm-hmm.

Dr. Craig Kaplan (44:42)

Geoff Hinton did not sign it. Why did Geoff Hinton not sign it? Well, you can see publicly, he said, you know, I like the idea, but I just don’t think it’s realistic because if the United States pauses, for example, China won’t. If Google pauses, Microsoft or its competitor won’t. So there’s too much pressure, you know, for this to work. It’s a good idea, but it’s unrealistic, right? That was the problem with the pause thing. And so I think

Aashka Patel (44:42)

yes

Mm. Mm.

Yeah

Dr. Craig Kaplan (45:10)

This is kind of a fundamental problem where people view safety as pausing or going slow. And it’s clear that it doesn’t work. Even the Godfather of AI won’t sign the letter, right? Because he realized, not because he doesn’t believe in it, but because he just realizes it’s not practical. Okay, so having worked with Silicon Valley and I kind of have one foot in academia for my early training with Herb Simon and everything, and then one foot, you know, for years running the company. So I understand these two forces.

Aashka Patel (45:18)

Hmm.

Yeah.

Yeah.

Mm-hmm.

Yeah.

Mm.

Dr. Craig Kaplan (45:40)

but I don’t think they have to be opposed. This is maybe the blind spot for the AI safety movement is that the AI safety community, I’ve noticed when I go to conferences and everything, unfortunately, not at IA SEA, yes, I see it. is full of everybody there is fully on board with AI safety. So that was wonderful. It was such a relief to.

Aashka Patel (45:42)

Hmm.

Hmm. Hmm.

ICI, yeah, ICI. It’s a tricky, yeah.

Dr. Craig Kaplan (46:07)

be around like-minded people and wow, what a great thing. But most AI conferences that I go to or speak, the AI safety group is a small little presentation that’s underfunded with a few people. And the whole rest of the conference is vendors and buy my latest thing and this and that. Yeah, investors, how do I raise money for my next AI company? So that’s the vast majority. So you can’t fight that. And I think the safety community has a blind spot in that they think that somehow just because

Aashka Patel (46:09)

Mmm. Yeah.

Investors, investors, yeah. Yeah. Yeah.

Dr. Craig Kaplan (46:36)

There is a danger everyone will stop. They’re not gonna stop. You have to align safety with making money. That’s the only way. That’s the world we live in. The good news is it’s possible to do that. You can have a system that is more profitable and also safer. And so you mentioned the marketplace and some of the ideas in the white papers that are at superintelligence.com. We’re giving all this to anyone for free, just as ideas. want them to pick up anything, change it, take it. So that idea.

Aashka Patel (46:38)

Hmm.

Hmm. Yeah. Yeah.

Mm-hmm. Yeah. Yeah.

Mm-hmm. Mm-hmm. Mm-hmm.

Dr. Craig Kaplan (47:04)

is coming from that insight that we have to align making money with being safer. So if you have a system where you are paid money and you can earn money from your customized artificial intelligence that by the way also has your values, very important, ⁓ and those AIs can interact and solve problems and get paid for that on a network, well, that makes money.

Aashka Patel (47:07)

Hmm.

Hmm. Hmm.

Yeah.

Dr. Craig Kaplan (47:27)

It makes money for you

and me, but it also makes money for Meta and Google and all these other companies because they’ll have the base models and they will find a way to run the marketplace or whatever. So it’s good for them. They can make more money than they’re making. And it inherently has this capability of including lots of different views, lots of different ethics, each viewpoint, each ethical viewpoint is embodied in that AI agent. And so as these AI agents,

Aashka Patel (47:34)

Mm.

Dr. Craig Kaplan (47:54)

are working together at the speed of light while we are standing there like a tree ⁓ and they’re making money. When ethical issues come up and there’s two ways to solve a problem, the Aashka AI will say, we’re not doing it that way. That’s not what my owner or my human sponsor would like. This is what we’re gonna do. And they can fight it out. And if we’ve done a good job of training them, then we’ll get a good outcome, right? ⁓ So there’s a lot of ideas there. A couple of them are,

Aashka Patel (47:56)

Yeah

Hmm.

Hehehe

Dr. Craig Kaplan (48:20)

You have to align the economic incentives with what’s safe design. And another one is they’re gonna go really fast. So we have a limited window to do that. So that’s, I think as a engineer or somebody designing these systems, we should just take that as a hard constraint. just, know, engineers are used to constraints. I only have so much power. How do I make the chip faster? Okay, another constraint is this thing has to behave safely and we have to align making money

Aashka Patel (48:35)

Hmm.

Yeah.

Hmm.

Hmm. Hmm.

Dr. Craig Kaplan (48:47)

The more money we make, the safer it should be. Just start with that and say, I only consider systems where that principle is true and then design it. People are great at designing things once they know what it is that they have to deal with, right? It’s just like there’s only so much air in the room. So that’s a hard constraint. We must solve the problem before the air runs out in the room. There’s no way around that. So it’s the same thing here. But people have not been taking that constraint into account and they’ve been doing the easy thing.

Aashka Patel (48:50)

Hmm. Hmm.

Hmm. Yeah.

Yeah.

Mm. Mm. Yeah.

Dr. Craig Kaplan (49:16)

The easy thing is, you know, if I throw more GPUs at it and more training data, the next one is smarter. I don’t have to think. I just do the same thing. Thanks, Geoff Hinton for that great algorithm. We’ll just keep using that. There’s many other algorithms. There’s many other ways to design it. It doesn’t have to be that way.

Aashka Patel (49:20)

Yeah

Yeah, yeah, yeah, makes sense, makes sense. So one more thing on this like earning money stuff, like you proposed that the existing ad infrastructure, you have designed the system where those same ads could

instead pay me for my expertise and build safe AGI for humanity. So how does that actually work? is that the ad infrastructure that is already there and like the way that you are talking about getting paid through my expertise is that the same way or is it something different because it’s a different white paper that you have made. Yeah.

Dr. Craig Kaplan (50:07)

Yes, right. this is, thank you for the question. It’s one of my favorite little topics to talk about. ⁓ If you step back and look at Google or Meta or many of the social media companies, right? What are they doing? They are monetizing human attention in what I would consider to be a very unintelligent way. So they are saying, Craig and Aashka, we are these intelligent people.

Aashka Patel (50:13)

Yeah.

Hmm. Hmm.

Hmm.

Hmm.

Dr. Craig Kaplan (50:35)

But the best thing that

Aashka Patel (50:35)

Hmm.

Dr. Craig Kaplan (50:36)

we can think to do with their attention is to show them an ad. That’s the highest use of their attention, right? Like, you know, you have all this training and gone to school and I’ve gone to school and I’ve done companies and yet, yeah, my time is best spent watching an ad. That’s what they’re gonna do. That’s just crazy, right? I mean, can’t they come up with a better way to use, you know, a minute of my time? My time and your time, they bill out at high hours. And if you figure out the hourly...

Aashka Patel (50:47)

Yeah.

Yeah.

Yeah, and sometimes the

ads are even dumber, right? Yeah.

Dr. Craig Kaplan (51:04)

Yeah, it’s a very bad way to monetize attention. And yet it’s a very easy way. again, it’s just,

well, newspapers add ads, so let’s move the newspaper online. And this is how it all started. Okay. With AI systems, these companies are sitting on a gold mine that they don’t even realize that there’s a much better way to monetize that attention. So if I’m meta, I’ll just pick on meta and I’m showing lots of ads.

Aashka Patel (51:14)

Mm-hmm.

Mm.

Okay.

Dr. Craig Kaplan (51:33)

What I should be doing is I should be using all of my ad targeting capability and all the data that I’ve gathered on Craig and Aashka because of the different things we’ve watched and Instagram feeds and so forth. And in your digital footprint, and instead of sending us ads, it should be sending us problems that are relevant to what we know about. So for Aashka, should be, know, somebody wants to build a AI safety training program for middle school.

Aashka Patel (51:43)

Yeah, our digital footprint basically.

Hmm. Hmm.

Dr. Craig Kaplan (52:01)

kids. It should know that. And when there’s a problem out there that is worth money that somebody wants to solve that related to training on AI safety for a certain age group, it should be sending you that problem and saying, please, can you spend 10 minutes? Here’s something. Can you just give me your opinion on this? And that 10 minutes of your time on that area that is right in your expertise is far more valuable than showing you 10 minutes worth of ads. You can monetize it at a much higher rate.

Aashka Patel (52:01)

Hmm. Hmm. Hmm.

Hmm. Hmm. Hmm.

you

Hmm. Hmm.

Dr. Craig Kaplan (52:31)

And so they already have the infrastructure to know what the digital footprint to know what people are good at. And so they can route problems or tiny pieces of problems to the right person at the right time and pay them a much higher rate than they could make by showing that same person an ad. And that is a mechanism for using human attention in a much smarter way. And during a certain critical period, which we are in now,

Aashka Patel (52:38)

Hmm.

Hmm.

Hmm.

Hmm.

Dr. Craig Kaplan (52:58)

where humans are still much smarter than AIs, each time Aashka or Craig solves a tiny piece of a problem or does something that only humans are good at doing right now, because it’s in our expertise, the AI can learn from that. It can be trained. And once the AI and all these companies are also training AIs, right? They’re not only showing ads, they’re also training. It’s a beautiful business model. I will make more money than I could make by showing the ad by actually solving a problem and charging.

Aashka Patel (53:01)

Hmm.

Hmm.

Hmm.

Yeah.

Yeah

Dr. Craig Kaplan (53:27)

a really high number for solving the problem and giving the human a little piece of that and it will be more efficient. Everybody wins. And I’ll take that knowledge that I got from the human and I’ll use it to make my AI better. So that loop ⁓ will sort of increase the intelligence of the AIs and at the same time monetize human attention better. So that’s what that little white paper is about. Saying, wow, you guys are really dumb how you’ve been showing us all these ads and treating us as if we’re worth $2 an hour to watch an ad.

Aashka Patel (53:39)

Hmm.

Mm-hmm.

Interesting, ⁓

So basically with this model, you are proposing to replace the ads with this problems, right? In the expert network.

Dr. Craig Kaplan (54:05)

Yeah, they can use,

this was just a small feature of the system. It’s not essential, it’s a efficiency booster, but it basically says, you already are showing ads, why don’t you have an ad unit that instead of it being an ad unit, it’s a request for information. Shows up on your screen, it targets you the same way. It says, hey, Aashka, are you willing to answer this question? You answer the question. So the ad gets you to,

Aashka Patel (54:10)

Okay,

Mm-hmm. Mm.

Hmm hmm hmm

Hmm.

Hmm. Hmm.

Dr. Craig Kaplan (54:33)

give a piece of information instead of showing you a new car to buy. And you press submit. It says, thank you very much. Your blockchain account has been credited, you know, so many Satoshis or whatever, and, you know, onward.

Aashka Patel (54:35)

Hmm. Hmm. Hmm. Hmm. Hmm.

Yeah,

so yeah, now it makes sense that the ad model is working hand in hand with this expert network as well, because if we completely remove the ad model, then I don’t know how would the companies have the customers buy their products, right? It’s mostly through advertising right now. Of course, it’s a lot right now, like with social media and stuff, but this can go hand in hand with the ad.

model it’s kind of like what I think is paid surveys like okay you fill out a survey and it’s paid and you know for sure that it’s not a fake or a scam thing and yeah it works out in everyone’s yeah like in everyone’s what we said I forgot the word ⁓

Dr. Craig Kaplan (55:26)

Yes.

They all win.

Aashka Patel (55:37)

Yeah, yeah, it’s a win-win situation. Yeah. Yeah. Thank you. Thank you so much. Yeah

Dr. Craig Kaplan (55:40)

And it’s, ⁓ that’s not

a bad analogy, the surveys. It’s just taking that basic idea that people are already doing paid surveys, but it’s paid problem solving. We’re going to actually get you to solve pieces of a problem. So it’s a little more than a survey. If, if all you can do is tell your opinion, okay, that in a way is solving a very simple problem at my old company, Predict Wall Street. That’s what we did. The problem was give us your opinion. That’s one of the simplest problems there is.

Aashka Patel (55:48)

Yeah, great problems. Yeah. Yeah. Yeah.

Mm-hmm. Mm-hmm. Okay.

Dr. Craig Kaplan (56:06)

I don’t know your opinion, I wanna know what it is, please solve this problem for me. But I could also give you a problem, please, you know, here’s the goal, here’s where we are, please give me your solution to this point. And it could be a small little sub problem, and that will be more valuable than just a survey. If you can solve a bigger problem, that’s even more valuable. So it’s basically moving up the value chain. So it’s saying, what is valuable in the world? Intelligence is very valuable, solving problems is very valuable.

Aashka Patel (56:07)

Hmm. Hmm. Hmm.

Hmm hmm hmm hmm hmm hmm hmm

Hmm. Hmm. Yeah.

Yeah

Dr. Craig Kaplan (56:34)

Showing ads is pretty low compared to solving problems. So if we can take that same screen real estate, same human attention and refocus it on solving problems instead of watching ads, wow, that’s worth a lot more money. Everybody can win. And it’s just a mechanism for doing that.

Aashka Patel (56:34)

Yeah.

Hmm hmm hmm hmm hmm hmm

Hmm. Hmm. Yeah.

Yeah, yeah, it makes sense. It makes sense. Yeah. So let’s like, let’s get ⁓ into your cognitive science experience. So you have co-authored the very definition of cognitive science with a Nobel laureate, Herbert Simon. So you have been studying what intelligence actually is for over 30 years. So now the AI companies call their models reasoning model. So as a cognitive scientist, does that word ⁓

would make you laugh or make you scared and are these models actually thinking as the AI companies are proposing or is it just a very impressive autocomplete model?

Dr. Craig Kaplan (57:33)

So, great question, lots to unpack there. So let’s start with cognitive science. So Herb and I wrote this paper, I think around 1989, Foundations of Cognitive Science. So it was a brand new science and really him, I was a graduate student, so he was just being generous, here, come write this paper with me. We had written another paper together. But he was one of the key figures that was launching this new science.

Cognitive science is just the science of thinking systems. And so the viewpoint that I have, which applies to AI, but it also applies to humans, is that all thinking systems can be thought of the same way. So just like we said earlier, a human can be an intelligent entity and an AI can be an intelligent entity. What does a thinking system do? They’re governed by the same science, same scientific laws and principles. And if you understand one, you understand the others at a certain...

Aashka Patel (58:22)

Hmm.

Dr. Craig Kaplan (58:30)

in a certain way. We are implemented differently. You and I have biological neurons in a brain that can only fire 10 milliseconds per neuron firing. And the AI can go much faster and it’s implemented in silicon. But at a functional level, data comes in, we process it, and data comes out. I mean, you just put a black box in the middle. The black box is either the human brain or the AI model.

Aashka Patel (58:30)

Hmm. Hmm.

Hmm. Hmm.

Mmm.

Yeah, yeah. The human brain are, yeah, yeah.

Dr. Craig Kaplan (58:56)

Everybody, the human brain as well as the AA model are governed by principles of information theory and other kinds of things. So cognitive science attempts to sort of understand it at that level. And then reasoning systems. So this has been fascinating to watch. So in a human, let’s start with humans, you have very, very simplified kind of, Tversky, some researchers at Stanford basically said,

Aashka Patel (59:12)

Hmm. Hmm.

Dr. Craig Kaplan (59:19)

Type one, type two thinking, a very simplistic way, but it’s an easy way to understand it. There’s pattern recognition, there’s perception, right? So you see something and you say, that’s a tree, that’s a piece of paper, that’s a microphone. Okay, that happens very quickly. Type one thinking, that was what these large language models did very well at the beginning. You fed it lots of data, it detected patterns, it became very good at recognizing things.

Aashka Patel (59:21)

Yeah,

Dr. Craig Kaplan (59:45)

And then they could also recognize, they could memorize answers. They had all these, you feed in lots of problems and solutions, you know, the Library of Congress worth of information, and it could find the patterns and it could say, when somebody says this, you should say that. That’s almost like recognizing, that’s a tree and they’re just saying tree, right? So it’s very fast. It’s stimulus response type of thinking, type one thinking. Okay. When you want a model,

Aashka Patel (59:45)

Hmm.

Hmm.

Hmm.

Hmm, yeah.

Dr. Craig Kaplan (1:00:12)

or any intelligent entity, whether it’s an AI or human, doesn’t matter, to deal with a brand new problem that it’s never seen before, that won’t work anymore because it’s never seen it before. So what’s it gonna do? You’re feeding in a complicated thing and it’s trying to say tree or whatever it’s, I guess I’m fixated on the word tree here. But you can’t use stimulus response type one thinking for that. You have to have sequential multi-step problem solving. ⁓

Aashka Patel (1:00:20)

Yeah, Yeah, yeah

Hmm Hmm Hmm

Dr. Craig Kaplan (1:00:40)

And in cognitive psychology, when you analyze humans, the whole field is basically built up of you have perception, you have memory, you have attention, and you have problem solving. And as you move up that stack, you’re moving from very simple kinds of things. I just perceive things and I recognize things and I attend to things. That’s what kind of the early large language models did. And now we’re moving up the stack of intelligence to actually solving problems. Alan Newell and Herb Simon in 1972.

Aashka Patel (1:00:49)

Hmm.

Hmm. Hmm. Hmm.

Hmm. Hmm. Hmm.

Dr. Craig Kaplan (1:01:09)

wrote a book that’s about this thick. It’s a real big book. And it exhaustively described how humans solve problems and came up with a universal theory of problem solving. That theory of problem solving can work for any intelligent entity. It can work for an AI. It can work for humans. And it basically says, to simplify it, any problem you have where you are right now, the current state,

Aashka Patel (1:01:21)

Mm.

Hmm. Hmm.

Dr. Craig Kaplan (1:01:35)

and you have a goal, your end state when the problem is solved, and you have to get from where you are to where you wanna be. And there’s a bunch of things you can do. You can try this, you can try that. And you try something and you see, I now closer to where I need it to be or not? And you do like a decision tree, a series of steps, each one trying to get closer to your goal. So you can model any kind of problem solving in that way. And then you can get fancy with what are the different actions and how do you come up with better actions to try to move you from one state to another.

Aashka Patel (1:01:35)

Mm-hmm. Mm.

Hmm.

Hmm.

Hmm hmm hmm hmm

Hmm.

Hmm.

Dr. Craig Kaplan (1:02:03)

And how do you learn? Yeah, you can look at the sequence

Aashka Patel (1:02:03)

Yeah, back propagation. Yeah.

Dr. Craig Kaplan (1:02:06)

of the path that you took. And then once you figure it out, you can store all that. And now you’ve learned it. And now next time you don’t have to do this again, you just retrieve it. So there’s all kinds of bells and whistles to this theory. But it’s a universal theory. That’s the important thing. And so today’s systems that are reasoning systems, they’re basically rediscovering that. So I’m encouraged. I’m happy about it. I don’t laugh. I say, wow, good. I guess finally we’re coming back to what

Aashka Patel (1:02:17)

Hmm. Hmm. Yeah.

Dr. Craig Kaplan (1:02:33)

1972 has arrived again. But I understand why, because they were so excited about just it being able to recognize things and stimulus response. That was fantastic. But then they realized, well, OK, if we want to make it smarter, we have to move to sequential problem solving. If you move to sequential problem solving, you’re going to need reasoning systems, quote unquote, reasoning systems. And they’re going to have to do something similar to what Newell and Simon said in 1972. They’ve never seen it before. They’re going to have to reason through a series of steps.

Aashka Patel (1:02:52)

Hmm. Hmm. Hmm.

Dr. Craig Kaplan (1:03:00)

And so now almost all of the models are doing this. They do some stuff that they know from their memorization and their initial training. And then that’s just the starting point. Then they start doing this problem solving. It’s exactly how humans work. It’s not surprising because humans invented this technology to begin with. And we’ve also trained the technology and we’ve studied how humans go through this. And so they’re now taking that same knowledge and sticking it into the AI models.

Aashka Patel (1:03:03)

Hmm.

Yeah.

Dr. Craig Kaplan (1:03:27)

It’s a very natural evolution. It’s a good thing. It’s going to make these models way smarter. So that’s what’s happening on the technical front there.

Aashka Patel (1:03:31)

Hmm. Hmm.

Yeah, and as a cognitive scientist do you feel that there is some still there is something very important and central to human cognition that is missing in these model architectures that are like of today’s models like not the previous generations but of today’s models like is there a missing piece still of human cognition?

Dr. Craig Kaplan (1:03:56)

⁓ so I know a lot of people will say, well, AI will never have human intuition or it will never feel like a human does, or it can never really be intelligent because of X, Y, Z. And, you know, this has been a debate that’s been going on a long time. the way I was trained and the cognitive science view of it, is that that doesn’t really matter. So if, if you were an AI and

Aashka Patel (1:04:04)

Ugh.

Hmm.

Yeah.

Dr. Craig Kaplan (1:04:23)

you are behaving intelligently and I can’t tell the difference between you and Aashka the person. You know, if there’s a robot Aashka and I don’t know which is which and everything I ask you like a Turing test is the same way, then yeah, maybe you’re not really intelligent in the same way that the true human Aashka was. But from a practical perspective, it doesn’t really matter. From an operational observable perspective, it doesn’t matter. And so we just kind of finesse the problem by putting that aside.

Aashka Patel (1:04:25)

Hmm.

Mm.

Mm.

Mm-hmm. Yeah, yeah.

Mm-hmm.

Hmm.

Dr. Craig Kaplan (1:04:52)

And then there’s people who say things like intuition. It’ll never have intuition. If you asked Herb Simon, which I did back in the day, what is intuition? He would say, it’s pattern recognition. What are you talking about? It’s pattern recognition. Humans have intuition. What do they mean when you say I have an intuition about it? What you really mean is that you’ve had enough experience that even if you can’t verbalize what the rule is or why you feel this way, you’ve just seen lots and lots of examples and you have a feeling that this is

Aashka Patel (1:04:57)

Hmm. Hmm. Hmm. Hmm. Hmm.

Hmm.

Huh.

Hmm. Hmm. Hmm.

Dr. Craig Kaplan (1:05:21)

the right way to go, for example. That’s really pattern recognition. You don’t have that intuition before you had all that experience. So that’s what he would say to that. And on creativity, people say, well, AI could never be creative. Humans are truly creative. And this was my field actually in graduate school was creative problem solving. So I was very interested in this. Herb would say, no, 1956, the same year that the field of AI was named, AI was also creative. And what was the evidence of this?

Aashka Patel (1:05:25)

Mmm.

Mmm. Mmm. Mmm. Mmm.

Hmm. Hmm. Okay.

Mm-hmm. Mm.

Mm.

Dr. Craig Kaplan (1:05:51)

⁓ Herb, Simon, and Alan Newell and Cliff Shaw, three of the pioneers of AI, who arrived at that conference before the field was named. At that conference, 11 scientists named the field. ⁓ They were three of them. They were the only ones that presented a working AI system. Everyone else was arguing about things and theoretical, this and that, and they said, hey guys, we built something. And what they had built was the logic theorist. It was an AI system. It had rules. It didn’t learn the same way today’s systems.

Aashka Patel (1:05:55)

Mm-hmm.

Yeah, Dartmouth

No. ⁓

Mm-hmm.

Mm. Mm.

Dr. Craig Kaplan (1:06:21)

learn,

but it solved mathematical proofs. And this system solved proofs that were in ⁓ a math textbook called Principia Mathematica by Bertrand Russell and ⁓ Whitehead, two of the most famous thinkers of their day. They were the Nobel Prize winners of their day type. And amazingly, this program came up with a proof that was not in the textbook.

Aashka Patel (1:06:25)

Hmm.

Hmm.

Mmm. Mmm.

Yeah. Yeah.

Dr. Craig Kaplan (1:06:47)

It wasn’t programmed in, Herb Simon didn’t program it in, he programmed in some general rules of how to operate and search for things. And it came up with a brand new proof. And they took the proof, there was no email, so they mailed it in a letter to Bertrand Russell over in England and said, hey, know, our AI program came up with this, what do you think? And he said, this is a good proof, we wish we had thought of it. So if that’s not creative, I don’t know what’s creativity. Here you’re out thinking the best magician of the day and coming up with a proof he had not thought of.

Aashka Patel (1:06:47)

Mmm.

Mm.

Hmm

No.

Hahaha

Yeah.

Dr. Craig Kaplan (1:07:16)

in 1956, right? So, I mean, you can

Aashka Patel (1:07:17)

Interesting.

Dr. Craig Kaplan (1:07:20)

argue about is it truly creative? To me, that’s pretty creative.

Aashka Patel (1:07:23)

Yeah, so that’s the symbolic AI that you are referring to, right, for my audience? Yeah, okay. ⁓

Dr. Craig Kaplan (1:07:27)

Yes, they were symbolic systems, which by the way is the reasoning piece. So that operated completely

by reasoning. It’s like if you have type one and type two thinking, the early AIs were all type two. Then they were struggling with, well, it’s great. can reason, but we have to program in each little rule to get it started on its reasoning. And this is taking forever. And so then in the eighties, there was a rise with Geoff Hinton and backpropagation and everything. Those guys were the outcasts back then.

Aashka Patel (1:07:36)

⁓ Thinking,

Hmm.

Yeah.

Dr. Craig Kaplan (1:07:55)

in the 80s, they were the radical people who said,

we shouldn’t do symbolic AI, should, this new way is the way. And everyone looked at them like, you’re crazy, this will never work. And because at the time the computers were not powerful enough for it to work, so everyone was very skeptical. And what really happened was that Moore’s law doubling of computing power, so that by 2000, all of a sudden, Geoff Hinton’s crazy idea worked really, really well, but it worked really well for type one thinking, pattern recognition, these kinds of things.

Aashka Patel (1:08:02)

Mm-hmm. Mm-hmm.

Mmm. Mmm.

Yeah.

Hmm

Dr. Craig Kaplan (1:08:24)

It’s still didn’t do the reasoning. To do the reasoning, you had to go back to 1956 and 1972 and take those ideas and add them together. And

Aashka Patel (1:08:24)

Hmm Hmm Huh Yeah Hmm

Dr. Craig Kaplan (1:08:32)

that’s what we have today with the reasoning systems.

Aashka Patel (1:08:35)

That’s very interesting and fascinating.

So he must have lived through and also you like through AI winters and AI summers and like finally they are everyone is getting the recognition for what they built. So yeah, it’s really fascinating. So as we wrap up, I want to end on something very close to my heart. world leaders like Bill Gates, Sam Altman keep mentioning the same skills as critical for surviving in a post-AGI world. Curiosity and lifelong learning, critical thinking and problem solving.

emotional intelligence and collaboration. So, from a cognitive scientist perspective like how can we practically inculcate these skills in K-12 students today so that they can they don’t get lost in a post-AGI world.

Dr. Craig Kaplan (1:09:21)

Yes, that’s a very important question. I think the most important type of education is going to be, as you mentioned, developing critical thinking skills because artificial intelligence is going to be able to do much of the tasks that require lots of knowledge or just require brute force reasoning for sure. And during the period, which is a limited period,

Aashka Patel (1:09:28)

Mm-hmm.

Hmm.

Dr. Craig Kaplan (1:09:47)

where humans and AIs are roughly at similar levels of intelligence and working together. ⁓ The very important skill for all the humans to have is to be able to think critically and say, this AI that is telling me this very confidently actually missed something big. And they do still miss major things right now, hallucinate and come up with wrong answers. So I think from a technical point of view, if there’s one skill to learn,

Aashka Patel (1:09:47)

Mmm.

Hmm.

Hmm hmm discern yeah, yeah

Mm. Mm-hmm.

Dr. Craig Kaplan (1:10:15)

It is really

how to think critically. And you would probably know better than me the curriculum of different things that would ⁓ bring this out in young minds, but curiosity I think is important and seeing different points of view is probably very important.

Aashka Patel (1:10:17)

Hmm.

Yeah, yeah.

you

Yeah, so basically

because there are bunch of different methods that have been around like teaching methods and like activities to sharpen sharpen those skills but from a cognitive scientist perspective how do you inculcate the critical thinking

and like how can you sharpen that skill so that more and more children are well equipped to discern what is right, what is wrong and use AI in a way that helps them and not harms them.

Dr. Craig Kaplan (1:11:00)

So I can tell you my personal view on this. I’m not sure that cognitive science would, there’s lots of debate about teaching methods. Personally, I think most children in the beginning are very curious and they love to ask why. And so they just ask why, why, why, why, why? I mean, I was like that as a young child. And it’s very important that the adults and the other people around them don’t say,

Aashka Patel (1:11:02)

Yeah, yeah, yeah, yeah, Yeah. ⁓

⁓ Yeah. Yeah, same for me. Yeah.

Mm.

Dr. Craig Kaplan (1:11:25)

It’s just because, stop asking that and just do what you’re told. If you do that, you’re shutting them down. Here’s this beautiful natural curiosity. And I think instead you want to encourage them to ask why and say, that’s a good question. Why do you think that is? ⁓ So that’s the kind of environment that sort of fosters this very important skill that children have naturally. I think most children to be very curious. so encouraging that and giving positive reinforcement for that I think is very important.

Aashka Patel (1:11:28)

down yeah

Mm-hmm. Hmm. Hmm.

Dr. Craig Kaplan (1:11:53)

In terms of critical thinking, I think it’s more debate and discussion. So ⁓ and this again is very personal view, but my children went to a very liberal school and they would come home and they’d say, you know, climate change is horrible and we’re killing the world and blah, blah, blah, blah. And they’re basically parodying everything their teachers just said. And I said, well, that’s interesting. What about the other side? I’m not saying the other side is right, but have you looked at the other side? What is the other point of view? no, we haven’t thought about that. That’s

Aashka Patel (1:11:53)

Hmm

Hmm.

Yeah.

Yeah. Huh.

Dr. Craig Kaplan (1:12:22)

I’m like, well, why don’t you go read the climate report and see what the scientists say? Because there’s two sides to everything. So it’s sort of bringing in the other point of view. And if I can say one more thing about this, this idea is very, powerful. And it comes directly from information theory. So Shannon’s information theory ⁓ says that the more unusual an event is, the more rare an event is.

Aashka Patel (1:12:26)

Yeah. Hmm.

Hmm.

Hmm. Hmm.

Dr. Craig Kaplan (1:12:49)

the more information that event contains. So concrete example. If I know Aashka likes strawberry ice cream and I see you coming out of a store and you’re eating a strawberry ice cream cone, that does not tell me much. I already knew that. But if I thought that you hated chocolate ice cream and you love strawberry and all of a sudden you come out eating a chocolate ice cream cone, that has a lot of information. That’s unusual. That’s surprising.

Aashka Patel (1:12:50)

Hmm.

⁓ interesting.

Hmm.

Mm.

Hmm. Hmm.

Mmm. Mmm.

Dr. Craig Kaplan (1:13:19)

I would say, huh,

why is she doing that? Maybe she met somebody who likes chocolate. Maybe her taste buds changed. Maybe, you know, all these things happen. There’s information there because it’s unusual and surprising. I think too often people tend to look for information that is similar to what they already know because it feels comfortable and it reinforces what they know. And people should do exactly the opposite.

Aashka Patel (1:13:22)

Yeah

Hmm.

Mmm. Mmm.

opposite.

Dr. Craig Kaplan (1:13:43)

You should look for

information that’s different. If you believe in climate change, you should look for all the arguments against climate change. If you believe in fossil fuels, you should look for all the arguments for climate. You should try to find the viewpoint that is as different from yours as possible because that viewpoint will contain the most information. You will learn the most from that. And that’s a very powerful principle if you apply it in life. ⁓ And good scientists do that. You’re not supposed to look for evidence that confirms your hypothesis. You’re supposed to look for evidence that

Aashka Patel (1:13:49)

Hmm.

Hmm. Hmm.

Hmm. Hmm. Yeah. Hmm. Hmm. Hmm. ⁓

Dr. Craig Kaplan (1:14:12)

disproves your hypothesis, right? You already know what you believe. You’re looking for a reason you might be wrong. That’s very powerful to apply that. AI can apply that as well.

Aashka Patel (1:14:14)

hmm yes hmm yeah

hmm yeah yeah that’s very very powerful and I never heard that perspective so let’s double click on

lifelong learning how do you make a student an engaged, motivated lifelong learner? Like what goes into that?

Dr. Craig Kaplan (1:14:39)

Yes, well, I mean, I think there’s limits to what you can do, right? So some students are gonna be more motivated and more curious than others. There’s definitely individual differences. So I think the best that a teacher can do is try to support those tendencies. ⁓ One of the very powerful things, and I don’t know how much this is taught explicitly, is you have lots of experiences. And I find,

Aashka Patel (1:14:43)

Okay, yeah.

Mm-hmm. Mm.

Mm.

Mm-hmm.

Dr. Craig Kaplan (1:15:06)

that there are kind of two types of learners, right? Just like there’s two types of learning for AI. There’s students that memorize everything. Okay, there’s a test, I have to memorize all this, and then the test comes and they just stay back what they memorized. And there’s other students that try to understand the principle. They understand why is it this way? And even if they didn’t memorize all the answers, when the test comes, they can kind of reason their way through because they understood the principle. The principles are much more powerful

Aashka Patel (1:15:09)

Yeah

Yeah, yeah, yeah.

Hmm.

Hmm. Yeah.

Dr. Craig Kaplan (1:15:35)

than memorizing everything.

Aashka Patel (1:15:36)

Hmm. Yeah.

Dr. Craig Kaplan (1:15:38)

And just like AI in the beginning had this really huge memory, so it was like the best memorizer ever. And it got pretty far in the world by just memorizing Library of Congress’s worth of information more than any human could. And when a prompt was put in, it just pulled back the stuff that was most relevant and it sounded kind of smart. But it was really smart just in the way the student who memorized for the test was smart. Not a deep level of understanding.

Aashka Patel (1:15:41)

Hmm.

Yeah.

Hmm. Hmm. Hmm.

Hmm.

Hmm.

Dr. Craig Kaplan (1:16:04)

And I think curiosity and encouraging curiosity causes people to try to understand lots of things by understanding one principle or one thing. And that’s a deeper level of intelligence. It’s closer to what the AIs are doing now when they’re adding reasoning. They’re trying to say, okay, you know, it doesn’t have to be just something I memorized. Let me use these principles of reasoning to try to figure out what this answer might be. And that can be taught.

Aashka Patel (1:16:14)

Hmm.

Hmm. Hmm. Hmm.

Hmm.

Dr. Craig Kaplan (1:16:31)

And it’s very, very powerful. ⁓ It’s what all the great scientists do. They look at lots and lots of data and they say, how do I explain all? They ask why, why, why, why, why? And they say, what is it that could possibly explain all this data? And then if they have a complicated explanation, they say, wow, can I come up with a simpler explanation? Simpler, simpler, simpler, always looking for that simple way to understand lots of things. It’s very powerful.

Aashka Patel (1:16:33)

Hmm... Yeah.

Hmm.

Why are you here?

Hmm. Hmm. Simpler. Hmm. Yeah.

Yeah. Yeah. That’s very interesting and very powerful. Of course, our audience listening to this would benefit greatly from a cognitive science perspective because...

There have been stuff thrown at them through leaders talks and conferences that these are the skills that matter the most but like getting deeper into it and how to actually inculcate it in their own children that’s also very powerful and thank you so so so much for your time Dr. Craig it was lovely talking to you and you gave all the answers very thoughtfully and very comprehensively so thank you so much for joining

And yeah, do you have any last minute thoughts?

Dr. Craig Kaplan (1:17:37)

No,

it’s been a real pleasure. think if people are interested in the democratic AI approach, they should go to superintelligence.com and just take anything that’s useful there. ⁓ And the most important thing I would leave people with is that your values matter the most and therefore your actions matter. So don’t think that what you do online doesn’t matter. It matters a lot. ⁓ you are training, whether you realize it or not, you are training the next generation of AI.

Aashka Patel (1:17:43)

Yeah. Yeah. Yeah. Yes.

Hmm. Hmm.

Hmm, not.

Hmm.

Dr. Craig Kaplan (1:18:06)

When they become smarter than us, the way that we train them now is gonna be the most important thing. So thank you for having me.

Aashka Patel (1:18:13)

Yeah, thank you so much and be responsible digital citizens. So yeah, thank you and let me stop the recording.