# Is More Compute the Answer to Model Performance

Why OpenAI Abandons Products, The Biggest Opportunities They Have Not Taken & Analysing Their Race for AGI · What Companies, AI Labs and Startups Get Wrong About AI with Ethan Mollick

20VC · Jul 31, 2024 · 69 min · 15,487 words
Speakers: Ethan Mollick, Harry Stebbings
Source: https://www.996.fm/episodes/20vc--ep-e1202e7f/

## Cold open

**Ethan Mollick** [0:00]:

OpenAI abandons products like crazy. They wanna build the machine god. If you have any talented people, you're going to have them building the next technology for AGI. If you have compute, that's what you throw at it. I mean, they're incidentally making $3,000,000,000 run rate this year, I think, by, like, just accidents. But there isn't really a product there right now. It's it's the chatbot and the API. I think a lot of people in this space are just assuming scale solves issues. The real problem right now is every startup in the world is betting against AGI, which I find really funny because all the funders are like, yeah, AGI is coming in next five years. If it is, why are you funding these startup companies? None of them are surviving an AGI world.

## Intro

**Harry Stebbings** [0:37]:

Welcome to 20 VC with me, Harry Stebbings, and I'm so excited to welcome our guest to the hot seat today. Joining us, Ethan Mollick. Now Ethan is one of my favorite writers on AI, and his blog, One Useful Thing, is an absolute must read for me. For those that do not know, Ethan is a professor and co director of the generative AI Lab at Wharton. Now there's a lot in this show. Time to get the notebooks out, maybe take down the playback speed to a naught point eight x. It is quite fast, but it is an incredible discussion today.

## Sponsor read

**Harry Stebbings** [1:06]:

But before we dive in, when a promising startup files for an IPO or a venture capital firm loses its partner, being the first to know gives you an advantage and time to plan your strategic response. Chances are the information reported it first. The information is the trusted source for that important first look at actionable news across technology and finance, driving decisions with breaking stories, proprietary data tools, and a spotlight on industry trends. With a subscription, you will join an elite community that includes leaders from the top VC firms, CEOs from Fortune 500 companies, and esteemed banking and investment professionals. In addition to must read journalism in your inbox every day, you'll engage with fellow leaders in their active discussions or in person at exclusive events. Learn more and access a special offer for 20 VCs listeners at www.theinformation.com/deals/20vc. And speaking of incredible products that allows your team to do more, we need to talk about Secure Frame. Secure Frame provides incredible levels of trust to your customers through automation. Secure Frame empowers businesses to build trust with customers by simplifying information security and compliance through AI and automation. Thousands of fast growing businesses including Nasdaq, AngelList, Doodle, and Coda trust Secure Frame to expedite their compliance journey for global security and privacy standards such as SOC two, ISO 2,701, HIPAA, GDPR, and more. Backed by top tier investors and corporations such as Google, Kleiner Perkins, the company is among the Forbes list of top a 100 startup employers for 2023 and Business Insider's list of the 34 most promising AI startups of 2023. Learn more today at secureframe.com. It really is a must. And finally, a company is nothing without its people. And so I want to talk about Cooley, the global law firm built around startups and venture capital. Since forming the first venture fund in Silicon Valley, Cooley has formed more venture capital funds than any other law firm in the world with 60 working with VCs. They help VCs form and manage funds, make investments, and handle the myriad issues that arise through a fund's lifetime. We use them at 20 VC and have loved working with their teams in The US, London, and Asia over the last few years. So to learn more about the number one most active law firm representing VC backed companies going public, head over to cooley.com and also cooleygo.com, Cooley's award winning free legal resource for entrepreneurs. You have now arrived at your destination.

## Conversation

**Harry Stebbings** [3:40]:

Ethan, I am so excited for this dude. I told you just now, I am like your biggest fan from afar. So first, thank you so much for joining me today.

**Ethan Mollick** [3:48]:

Oh, I'm I'm thrilled. It's, it's great and I've I've been an entrepreneurship professor for a very long time before anyone knew about my AI work, so it's always great to be connecting to the VC entrepreneurship world.

**Harry Stebbings** [3:57]:

Now for anyone that doesn't know your work, can you just give a sixty second intro on your work and how you've become much more well known in the last few years?

**Ethan Mollick** [4:06]:

I'm a former entrepreneur myself. So the startup company I I helped cofound invented the paywall. So I still feel like I'm I'm trying to make up for that in the late nineties. So just pay trying to pay back after that. Then I I've been a professor of entrepreneurship. I got trained in MIT, and then I've been at Wharton ever since. You know, I do a lot of work on teaching and research on how entrepreneurs become successful, but I also have this side gig of thinking about AI and teaching for a long time. So I worked at the Media Lab with a guy named Marvin Minsky, who's one of the founders of AI. And I was like the nontechnical person there who was like trying to translate what the lab was doing for the world. And then I've been building tools for how do we teach entrepreneurship at scale because it turns out it really matters. Little bits of entrepreneur training make a huge difference in people's lives. And we play with AI and other tools. So when AI sort of came out, I was in the weird place of actually practically using these tools for a long time beforehand. It turned out everybody else who was taking this stuff seriously was computer scientists. So I sort of was there at the early days of like, oh, I know business stuff and entrepreneurship stuff and education and these things are actually quite useful. And I already have a fairly large Twitter following, so I just sort of became the go to person. And then there is this Matthew effect of like, all the labs started talking to me, and I get insider information and everything. It becomes a sort of self reinforcing prophecy.

**Harry Stebbings** [5:17]:

By the your Twitter game is fantastic, so, like, don't change that at all. I love it. I wanna start though, and it's pretty perfect timing. I said we were pretty casual in how we did this. You know, we saw the new LLM a 3.1 model come out yesterday. I'm just really intrigued to hear your thoughts, Ethan. What did you think? Is it what you expected?

**Ethan Mollick** [5:36]:

So there's, like, four or five dimensions that the LLM a model is super interesting in. We could talk about open weights and open source being one model. I'm not surprised that they caught up with the leading edge state of the art models. I think people are underestimating how much ammunition the closed source labs have and are gonna release in the near future. I think it's great. We now have open source GPT four capable model and it's going to be everywhere. And, you know, it's just interesting because how much of that gap gets closed by that model. So every national government had worse AI than every kid in Mozambique had access to through GPT four o. So now there's a openly available fine tuned model. We're gonna see a lot of weird effects from AI that were delayed happen as a result. Actually, using it, it's pretty good. I mean, I I don't think it stands out compared to a Claude at this point or something else, but it will soon because people will be working on it. And it is a downloadable open overweight model, which is kind of a big deal.

**Harry Stebbings** [6:29]:

How much of that chasm do you think will be closed by the closed source providers with their next releases?

**Ethan Mollick** [6:35]:

I think we don't know a lot, and even the people training the models don't know a lot. I mean, part of the weird bit here, right, is the people training the models are all computer scientists, basically. I mean, doing computer science, and they don't have a huge idea of the implications of their systems. When OpenAI released GPT 3.5, they didn't expect to destroy higher educator, you know, education and then they have to rebuild it because everyone's cheating all of a sudden. Right? I mean, they're already cheating. Now they're just cheating really well. But we weren't expecting, like, a large scale revision of, like, how the world works. Right? Everything I'm hearing from everybody is the next generation models is going to be smarter. Right? The exponential continues. Whether or not that translates to the real world implications is a different kind of concern.

**Harry Stebbings** [7:13]:

Do you know what I find challenging Ethan? It is every week you go on Twitter, and there's this transience of dominance between the different providers. You know, OpenAI do something and it's like, wow. That's incredible. And then Claude do something and it's like, wow. That's incredible. LLM, what and every week it seems like this one's the winner and the rest are losing. And there's just such transience and speed. I almost don't know where to go.

**Ethan Mollick** [7:36]:

Is that understandable? It makes complete I mean, it doesn't help that social media likes buzz. For normal people sitting back yeah. We're just gonna keep using ChatGPT because that's what they're using. Right? They, like, gradually switch to club. Like, the enthusiast community is very different than when I talk to the outside world about this stuff. And I think that on the grand sweep of things, what really matters is when these models top out and how long that takes. And I think worrying about who's in the lead at one moment is probably less of an issue than the big labs are all gonna keep building. There's no tricks in LAMA that they really told us that were unusual or indicated some sort of secret breakthrough. We still didn't know if there's secret sauce in some of the other labs that are very different. Like, it's very early days in some ways, so I think trying to get you know, if you're enthusiastic like me about this technology, great. Follow along and keep track of the m m l you know, m m l a ratings. But otherwise, I do think there's a little bit of, like, unnecessary to get every detail at this stage.

**Harry Stebbings** [8:28]:

I totally get you. And you mentioned there about topping out. Before we discuss kind of the potential topping out and what happens when that does, I do just wanna start on actually the four potential outcomes. You highlight this in your book, which I loved, and I just thought it'd be helpful to start there as a framing. What are those four potential outcomes first?

**Ethan Mollick** [8:48]:

Okay. The other four outcomes, Twitter, it only talks about, like and, you know, the press only talks really about one and four. So let me go through one and four first, then I'll give you the boring middle. Right? So option one is this is it. Models don't get much better or, you know, and it's sort of this whole thing sort of fizzles out. I think it is unlikely because I think not only will models get better, but also we haven't even started integrating them into work yet. Right? Like, the way you work with these things is an insane process of actually using a chatbot and having a conversation with with a with a chatbot is how people are using it for work at this stage. So we're we're not even at the stage of integrating, but it's possible that things kind of that, you know, in this in which case, we have 10 or so of integrating the system slowly into human systems. I think everything sort of stabilizes out where it is. We're not gonna see we'll we'll see an economic improvement, but we probably don't see kind of a massive large scale shift except in like, some industries change more than others. Right? I think it's very likely that photography changes a lot and that there's other field, like, customer service changes a lot with our current systems. But that's one option. Right? Nothing much happens. And option four is the machine god. Right? We sort of achieved this AGI plus super intelligence thing. Machines are smarter than humans. We have a intelligence explosion and god knows what happens next. We've had a good couple hundred thousand year run as a species. You know, we'll figure out what our successes are. And there's a lot of obsession with this because I think that's where everybody, both boosters and people feel negative about this. This is where their minds go first. Right? It's like super intelligence. I think the more common scenarios we see a technology, right, are either continued exponential growth or else linear growth and ability. And I think that's what we're under preparing for. So if you look at the trends lines of this stuff, everyone's, like, trying to anticipate that every model is either going exponential from here or is, you know, everything is gonna top out. I think much more likely, you know, our study, this AI was as good at the 80% dollar level of consultants. Next year is the eighty fifth percentile, ninetieth percentile, eighty first percentile, hundred and eightieth percentile. We don't know. So I think a large part of this is that kind of world. And a linear growth world where the models get a little better every year, I think that's much more adjustable to one where it could use it exponentially, and we sort of get close to that AGI world.

**Harry Stebbings** [10:54]:

Maybe rightly or wrongly, I always think about actually iPhone releases. And, you know, the first iPhones, there were big differences between the early releases of, like, the three and then the four. And then slowly, it just became kind of a little bit better camera and a little bit better battery and maybe, you know, the calculator, slightly bigger buttons or whatever it is. And I'm like, what is it that AI or people believe AI has that believes it will have escape velocity of development and it will never achieve that plateauing?

**Ethan Mollick** [11:23]:

You're right. Now we're at the classic sort of top end of a technology where it's all about like the calculator was a major factor in the release of the new iOS. And you're like, this is where we are right now is much better calculator. I think hilarious. So there is a topping out. Now if you look at a process like Moore's law, it's been a sustained exponential curve for years. The difference is that it's a bunch of underlying technology that gets swapped out for each other. So the real question is what is the sort of top line intelligence? What does that max out at for what an AI can do? Are its limitations exceedable? Right now, AI is jagged. So it's really good at some stuff, really bad at other things. So and as a result, it can't sub in for all of human work because on one hand, it'll do a great job in some of the job stuff, so it will do a bad job in other things just like any machine does. The question is, can that jaggedness get overcome? We we don't know the answers to any of these questions yet.

**Harry Stebbings** [12:14]:

Kevin Scott always says that, you know, compute will solve all problems, and many have always believed that, that performance will be answered by compute and just more brute compute. I have other people on the show. You're Alex Wangs at scales who say that data is the core bottleneck. When we think about compute data or algorithms, what do we think is the core bottleneck to performance now and in the next twelve to twenty four months?

**Ethan Mollick** [12:39]:

I'll try and answer that, but I want to do the contrarian view first that I always wanna indicate first, which is for most people, just don't care. Let's say that LLM's top out and it turns out we have to switch to, you know, mamba or some other like, who cares? Nobody cares. They're using these systems. There's a lot of, like, in the weeds that you get when you're watching this, like, like, a sports game of, like, who's winning and what situation. Yeah. Top line capabilities matter, and there's a lot of room left there. Like, to me, the thing that gets left on is computer science discussions are often the system the human systems that these things have to interact with, the organizational systems they have to interact with, and that's where we need to see kind of more growth. Right? That being said, we don't know what the bottleneck is. Right? There's there's this idea in the history of science called the reverse salient, which is that technology sort of moves forward, but there's always something that's kind of lagging and all the effort goes into fixing the lag. So in the early days of electricity, we had generators where transmission was a problem. So there's a huge amount of work to make transition better. In our current electrical sort of new economy, it's been batteries. So there's huge amounts of work going into batteries because solar panels are good, but batteries aren't good. I kinda feel like we're just gonna hit a whole bunch of reverse salience. Right? So, like, oh, the data pipeline isn't good enough. Great. Is it gonna be real world data, synthetic data, or maybe this is the end. But once every all of science concentrates on one thing, we tend to find ways forward. So I think it's gonna be a bunch of debates over what the trailing indicator is, and then everyone forgets about that because so it gets solved. It's not a bad approach. It also is just kind of how technology works. Right? Because the money is all to be made in reverse salient. If you can make a billion dollars as a data company and but, you know, because that's the area everyone's stuck on, you become a data company. Right? This is capitalism and science at work. The hard problems are the ones where all the money and prestige comes from. You said where

**Harry Stebbings** [14:20]:

the money is. I loved an analogy that you said before. And it's you said a lot of people use the analogy of picks and shovels in the gold rush. You said that's maybe not such a good analogy and that the steam train was more apt. Why do you believe that it's not a good analogy, and why is the steam train more apt, Ethan?

**Ethan Mollick** [14:38]:

It's a great question. I I just analogies are really powerful, and we have very bad ones in AI. VC people get taken in by this. Right? So it's like, I hear you wanna sell picks and shovels. And first of all, I don't a 100% know. Like, everyone defines us slightly differently. They're like, oh, no. No. You wanna sell compute. You wanna sell the tools that help people scale up and pick their you know, first of it's unclear what the analogy is. But the second deeper problem of this is that that isn't actually how a new technology spreads across an organization. You don't wanna sell picks and shovels of the people trying to mine gold. What you wanna do is figure out how to get them to use this new technology, which isn't a gold rush analogy at all. Right? Instead, the steam power and the steam power, the secret was not James Watson's steam engine, was important. Right? Huge breakthrough. Two interesting things, by the way. The things didn't really take off until Watson's patents expired, and it can be openly adapted. But the real value of the steam engine came from having skilled artisans in your factory who said, I've got this thing that can make power go back and forth. How do I create the gearing to connect that to my, you know, my spinning Jenny, my ammunition manufacturing machine, my bottle shaping tool. And it was the skilled artisans that made all of this work and made the manufacturers capture all the money. So you wanna be a skilled artisan right now. You wanna figure out how to take the back and forth power of an LLM and convert that into usable work inside your organization. What floors

**Harry Stebbings** [15:58]:

me is the lack of human descriptions around how to use these tools effectively. It's like no one's written using LLMs for dummies, using AI for dummies, which everyone needs. Why are these providers not doing what is so obviously required?

**Ethan Mollick** [16:16]:

I think that if you talk to Silicon Valley people, they are very obsessed with the race for superintelligence, and I totally get it. Right? If you could build a machine god, you win. So that's kind of the secret story behind what's going on here. So the real belief that if scale solves everything, the biggest thing you could do to waste your time is do anything that isn't scaling. Your smartest people have to be scaling. All of your compute has to be scaling. And the bigger models will solve all problems as you were saying. You know, that's a sort of view in Silicon Valley. So they're gonna come back and figure this out later because why would you bother? You know? And there's some truth to that. Right? I I spoke to a very large financial institution, spent a huge amount of money building a GPT three powered sales assistant tool that as soon as ChatGPT came out was instantly obsolete. Right? So, like, why why bother with this guy? Mean, it was a smart idea at the time. They were way ahead of the curve. Right? But I think that the real issue is is that as a result, all use of this stuff has kind of been dropped. Right? There is no manual out there for this stuff. There's not even a dis like, there's not even a set of points about what AI is good at and what it's bad at. As a result, like, it's I would call it documentation by rumor. It's a bunch of people at Twitter. There's, like, 17 people posting about how they're figuring out how LMs work, and then everybody else is just kinda using it like a chatbot. It is a very weird situation.

**Harry Stebbings** [17:28]:

Can I ask you? You said that kind of about Silicon Valley and how they think about where the true value lies. You know, we have on the one hand that says we cannot have such powerful models open source. We have Marc Andreessen and others say that they have to be. What do you believe is best?

**Ethan Mollick** [17:48]:

I am generally in favor of technological progress, and I think that openness frees people to do lots of really interesting things. There's a very obvious low hanging fruit with AI and healthcare and education that I think are going to be very helpful in large parts of the world that don't have access to good doctors or good tutors. Right? For places that do have access to that, there's a lot more nuanced discussion about when do you turn to AI for some of these things. So I think open models will make a big difference. They'll spark entrepreneurship. We know that people who get advice from AI do better as founders in Kenya if they're already doing well. There's a lot of really exciting stuff here about openness, but there's also downside risks. And it feels very weird for people to say, like, it's all one thing or another. I do think that open models will immediately have their guardrails breached, and we already know three or four low hanging threats. I think people are overly worried about science fiction threats. Right? It's not good enough to help you build a virus at this point, but it could be in the future. We just don't know. But what I am worried about is our entire computer security system depends on it being very expensive to spear phish somebody. This does spearfishing at scale. What do we feel about that? Like, that you could do this. These systems are gonna be harnessed for very good catfishing campaigns. How do we feel about that stuff? I just feel like there's not this conversation. So I think the open models both carry risk and reward. I don't think there's a lot of thought going into this stuff. I think it's all corporate strategy at this point. Right? So Meta doesn't really wanna make money from models, so they're gonna spoil, you know, their rivals. Microsoft had a chance to go after Google, so it adds AI into Bing. Like, there's a lot of, like, back and forth among a few firms, and I don't think we actually know the full meaning of open source AI. And it's a little weird to both say it's super powerful. It could do everything, and therefore, it's high risk. And also, it's not that big a deal. You said there's a

**Harry Stebbings** [19:25]:

lot not a lot of thought going into it. What thought would you like to see going into it? Like, what do you think would be a commensurate level of thought and analysis?

**Ethan Mollick** [19:34]:

I think that we need to be built for fast reaction to these models. I what I'm worried. So there's Joshua Ganz who's a professor at University of Toronto, I think has a really nice model for AI regulation that I think is probably right, which is when you have a new technology, you don't know what the problems and issues are gonna be, you do fast follow-up regulation. You don't try and preregulate because you don't know what it's good or bad at, but you do watch what's happening and have rules that you put into place and policies and fast reaction. Now, we can talk all about how government's not built to do that, how it's not cooperating well with industry. That's the same way we think about open source right now. So we've just released a very powerful model open source. Who is setting up to learn for what the implications of this are going to be? And do they have a pipeline back to the open source makers of these models? Is there something that would stop Meta? Is there an event that would stop Meta from outsourcing, from open sourcing its models? I don't know. Who's watching that stuff? Are we have is there any kind of monitoring system out there to find out how this is disrupting the world one way or another? There doesn't seem to be. To me, a really responsible view would be, sure. Let's release open source, but then let's be watching over the next six months to get a sense of what this is good or bad at and, you know, react to it. And that's what's worrying me a little bit. I'm

**Harry Stebbings** [20:43]:

sitting in Europe where we have the EU AI act, which is incredibly stringent. EU is also particularly talented when it comes to regulation. I'm very worried that actually we will have such constraining regulation that it will actually cause the plateauing effect of AI both in development and in adoption. Do you think intense regulatory scrutiny is a cause for concern in the path to much more developed AI systems?

**Ethan Mollick** [21:08]:

Yeah. I mean, I think that not being fast and reactive is a problem. Right? You wanna have people develop this new technology. You wanna develop by the societies that you wanna develop these technologies in. Like, you want to be used in democratic ways. All of that stuff indicates, like, we wanna see continued growth. It just feels like it's either or for so many conversations. Like, either there's no regulation and no scrutiny whatsoever and technology always benefits everybody. And I'm technology optimist. Like, it does benefit people, but, like, it's weird to have no downside risk. On the other hand, you have the we must regulate in advance to stop a bunch of harms that haven't occurred yet and that the current levels of models clearly will not cause. Right? We're not gonna get a runaway superintelligence from a LLM a LLM a 3.1. So we have to have some sort of balance here. I think the EU has EU has has definitely put in a lot of stringent things in place. I don't know whether Europe would be leading in AI anyway. I mean, there's a weird ecosystem problem. Talk about, you know, where is this 20 BC? BC has always been a US thing. London did okay for a while there. But aside from that, you know, more money went to graduates from Penn, from the school I teach at, than everybody in France and Germany put together. We already have a whole bunch of innovation ecosystem problems in place. Regulation is one of them. I think a lot of people are pointing at EU regulation being like, this is the cause. There's a multi causal problem here in terms of Europe versus The US on technology development. Everyone moves to Silicon Valley because you kinda have to and all the stats show that's actually a really good idea for almost every venture. Like, there's a machine here that keeps working. Right? And so to go back to the bigger issue, I think putting a lot of tight regulation on AI at the beginning is definitely an issue because LLM breaches the high security risk level in terms of number of

**Harry Stebbings** [22:49]:

flops with the EU. Would you tell your students today that they have to move to the Valley if they wanna increase their chances of winning?

**Ethan Mollick** [22:56]:

That's an empirical result from a bunch of studies. Companies that you know, there's been a study of Israeli companies in the Valley, New York companies. The issue is is that that's where the connections are, and it turns out Zoom only gets you so far. The the average distance between a VC and a company invests in is about 40 miles. That's because when you look at what where VCs spend their time, it's networking and it's monitoring. It's networking with other with you know, and learning about companies, and then it's monitoring the portfolio companies. And that's much easier when you're local. Zoom doesn't let you do monitoring the same way. In fact, when direct flight is added between SFO and another city, VC investment in that city goes up because it's just get it's easier to fly there and help do and do monitoring there. It's a local business. Right? Everyone's like, oh, it's global. It's connected. It's a local business.

**Harry Stebbings** [23:40]:

Can I dive into a couple of the different market participants? We've already touched on some of them, but I wanna start on AI Labs. We've mentioned LLM and model progression earlier. In terms of the AI Labs, what do the big AI Labs not understand about companies themselves, do you think?

**Ethan Mollick** [23:55]:

I think that's such an important question. There is just the products being released are just super weird. Right? Like, I think there's very little consideration of use cases. I mean, you look at the number of people inside these organizations who worked at large companies. I I often joke when I go to the West Coast, there's like, you know, it's all like cold plunges and how do you live forever and really know, and then in the East Coast, it's like we do you know, we're drinking coffee till we die and the goal is like get our work done, get home, like, you know. In a large company, it's very different. There's a lot of, like, contempt, I think, for large companies. That's where most smart people are, right, are in large organizations doing, you know, other work that is not Silicon Valley work. For every coder, there are 16 managers. I I think that there's not a sense of what this stuff does for them. And as a result, there's a lot of half built products that are brilliant and then get walked away from. Code interpreter is a huge world changing product for data analysts that got partially abandoned by OpenAI. They haven't they haven't moved the needle in that sense. Chatbots and APIs remain the main area. Almost all the documentation is technical documentation, and almost all the interesting use cases are not being discovered by technologists who are actually quite bad at using AI often because it doesn't work like a normal technology. They're being discovered by end users, managers, and the system's not built for those things. So there's just this huge gap between technology and use.

**Harry Stebbings** [25:08]:

I'm so sorry to be so naive. Why if CodeInterpreter is such a generational defining product for analysts, why would they walk away from it or not walk away from it, but, you know, not progress in the same manner as they started?

**Ethan Mollick** [25:20]:

I think OpenAI abandons products like crazy. I think these products are passion projects from various people. Again, they wanna build the machine god. If you have any talented people, you're going to have them building the next technology for AGI. If you have staff, that's what you throw at it. If you have compute, that's what you throw at it. I mean, they're incidentally making $3,000,000,000 run rate this year, I think, by, like, just accidents. But there isn't like a there isn't really a product there right now. It's it's the chatbot and the API and the system gets smarter to solve more problems. I think a lot of people in this space are just assuming scale solves issues. So why would I or or development solves issues. So why would I bother spending some time thinking about, you know, how to productize this when the proxy be obsolete in a year anyway? Can I ask you, on the flip

**Harry Stebbings** [26:03]:

side, we have the companies themselves? What are companies getting wrong about AI that they should know more about?

**Ethan Mollick** [26:10]:

I mean, I speak to organizations all time. Mean, first of all, almost nobody uses these systems. I mean, they all try ChatGPT. Every when I ask my hand, everybody's tried ChatGPT. Five to ten percent of people in any room whether and that's by the way, Silicon Valley actual people, right, who aren't in a lab, whether that's at a large bank, whether that's at a conference of innovation professionals, maybe five to 10% have used those models, and maybe two or 3% have used 10, which has been my guideline, you know, minimum number. Again, there's no onboarding. You're faced with a chatbot and when people are faced with the turning the blank page, they panic. What do you talk to the system about? There's no information. There's no instructions. And so people aren't really using it. So the issue is that partially it's that they need to adopt because when people start using it, they find uses. Right? So a new study just came out of Denmark of people who are using ChatGPT in, you know, in knowledge intensive work environments. And, you know, they're estimating that over 30% of their tasks, they're saving 50% of their time. So once people use it, they find productive uses. So then the question becomes, how are you harnessing those uses? What policies do you have? I mean, there's so much we could talk about about what companies are getting wrong.

**Harry Stebbings** [27:13]:

In terms of how we harnessing the uses, what would you like to see change there? Because that's where we can fundamentally drive productivity, which is arguably the most important thing.

**Ethan Mollick** [27:21]:

I mean, so first of all, just starts with policies. When you look at companies, a lot of them don't even allow access to GPT four because the regulatory environment's unclear. So one thing that'll be great for a regulator perspective is not just we've talked about the negative side of regulation. There's a reason why banks are regulated or pharma companies are regulated. It'll be useful for clear guidance about how to positively use AI. And I think there has been some movement towards that. That that will be something I would want the EU to be doing a lot more of too. It's like, okay. What are the ethical use cases that we should be pushing and opening regulation for? So that but that extends the company policy side. Company policies are often very vague. You know, don't use this or use it, but don't get in a way that doesn't get you fired. And then there's a whole bunch of, like, uncertainty over how you get rewarded. What happens if you figure out a solution to your work? So what I find is inside organizations, when I finish with the talk, all these people come up to me and and reveal that they were secret cyborgs all along. So they've been using this for all of their work, but they're not telling anyone. They're not telling anyone because they're worried they get fired. They're worried that people stop respecting their work because they realize it's AI written. Right now, Reddit's full of people saying, I'm a people think I'm wizard of work. They're worried that people will realize you don't need as many staff members, so you fire them or you fire their their colleagues or you won't reward them for it. So everyone's hiding AI use. I just spoke to a woman who banned ChatGPT at a major bank. She used ChatGPT on her phone to write the ban Because, like, why do it by hand? So once people start using it, they're all using it secretly. And I said, we need clarity around how do you get rewarded for this. So, like, what happens if I automate my job? And to go back to our industrial revolution analogy, if you were a brewery in the early seventeen hundreds and you are serving your local community, right, because they get everything was kind of local, and you had steam power, you kind of have a choice. Do I want to fire a lot of people and make the same amount of beer for less money and have a higher margin? Or do I want to be Guinness and expand my production around the world and hire another 100,000 people? And we're used to IT solutions being a cost saving measure. Right? If I get 30% of productivity boost, fire 30% of people. Your people are never gonna show you how to use AI at that rate, and you're never gonna win in a world if we really believe there's industrial revolution happening. So policies are really at the heart of the problem. And I always feel

**Harry Stebbings** [29:26]:

that in the majority of cases, we just redistribute talent, as you said, that more effectively for new projects, for new initiatives, for expansion. What I find worrying is here, especially in the early days, it is killing the lower classes if you're being horrible and blunt, which is like, have, like, 70% improvements with AI in terms of customer service and cut so many of their workforce. You're seeing especially customer service be the core kind of Trojan horse, which is replacing 90% now of customer service teams. To what extent do you think I'm overwiring and actually we'll see the continuing redistribution of talent, not the removal of talent?

**Ethan Mollick** [30:04]:

So this is a case where I think we're being a little sanguine about this. I mean, I I think every technological revolution, people lose jobs and then new jobs are created. But there's you know, and we talk about this all the time. There are two big caveats to that. Caveat number one is not always. Right? When the telephone switchboards went from sort of manual to digital and starting not digital that way, but mechanical in the '19 starting in nineteen thirties. At that point, I think one out of every 16 women had spent time as a telephone operator. It was like a job. And then if you got fired from that, if you were young, you found other jobs. If you were older, you never found another job as good. Right? Because you were really good at telephone as a telephone operator. So not every job ends up with a new category replacing it. And the other thing is living through the industrial revolution kinda sucks. Right? Like, you can look backwards and say, like, oh, great. Everyone got better jobs. They were much richer now. But there were also people, you know, smashing machines because they didn't want their jobs replaced. There was a lot of unrest. There's a reason why that there was that's when the great debates between capitalism and and Marxism arose because there was unrest during this period that was serious. So I think part of what I worry about is a little bit of, like, even if you have a sanguine view that everything is gonna be fine, that doesn't just happen automatically. Like, you can't say the market makes everything great, so let's not worry about it, or else everyone's gonna lose their job and we need UBI. There has to be something much more specific about, yes, there's going to be waves of disruption heading through the economy. How do we do things we're very bad at, like retraining? It's a problem to actually be solved. It's not something that has to be made as science fiction. What what

**Harry Stebbings** [31:31]:

I worry more about is actually the distribution of knowledge productivity versus the distribution of wealth. And what I mean by that is there is this, like, 1% of, like, Silicon Valley and tech elite, I think, who are using AI and the surrounding products incredibly well to do 10 x the work that we used to do and to be way more efficient and way more cost effective. And then there's the rest of the world. You know, we joke about Europe, but I live in The UK. You go to place in The UK, they've got no idea what ChatGPT is, let alone how to use it to create marketing campaigns that are 99 cheaper in 10% of the time. I think it's just creating even more knowledge and productivity to 1%, and the world could get left behind. Am I right to fear that too?

**Ethan Mollick** [32:14]:

Yes. I'll say yes and. Okay. So Denmark study I told I I talked about did find that the people using the skewed mostly male and mostly wealthy. Right? There were people finding use cases because that tends to be a fairly common tech adoption curve. The thing that is unusual about AI though is, first, it's ubiquity. Normally, getting your new tech installed means I've gotta have you know, know how to use a computer really well and be really in with, like, you know, how do I get a, you know, a distro from GitHub and, like, you know, there's work involved that is that is a narrow set of work that requires time, effort, money. That isn't the case here. Right? The chatbot is accessible through a phone in, you know, a 169 countries around the world with access to the world's best AI systems. That's one thing. Right? And chat is a fairly normal interface, especially when you have voice. The second is early evidence is that coders are not particularly good at working with AI. Right? Because it doesn't do the things you expect it to do. My favorite example is Simon Williamson, if you don't follow, is is terrific and really rated this stuff. But he works on data journalism. And he built he was using Claude for OCR on political campaign donations. And when he checked back, he just found that Claude refused to do the work because there were names and addresses. And even though they were public, he was Claude was like, I don't wanna violate anyone's privacy. Like, we're not used to systems that object to the task that they're given or, like, sometimes argue with you or give you different answers every time. So coders are often not the best users. Often, the best users are people who are actually really good at working with humans. I mean, my my wife is probably one of the best prompt engineers on the planet. She's got a doctor who worked together with co directors of the AI Lab. She's never coded a day in her life, but regularly does stuff at OpenAI and Anthropic are like, wow, that's a really amazing prompt. We didn't know that Google used her prompt as the gold standard to measure their fine tuned models against. Right? But what she has is a, you know, doctor in education where we've been building educational teaching games for a long time, she has good theory of mind for other people. If you can write instructions, if you can manage, you can use this. So that's what I'm hopeful for. It looks like a tech adoption curve, but tech people shouldn't have the advantage they had in other spaces, and word just has to get out.

**Harry Stebbings** [34:11]:

I I do have to ask. You mentioned that kind of about the quality of prompts and how amazing your wife is with her quality of prompts. You also mentioned kind of the white screen of death and when you have kind of that blank template not knowing what to do with it. You said before about bluntly the kind of challenges of the chatbot interface and what a weird interface it is. What do you think will be the interface of the consumer between the power of AI and consumers?

**Ethan Mollick** [34:34]:

I think multimodal is really the answer here. All the pieces are in play. Some of the most interesting people, you know, people I talk to are really using it or just having conversations with it. Like, I think about Ali Miller, is a really great sort of person who has been thinking a lot about AI, ex Amazon person. And she has conversations with the AI every morning while she's doing her hair. Right? Just the limited chat interface. Once these things have full visual, which they do. Right? They have a lot of latent capabilities that people haven't recognized yet in multimodal, and you could chat with them. Then it starts being more like, you know, a human on call. I think once you start adding agency into that where they can take action in the world, I wonder if we just sort of skip the step of, you know, how do you use these things to, oh, yeah. You talk to your phone and your assistant does the thing that you wanted to do. There is this narrow window, I think, where prompting style really matters, where being really up to date in these systems matter. But then they come to your phone. And also, by the way, if they save you time and work, if they really do do that, humans are exquisitely designed to figure out how to minimize the effort they put into things. There's a reason why adoption rates are over 70% in universities for ChatGPT and while they're, like, at a few percent elsewhere in the world. We figure stuff out like this, and I think that that's the other piece that's missing. One element

**Harry Stebbings** [35:45]:

that we haven't discussed is is startups themselves, actually. And so on that, like, you said before that you don't think startups are being ambitious enough in the face of AI. What should they be doing, Ethan, and why are they not doing it?

**Ethan Mollick** [35:58]:

I think we're I think the problems of the lean method are coming home to roost. What every VC wants to see is they wanna see product market fit. There's a method we have. Right? You come up with like a, you know, rough business model canvas and then you go out and you do talk to people and then you tested the world. That is not a good model for breakthrough innovation. That's a really good model for incremental innovation where you find market need. So part of this is that we're incentivizing startups to find solutions right now for a moving technology and they're just gonna get lapped. And they're not trained to be imaginative. They're not like it's they're trained to think money first. And, you know, how do I get a market product market fit, which is fine in normal technological regimes. Not a great idea in radical regimes. What is a good

**Harry Stebbings** [36:43]:

model model for a radical regime then? Because I've been brought up quite rightly, it as you mentioned there, the incremental innovation kind of economy where it's like test, iterate, find product market fit, someone pays for it. Good. Well done. So what is the right model in this new age of kind of radical innovation shift?

**Ethan Mollick** [36:59]:

So, I mean, we we see have funded this model. Right? And it's like deep tech medical. Things where you're making larger bets in the future, where there, you know, where there's payoff is where where when it's revealed to the world, gonna succeed or not. Right? And where you're making a bet on technology itself, that's where VC got its start. It sort of became, you know, perverted a little bit to this, like, how do I get, you know, big money fast machine? I mean, not that fast. Right? Still years till exit. But there's the idea of, like, yeah, it's all about pro rata rights and the idea of, like, I make a lot of small bets initially, and then I could double down on the people doing well and not do double down on others. And it's about finding the diamond in the rough. Like, all of that stuff is, like, a great model for funding incremental innovation. If the market's changing, like and we're used to market changing slowly enough that, like, that's not a problem. I think it's an issue here. I think you need to be imaginative. I think you need to be subject specific. I need think you need to assume model. I mean, it is very strange from one hand for all of these people at Silicon Valley to be like, yeah. AGI is coming. And then the applications they're building are like these very narrow, like, hey. I something at top of LLM. That's not gonna do it.

**Harry Stebbings** [38:01]:

What should I and what should my fellow venture investors change then about the way that we invest, do think?

**Ethan Mollick** [38:06]:

I think that what you should be thinking about is have a position in the future. And if and startups you talk to have to have a position in the future of AI. How good does it get, and how does your model work? The second thing I think people need to be thinking about is how actual adoption happens again. It used to be that if you have a large enough market to play with, we just go after all of it. And, you know, some part of it starts to respond and we double down on that section. You're gonna be much more opinionated about how you imagine your technology being spread or adopted. How does it spread throughout our organization? Is it fit how does it do with the organizational structure and approach? Requires people to have more plan and strategy than they did before rather than just letting the market tell them the answer.

**Harry Stebbings** [38:46]:

Do they not go in contradiction? You said there about, hey, you know, people work on small kind of minute things on top of LLM, say. And then it's like, well, you need to be opinionated about who you're going after and who you're not going after. You need to be more targeted. Is that not kind of one in the same, which is like the verticalization of approach and the targeted approach being the core?

**Ethan Mollick** [39:05]:

Well, I think it's not about verticalization as much as opinionated. Right? I think you need to have a strong opinion of what the future looks like and where the gaps are gonna remain. This is a jagged technology. Figure out where you think there's going to be jaggedness, and that could be organizational jaggedness, interface jaggedness. But, I mean, you're also basically, the real problem right now is every startup in the world is betting against AGI, which I find really funny because all the funders are like, yeah, AGI is coming in next five years. If it is, why are you funding these startup companies? None of them are surviving an AGI world.

**Harry Stebbings** [39:35]:

For those that don't understand, why will none of them survive in an AGI world, Ethan?

**Ethan Mollick** [39:38]:

So a the common definition of AGI is a machine that's smarter than humans at every task. The machine will decide what to do. You're not gonna like, who cares about your stupid product? Right? Like, been making this for humans and getting product market fit. And then but the humans will say, you know, optimize my trading strategy or the AI will just decide to optimize your trading strategy. I mean, no one knows what AGI looks like. So I'm not gonna try and paint a science fictional future. But I will say there's a huge contradiction between a market and saying AGI soon And, like, we're funding a bunch of companies that are helping you. Like, already, I don't know if you play with them. Not that we're in anyone in your AGI with this, but, you know, you could tell Claude, come up with 30 ideas for a product to serve market x. Then rate them all on quality and feasibility level. Then create this is one prompt, by the way. Then create a playable prototype of the interface for the application. Then interview me as a user about how to change it and adapt it as we go. And it does it. I get a little playable interface for a game, and I can then edit the game and, you know, say, like, oh, I wish it was more it's just not fun enough in some way. And it's like, okay. Great. I'll make it more fun for you. If that cycle is really there, what is your stance and what an AGI world looks like becomes very relevant is all I'm saying.

**Harry Stebbings** [40:43]:

Do you find it interesting to see the different people's opinions on a especially on the founder side, different people's opinions on the time to AGI and their requirements for funding. And so what I mean by that is, like, Dennis and Zach are very long term minded in terms of how long it will actually take to achieve AGI, and they also don't need any money. And then there are other founders who I will remain nameless, who are pumping it as being much sooner, but they need to present that future because they need the money.

**Ethan Mollick** [41:13]:

I don't trust anything. People are very self motivated. Right? I think the signal you should pay attention to is that people are betting their careers to a large extent, right, on this being possible. And I think there are people who care about their reputations. That is a signal to me. They don't have be right. I mean, look, I work with Marvin Minsky, like I said. He was there in in the fifty seven conference where they out in Dartmouth where they outlined the concept of AI. I mean, we're in a world where, like, you know, AGI is always soon. So I think you would take everything with a grain of salt, but I think you need some coherence about your own viewpoint on this set of stuff. Now the large companies, I mean, we're seeing a lot of, you know, people warning that this is coming soon. I mean, in the in the meta paper, the the paper outlining the release of Lama 3.1, the keynote yesterday, it says we see exponentials continuing for the new we don't see any reason why exponentials are gonna stop. What does that mean for you as a startup feels like a relevant question. You're betting for a future world. So what does that future world look like? And you can't both say everything is changing, but also I'm doing this minor thing. I also think crypto did us dirty in this kind of front, which is like it made all technology feel like hype, and it emphasized again short buck return if you just believe something will happen.

**Harry Stebbings** [42:17]:

And then that's really a great way to think. I actually like Sam Altman when he said on the show, the simple kind of heuristic of, like, whether you're gonna get steamrolled by OpenAI is would you be excited or scared by a 100 x improvement in our model? If yes, then you're gonna get, you know, steamrolled. If no, then great. But I liked it as a heuristic.

**Ethan Mollick** [42:35]:

But I don't think it's useful as a heuristic. What does that mean? What what is a 100 times better GPT for like, it it is it's a baffling heuristic to say better. Like, what does that mean? Right? It's a uneven system. It has gaps in the world. That mean a 100 times better reason. Like, how are you supposed to like, this is what I mean. When you start looking at these things, it's like, what the heck am I supposed to do with that? It's a 100 times better. It's a machine god. Like, what? And so I I don't like this heuristic because I'm like, I don't have any way to operate within that. A A 100 times better. Does that mean it will be able to process an entire legal document and do a very good legal review of a document on its own? Great. That disrupts a huge industry, but that is a actual question about hallucination rates, its ability to handle words, you know, to think about words instead of tokens, to understand precedent, to be okay across different languages, to hold a huge amount in its context window. That feels like a useful question to ask. Could it write an academic paper on its own, right, where you give it a dataset, it generates hypotheses, tests them, writes a really good paper, formats in latex, writes a letter to the editor, and handles reviewer responses. We're getting close, but there's a lot of gaps there. Give me a concrete example of what this thing does, and then we could talk about a heuristic. But, like, a 100 times better is a really hard one.

**Harry Stebbings** [43:47]:

No. Those use cases are 28 times better. They're not a 100 times better, specifically 28 times. I'm joking.

**Ethan Mollick** [43:54]:

But but but maybe. But I mean, I think that's a valid question. Some of those things are huge gaps. Some of those things are small gaps. If you ask me about that, reviewing the legal document, not really a problem. Review, like, we're close to that. But, like, if it does that out the box, that also implies a lot lowering of hallucination rates below a threshold that they're not currently at. There's no benchmarks in hallucination. So we have no idea how good we're getting on the hallucination rate side. It also though implies the ability to seamlessly move between different perspectives. We could do that with agents today. Is it an agent based model taking action? Like, there's so many questions. I would love some specificity, and that's why I'm saying field specific is great. If you are a lawyer who knows law field really well, you probably might have some interesting things to think about and where the real gaps are or not. And I don't think a lot of the the AI firms know that. I know this because we're deep working with all of them on things like education. And, like, they don't really understand education. There's no educators there. So they don't really understand what teachers do, and they don't really understand what classrooms are for. And so it's all the AI workplace everyone, and I think we're a lot and we're not there.

**Harry Stebbings** [44:55]:

That was a trigger, wasn't it? Just give you give you a Sam Altman heuristic. Ethan, I was like, oh. Tell me. You said there about kind of the importance of being opinionated and for startups to have strong opinions about where, you know, AGI will be, how they fit into it, organizational design. If I were to ask you and flip that on you, where are you most opinionated in your views around it? Where would you suggest or point to first?

**Ethan Mollick** [45:18]:

So I think education is a good starting point. But in education, tutoring is the gold standard for for interventions based on the research we have. And AI is incredible one on one tutor. Like, it's transformative. But what I find Silicon Valley people and and AI and education people often think is like, well, once we have a really good tutor, we don't need teachers. Or like, I hated this subject in school. Or people will be self motivated to learn. Absolutely untrue. People are not self motivated to learn. And even all the computer scientists out there were like, I was. Like, yeah, you're autodidactic some narrow area, but you would have learned nothing about very important topics because you only cared about one topic. Right? People need extrinsic motivation to learn. It turns out that there's value in having an instructor guiding the direction of a class, that there's value in putting things into practice. So even an incredible AI tutor that knows you and loves you really well doesn't sub in for teachers. And also forget all of that. Let's talk about systems. Schools are in a complex system of society and where they are for, you know, providing daycare services to how they fit into education networks, how we do credentialing to the teacher unions to like, there's a billion things about schools that don't get replaced by AI by having a magical button you push to make stuff happen. There's gaps and opportunities that are very different than a naive view of how education changes.

**Harry Stebbings** [46:32]:

So one of the biggest problems in UK education today, I'm not sure if it's the same in The US, so you can tell me, but it's the exponential increase in class sizes that we've seen particularly in public schools, which is, you know, the schools provided by the state. And the quality of education has gone way down. When we look at AI's ability to increase education standards, will we see the ability to maintain high education standards with increasing class sizes? How do you think about that?

**Ethan Mollick** [47:00]:

I mean, I hope so, but let's just start. The the first randomized controlled trial we have by some of my colleagues at Wharton was giving a GPT-four people for math tutoring in Turkey. Now they didn't do a huge amount of, like, you know, it was an assigned class and they used the system, but it turns out that everybody who used it used GPT-four without any special prompting or anything else, had much higher homework scores, and they did much worse in the test. Because, basically, the AI just did the work for them. And once you have better once you have better prompts, that that effect disappeared, though we didn't see educational gains from it. But I think it's an early sign of, like, not being naive about how these systems operate. Right? Like, we need to put the work into building scaffolding. I absolutely believe that we can't be naive about the work that needs to be done here to make this stuff operate. So you can't just drop these systems in, but a good tutor will make a difference. I think in the long term, we'll have flipped classrooms where that 20 per where that giant classroom is actually fine because a lot of your learning is done outside of class, you know, AI tutor help. And then inside of class will be activity exercises application where large class size doesn't matter as much. But there is a road to get from here to there.

**Harry Stebbings** [48:05]:

I think this show has done well because I'm not scared to admit my own flaws and stupidity. Everyone talks about kind of the incredible optimism that AI brings for education and talks about tutoring. Great. But I don't really understand what that picture of the future of education looks like then. Does that look like when kids come home from school, they just have perplexed steel OpenAI up and they have another tutor with them, where as you said, in most cases, they end up doing the work for them and so they don't learn? Is it a crutch that they I I don't understand actually intangible reality. What does the future of education look like with AI, and why is it optimistic?

**Ethan Mollick** [48:43]:

So we actually have a lot of research on this. It turns out that, you know, first of all, there's a couple things you need to know about learning that people don't tend to think about, which is learning is hard and sucks. What makes you feel like you're learning isn't what's learning. Like, you have to do grinding work. There's no solution to it. It's just like any other thing like exercise or anything else. You have to be pushed to desirable difficulties where you're having trouble throughout failing at a thing you're not working hard enough. Like, there that's is it which why you often need intrinsic extrinsic motivation. And the second thing is we actually have some research. We know things like active learning, where you're in a classroom doing activities, beats the idea of passive just receiving a lecture. When we have those sets of pieces, there's been a move that kind of fizzled. It's called flipped classrooms that has some early evidence in its favor, which suggests this idea of like, classroom should be about doing stuff and outside of class should be about getting the basics. Because we can get you to do stuff in in the classroom setting. So that would mean that outside of class that what that practically meant is you watch videos outside of class, your teacher talking. So the lecture stuff is all outside of class. That's your homework. Read the book. Do that. Then your homework is in class where you can mess up in front of people and work in teams and you learn by kind of watching other people and how you're doing it. The teacher can help you solve problems. I think flipped classrooms are a very natural fit in active learning with AI based approaches. So instead of having a passive video you watch, you'll have an AI tutor outside of class. You'll log into the school's website and that tutor will be amazing. It'll be adapted to you. And then it'll pass that information on into the classroom setting where you actually the teacher gets advanced stuff. And by the we've actually built a version of this already at the generative AI Lab at Wharton. We'll be open sourcing all of that, like, that does this kind of stuff. It's not that hard to imagine. We just have a ways to go still.

**Harry Stebbings** [50:18]:

Is that really an order of magnitude improvement if we compare that post classroom? You could give me incredible high quality videos of you talking, lecturing, giving examples that you give to your students now, very easy to do, versus that AI tutor. Is it is it 20% better? Sure. Maybe it's personalized. But is it really an order of magnitude better?

**Ethan Mollick** [50:39]:

Education is a complex system. So I think order of magnitude's a very weird thing to talk about because every student has their own talents, abilities, interests, and gaps. The early work on in one on one tutoring, we don't talk about order of magnitude improvement because that doesn't really work in the education world. It's very hard to say what our order of magnitude is, but we can talk about grades a lot. The classic study that is probably would not be replicable, but it sets up our model is that one on one tutoring creates a two sigma increase in classroom outcomes. That's two standard deviations, which is a fairly huge improvement. You go from the fiftieth percentile to ninety seventh percentile of class. We have no idea if that's gonna hold up with, you know, AI tutoring. But if we could do that, that is is amazing an improvement as you could possibly ask for. And, I mean, a 10% improvement's amazing. I I kinda feel like we aiming for order of magnitude education, if we can get improvement in a system, we're in great shape.

**Harry Stebbings** [51:28]:

I also think that doesn't include a lot of different elements. Like, you mentioned extrinsic motivation being a big part of it. I think a big part of, like, having a tutor means you actually have a bond with them. You want to impress them. You want them to feel proud of you. Does that extend to an AI tutor where you don't have that humor?

**Ethan Mollick** [51:44]:

Maybe. It's not clear that that is the key to tutoring is the bond with a human being. Forcing people to confront what they don't know turns out to be a lot of the value of tutoring. So tutoring is often reflective back. So when we built a tutor chatbot, right, what that tutor chatbot like, the way we test, by the way, education technology, chatbots, our rule of thumb is that if it asks you if you understand a topic or you're ready to move on, it's a bad tutor because humans don't know when they're ready to move on or not. What the AI should be doing is asking you questions, probing what you know, and making you expand on what you don't understand and helping you fill those gaps. It's not the one on one bond. If there is a there are methods to teaching that we actually know make a difference. Self reflection makes a difference. Repeated practice makes a difference. Low stakes testing makes a difference. Like, to zoom back out to what we're talking about before, subject matter expertise is gonna be absolutely critical in making AI work. It's a system that experts I can look at a prompt in entrepreneurship and education and instantly tell you about whether that's gonna work or not or whether it's a stupid idea or a good idea or whether the subtleties that the system is missing are a problem or not because I am an expert. And if you're not an expert, you're gonna be like, that looks really good. So, like, expertise actually matters. I'm sure in the same way. You know, it's one of the things I actually when I talk to my students in, you know, and teach them how to pitch, right, one of the things I talk about is there's this really interesting research that shows that venture capitalists are not swayed at all by the quality of the the speaker. Their ability to be a good speaker is absolutely irrelevant. Amateur and angel investors are swayed by that. Why? Because you're an expert. You've seen so many pitches that you instantly see through all of that stuff, and you're like, you know what the core issues are right away because you've seen 10,000 pitches. You've seen how they play out in the world. You know, you have to be a really amazing speaker to pull off I'm persuasive on top of that. And so in the same way, think I expertise is gonna matter a lot here.

**Harry Stebbings** [53:30]:

When you look at the pervasiveness of AI and specifically ChatGPT in homework and in coursework and in the answers that many students give today, is there any point in university or educational facilities doing homework or coursework when it's largely done by AI today?

**Ethan Mollick** [53:46]:

Of course, there is. We like, everybody was already cheating. Like, if there's this great study at a repeated university that found that homework it improved when you did the homework, it improves in, like, 80% of people's test scores in 2008. And by 2020, it only helped twenty percent of people. And that's not because homework stopped helping. It's because everyone was cheating. And so we have ways around this. There there's really two options in how to use AI in education. One of them is to ban it cautiously. Alright? People are still gonna use this explainers and stuff like that, but you you have in class tests and blue book writing. Like, we've solved this problem in math. And, like, you make people do exercises and do work. Nobody likes it, but there's no shortcut to learning. It sounds dumb. It's like what your teacher said. Oh, turns out it's true. You need to do a grinding amount of work to understand something. You need to do interleaved practice. You need to like, there's a lot of stuff you need to do to learn something. And so we absolutely can make you do blue book work in class. We absolutely can install terrible monitoring systems. I don't like this approach, but, like, couple companies already have this. They watch what you're typing and make sure you're not pasting stuff in from AI. Again, I don't necessarily recommend it, but, like, these are possibilities. I think you are underestimating how much you can do those kind of things. Homework is valuable. Cheating is bad. What is AI cheating? We have to define that. I'm a big but the other option is transformation. My classes are a 100% AI based at this point. Students have AI mentors and tutors they talk to. They have AI based assignments. When they learn how to do hiring, I build a we build a simulator that actually makes them have to fake hire somebody, and the AI plays the person they're interviewing and gives them multiple choice answers, and they have to reflect on the assignment. There's one of the other assignments is they have to teach the AI to do something. You could do really exciting stuff. It's just not gonna happen right away.

**Harry Stebbings** [55:22]:

All of the different avenues, functionalities that we've spoken about require a lot of intense compute. Considering it was such a trigger when I gave you the last quote from Sam, thought I'd give you another one. Compute is the currency of the future, what Sam Altman said, and energy is a concern when looking at the energy requirements that this next generation of AI will bring. How do you think about the energy requirements required for this next generation of AI usage in society and whether Sam is right that compute is the currency of the future?

**Ethan Mollick** [55:51]:

That's what Sam believes. I mean, Sam believes in AGI, and he believes that it's gonna be achievable in the near term. Right? And when you talk to OpenAI insiders, they feel the same way. If that's the case, if intelligence on demand is the case, intelligence on demand is power hungry, and there's infinite demand for intelligence on demand because there will be. Right? If you have an AGI, I want that to be looking over all of my medical records and monitoring our airspace and, you know, finding scientific ideas and helping me with a project I have to do and also booking tickets for the ultimate trip. Like, there is infinite demand for intelligence. Right? So then compute becomes the currency and energy becomes the big deal. That that will make a big deal in that case, and we're gonna build a lot of nuclear power plants, I guess, in, you know, in relatively short order. It seems like that's a or or AGI figures how to do fusion, and it doesn't matter or we all get turned into batteries all in matrix, although we don't produce enough wattage. You know? So I I don't think that's really the issue. Training data, that's what the AIs will use this for. But, anyway, mostly joking. But right now, I think the energy debate's an interesting one because, again, it's one where doomers and optimists sort of like to talk about. Because on the downside risk, when I meet people who are skeptical at AI, the first thing they talk about is energy use. And the truth is that AI uses a lot more energy per query. We don't know exactly, probably two orders of magnitude than a Google search, but a lot less orders of magnitude energy than a human doing the same amount of work, right, with a laptop. You know, how do we balance those kind of things becomes an issue? Right now, 1% of US power goes to data centers and 10% of that goes to AI at most. So we have a lot of room left at the top before this becomes an issue. So again, we're assuming AGI is available, instantly useful, and in which case, absolutely compute becomes an energy becomes the the issue. But then that becomes the reverse salient. And, you know, there's a lot of money to be made that if if if the currency of the future is compute and compute is energy, then there's a hell of a lot of money to be made in building your own nuclear power plants.

**Harry Stebbings** [57:40]:

The final one before we do a quick fire. A friend of mine who's also a quite a well known venture capitalist, Jeff Lewis, said that when it comes to democracy, in the future, we will vote for algorithms, not for people. To what extent do you think AI pervades into electoral systems, electoral voting, the political fabric of our society?

**Ethan Mollick** [57:59]:

When something feels like a dystopia to most people, it probably is something that's not gonna happen very quickly. Human systems are complicated. I I just keep seeing this technological view, which is like, in a rational world, the machines will rule us all. It's just people don't want that. Right? So, like, we already have algorithms ruling lots of what we do. You know, your FICO score determines a huge amount of of, you know, things that happen in your life, and that's an algorithm. Like, we have these kind of systems in place, but the idea of an overall all seeing kind of approach, it's it's hard. Like, now on the other hand, we do find that AI is hyper persuasive already. Right? In a controlled experiment where you do where you're asked to be talk to a normal person versus the AI, you're 81.7% more likely to change your views to the AI's view than to a human's view. That is going to change marketing in very big ways, which is gonna change politics. Right? Deepfakes are going to are a big deal already. Although, it's been funny how little big big deal they are because it just turns out all you need to do is show a video of politician x talking and say, I can't believe he said he's gonna eat babies in minute three, and everybody shares it online who should know better and without actually watching the video at all. Like, when I post up a viral tweet, nobody clicks the link. I feel like we way overestimate people and therefore how much this stuff was gonna matter. But in a world where AI is hyper persuasive, this does change things. In a world where AI gets really good advice on everything, people should have an AI second adviser happening in every role including in politics. Right? That would make things better. But people aren't gonna listen to it. Politics changes much more slowly as much more human than people think. By the way, it plugs into larger issues of, like, when we can produce all this stuff on demand, what what is actually valuable or not? I mean, everything is gonna change. It's very hard to make predictions how a general purpose technology rolls out. But I do think people overestimate how quickly the short term change is gonna be. And as usual, from Moore's Law, underestimate the long term.

**Harry Stebbings** [59:41]:

I completely agree with you that. My biggest concern actually, you know, as a content creator in many respects is with the infinite supply of content, the value goes down and discovery becomes much more challenging. That is a big concern.

**Ethan Mollick** [59:54]:

I mean, that problem has already happened. Right? I mean, you know, to me, the really interesting thing is, like, Suno and Odeo and com like, they're getting pretty good. At what point does having an AI generated song playlist, you know, that has a couple real musicians but also makes up songs based like, that doesn't feel as far off for in terms of people enjoying it. Like, what happens to content creation is a very big deal. Right? I mean, like, I'm an author. My book's a New York Times bestseller. That's amazing. Don't I think people realize how few copies you need to be to be a New York Times bestseller. Like, you're if you're, like, you're selling, like, 6,000 hardcover copies in a week. That's getting on the New York Times bestseller list. Attention's already scattered across a huge amount of content. The one thing you'd hope for is maybe AI creates better connections, right, in, you know, in some ways. I'm not being rude, but could you not just buy

**Harry Stebbings** [60:38]:

I know you haven't, I could could I not just do it, do a book and spend $75,000 and be a New York Times bestseller then?

**Ethan Mollick** [60:45]:

So people do that all the time. The New York Times has a small cabal of people who refuse to talk about how they do this. So they use the number ranking, but then they also try and exclude bulk buys. They actually try and cut that out. So those you'll notice there's a little dagger next to the name of, of companies in the bestseller list that they think that they're including, but they still had potential bulk buys. I actually got the little dagger on mine because a company bought 500 copies, which wasn't the main reason for the list, but they would have found that suspicious. They're trying to filter that out by hand. But, yes, you can often buy your way into the list and people do do that all the time.

**Harry Stebbings** [61:17]:

Yeah. No. I've seen many of my friends who have VCs who have books, and I'm like, really?

**Ethan Mollick** [61:21]:

There there are ways of doing this. You scatter buyers across multiple locations and they all do buy like, it is very much true that a lot of your unnamed VCs do seem to have asked a lot of friends to buy book copies of their book.

**Harry Stebbings** [61:33]:

Amazing. I love that. Listen, Ethan. I could talk to you all day. I wanna do a quick fire round. So I say a short statement, you give me your immediate thoughts. Does that sound okay? Sounds great. What do you believe that most around you disbelieve?

**Ethan Mollick** [61:45]:

I feel like the very simple idea that AI is very profoundly is much better than people think and is gonna keep betting better is something that I think most people don't actually believe. What's the most concerning future

**Harry Stebbings** [61:55]:

that AI could bring?

**Ethan Mollick** [61:56]:

The most concerning future, I think, is one where we lose agency and not necessarily to the AI systems, but to the systems that incorporate AI. What I mean by that is we have a chance to make AI be used for human thriving. That's not an automatic process. Right? That means not firing people when you have AI in in your company, but it means figuring out other uses for them that are valuable. It means building systems that help people feel like they're accomplishing more as a result of using these things. And I worry we're not seeing enough people modeling that kind of behavior. It's all about just the technology itself and then how do we get cost savings. What have you changed your mind on most in the last twelve months? I have gone back and forth on how much juice the technology has left, and now I'm back to the it has lots of juice left. Like, the exponential continues for a while. And I think I was not clear on that for a long time. What caused that shift backwards? Accumulation of evidence. Right? So we talked about Kevin Scott saying scale. Like, there's a bunch of people who weren't talking about scaling solving everything six months ago or eight months ago who are now more confident, which indicates to me another generation of models came out and everyone at all the labs are getting that haunted look in their eye again. I don't know when we'll see these models, but they're clearly people are seeing things that indicate to me that there's more more left in the curve, and they're all talking about it.

**Harry Stebbings** [63:07]:

Do we see all large players and incumbents move into the chip player? We've seen Apple move into the chip player, internalize margins, remove reliance away from NVIDIA. Do we see that as a large shift in all providers?

**Ethan Mollick** [63:21]:

Your requirement in any supply chain pipeline is to eat the value. If you're you know, like, that's the whole idea of, like, how's how, you know, how those things work. So, like, if the if you're spending a lot of money on chips, you we can't go into chip making. Just like if you're spending lots of money on, you know, warehousing, you figure out a way to, you know, reduce your warehouse costs. Like, they're gonna figure something out. What elements, sorry, of AI development has most positively surprised you? How clever these things are. Like, I'm having like, if you haven't seen my Twitter feed where I asked the AI to remove references about squid from the novel, Al Quine and Western Front, would just look for for that because, like, these systems are really clever. They're they're kinda joyful to use, and I I think that that's kinda surprising. Final one for you, Ethan. What question are you not ever asked that you think you should be asked more? The question that I think people should be asking and that I don't have an answer to yet is why are so many people bouncing off these systems? Like, why are they not you know, why are so many people using them a little bit and not forever? Because I don't think it's just simple, like, it didn't work. People are getting kind of freaked out by these things in ways that we don't really understand how humans are relating to these tools. Like, we always talk about the systems, the technology, how industry is gonna change. And I don't mean just like the dating thing, tends to be like, well, you have a relation with that. But, like, how are we relating to these kind of tools? That's one of the questions. Let me do a second take on this. The thing that I would be thinking about, because it's related, is meaning. People don't ask me enough about meaning of work. That matters a lot. You know, Graber's bullshit jobs, think, was mostly not correct based on survey data and other stuff we saw, but it's real. People do feel inundated from work. People do like, most employees say they're bored at least 10% of the time at work, but they're doing work that they feel is meaningful. When you survey people, most people think their jobs matter in the And what's going to happen that I'm very worried about is when you realize as a middle manager that AI does your work and nobody cares, what does that mean for the nature of work? How does that matter if people don't care? Like, if AI subs in and does stuff, if your boss is responding with the AI answer to the emails you send them. And I think that that meaning crisis is one we're not talking about enough. It's one thing we replace in a job. It's another to semi replace yourself and realize why am I doing this. And I think that's gonna be a bigger issue that we're not talking about.

**Harry Stebbings** [65:31]:

Ethan, listen. I apologize for going in so many different directions. I so appreciate you putting up with some of my base questions, but this has been fantastic. And for me, as a lover of your writing, it's been a huge pleasure, my friend.

**Ethan Mollick** [65:42]:

This has been wonderful wonderful too. Please don't tell Sam Altman I disagreed with him because he's building a machine god, and I don't want him to be angry with me.

**Harry Stebbings** [65:51]:

As I said at the beginning, Ethan's writing is some of my favorite writing on AI. So you if haven't checked that out, you can check that out by checking out Ethan Mollick on Substack. And if you wanna watch the full interview with video, you can find it on YouTube by searching for 20 VC. We always love to see you there. But before we leave you today,

## Sponsor read

**Harry Stebbings** [66:07]:

when a promising startup files for an IPO or a venture capital firm loses its marquee partner, being the first to know gives you an advantage and time to plan your strategic response. Chances are the information reported it first. The information is the trusted source for that important first look at actionable news across technology and finance, driving decisions with breaking stories, proprietary data tools, and a spotlight on industry trends. With a subscription, you will join an elite community that includes leaders from the top VC firms, CEOs from Fortune 500 companies, and esteemed banking and investment professionals. In addition to must read journalism in your inbox every day, you'll engage with fellow leaders in their active discussions or in person at exclusive events. Learn more and access a special offer for 20 VCs listeners at www.theinformation.com/deals/20vc. And speaking of incredible products that allows your team to do more, we need to talk about Secure Frame. Secure Frame provides incredible levels of trust to your customers through automation. Secure Frame empowers businesses to build trust with customers by simplifying information security and compliance through AI and automation. Thousands of fast growing businesses including Nasdaq, AngelList, Doodle, and Coda trust Secure Frame to expedite their compliance journey for global security and privacy standards such as SOC two, ISO 2,701, HIPAA, GDPR, and more. Backed by top tier investors and corporations such as Google, Kleiner Perkins, the company is among the Forbes list of top a 100 startup employers for 2023 and Business Insider's list of the 34 most promising AI startups of 2023. Learn more today at secureframe.com. It really is a must. And finally, a company is nothing without its people, a global law firm built around startups and venture capital. Since forming the first venture fund in Silicon Valley, Cooley has formed more venture capital funds than any other law firm in the world with sixty plus years working with VCs. They help VCs form and manage funds, make investments, and handle the myriad issues that arise through a fund's lifetime. We use them at 20 and have loved working with their teams in The US, London, and Asia over the last few years. So to learn more about the number one most active law firm representing VC backed companies going public, head over to cooley.com and also cooley go dot com, Cooley's award winning free legal resource for entrepreneurs. As always, I so appreciate all your support, and stay tuned for an incredible episode coming Friday.