Scale's Alex Wang on Why Data Not Compute is the Bottleneck to Foundation Model…

Yeah. I think it’s pretty fascinating. I mean, I think there’s been this is especially coming up now where OpenAI has had GPT-four since fall of twenty twenty two. And since that time frame, we haven’t yet seen a new base model or a new a new model that’s jaw droppingly better than GPT-four. You know, we haven’t seen the GPT-four point five or the GPT -five or the other labs having yet come out with models that are leagues and leagues better than GPT-four despite way, way more compute expenditure.

Since when ChatGPT came out, you know, you can look at the graph of NVIDIA’s revenue, and it just inflects. It’s just like it just goes straight up after GPT four came out, and it goes from I think the NVIDIA’s data center revenue, they were doing roughly about 5,000,000,000 a quarter, and then it shoots up to now it’s, you know, north of $20,000,000,000 a quarter. So there’s been tens of billions going to 100 more than a 100,000,000,000 of spend on high end NVIDIA GPUs, you know, all in the same time frame.

We haven’t yet seen the big breakthrough since GPT-four, which actually that model was came out before this huge inflection in NVIDIA expenditure. So overall, it’s this interesting thing where we’re seeing investment into compute go up dramatically, go up exponentially right now, but we’re still, I think, as a community, as an industry, kind of waiting for the next great model.

It’s kind of this interesting thing. So there’s three ingredients that go into these AI models or three pillars. So there’s compute, of course, there’s data, and there’s the algorithms. The history of AI is that progress comes from sort of all three of these pillars sort of being built altogether. You certainly need a lot of computational capability, but you need the algorithmic advances like the transformer originally or RLHF or, you know, whatever future algorithmic advances come. And then you need the data pillar to support it as well.

And I think a lot of the plateau that we’ve recently seen can almost be explained at a very high level from hitting kind of a data wall. The GPT-four was model basically trained on nearly all of the internet and using a huge amount of computational capabilities. And a lot of the a lot of, I think, what the industry has been doing over the past few years is scaling the computation computation dramatically, but not necessarily by building up the other two pillars in tandem. So there needs to be, I think, a combination of more algorithmic improvement, but in particular, we need to ensure that there’s more data to support it.

Yeah. So at a super high level, we’ve used up all the easy data. We’ve used up all of the internet data, the common crawl and newer versions

Anything that’s easy and free to crawl or stuff that can be torrented. There’s a lot of, you know, reports that there’s like a lot of torrented data in some of these models or, you know, basically anything that is sort of, like, already written down and easy to get from the open Internet. Mhmm. And then the first stage of a lot of this AI improvement has been this these advances in pre training, which is basically training these models to be really, really good emulating the Internet. And right now, we’re at a point where, like, these models are exceptionally good at emulating the Internet.

They’re better than any human at emulating the Internet. But the problem is when we think of AGI, when we think of powerful AI systems, we want much more than just emulating the Internet. You want AI systems that can do tasks. You want AI systems that can solve difficult problems. You want AI systems that humans can collaborate with to solve all their daily problems. This process of building agents and AI models that are capable of all these things, know, we’re not gonna get there from Internet data.

And we’ve already used up all the Internet data.

The simple answer is like a lot of the thought process and a lot of the thinking that humans go through when they are doing more complex tasks, that doesn’t get written down on the Internet.

So for example, if I’m a fraud analyst inside a a large bank, my job is like understanding based on a set of transactions that seem suspicious whether or not it’s a fraudulent transaction, and I need to analyze all sorts of different pieces of data and use my deduction, and use all my human intelligence to make that decision, that process that I go through, it’s not like I’m writing down step by step, like, oh, I looked at this piece of data, and I looked at this piece of data, and then based on that, I deduced this.

I’m not writing all that down on the internet to later be crawled by these models. One way to think about it is like, all of the reasoning and thinking that is powering the economy today, none of that gets written down on the internet. And so if you just train on the internet, the model has no ability to learn from all of that.

What I really believe is what we need from now forward is frontier data. We need to basically have data abundance of frontier data, where right now we’re in a sort of data scarcity mindset or a we’re we’re hitting a data wall. And this frontier data is exactly what we’re talking about. Frontier data in my mind is, you know, complex reasoning chains, complex discussion, agent chains of models going and looking up a piece of data, doing some reasoning, looking up another piece of data, maybe correcting if it has an error, tool use, all of the key components that we would think of an agent being able to do, that all needs to be encapsulated into the frontier data to power the four capabilities of these models.

How do we capture that data? There’s basically three pillars. So first is, there’s a lot of this data that’s locked up in the world’s enterprises today. And none of that gets on the Internet for very good reasons. But just to give a sense of scale, you know, JPMorgan’s proprietary internal dataset is a 150 petabytes. The GPT-four was trained on an Internet dataset that was less than one petabyte. So the amount of data that exists inside large enterprises is just absolutely astronomical. So there’s one process of just sort of mining all this existing enterprise data for all the goodness that exists within

Exactly. This has to be a process of like every enterprise, like I have a set of very important problems for my enterprise. Yeah. Then I need to go through process of, like, basically mining all my existing data and refining all that existing data for use for AI systems to solve my own problems.

Yeah. I actually think that, you know, if you look at what these models can do, they’re very good at reasoning in situations where they’ve seen a lot of data before. You know, I think we like to think about these AI as if they’re like little human intelligences, but they’re they’re very different. Human intelligence and machine intelligence are very different. Humans are very general form of intelligence. If a kid is raised in a very small neighborhood, they can live their whole lives in that small neighborhood, and then go to an entirely different part of the world, and they can navigate, understand what’s going on.

No AI system today would be able to do that level of sort of drag and drop in one situation to another situation and figure out what’s going on. Yeah. I think we have to be cognizant that that’s a limitation. But what that means is that for any situation that we want these models to perform well in, we need to have data of that situation or that scenario, and actually, the model will perform really well. So there’s kind of two ways to think about resolving the reasoning gap that exists in these current models.

One is, obviously, you build some sort of general reasoning capability, which would definitely be a big breakthrough. The other one is just it’s a data problem. It’s like you need data for every scenario where you want these models to reason well in. You just need to overwhelm them with data in all those scenarios, and you’re gonna get models that can reason really well.

Yeah. So I think the second part is, to your point, new data that has to be produced. Yeah. We need the means of production of new frontier data to get us from you know, GPT-four to GPT-ten. I think when we think about chips, this is very natural, which is, oh, yeah, we need to build more and more fabs. We need to build bigger fabs. We need to, like, increase the resolution and get lower, lower nanometer fabs. Like, for compute, it’s very natural for us to think about increasing the means of production.

Yep. But I think we don’t think about this with data. And I think we need to do something very similar. And this process of producing data, it’s sort of a hybrid human synthetic process. And that’s really how we think about it, which is you need algorithms that can do a lot of the heavy lifting in producing synthetic data, but you need human experts who are going to be able to guide the AI systems, and basically help provide input as to, you know, when the AI system gets stuck, or when they have a factuality issue, or when it’s in a situation where it hasn’t encountered before.

A lot of autonomous vehicle scale up has been through the safety drivers. You know, you have safety drivers inside the car, and when the car starts screwing up, you have the safety driver disengage and sort of take over. And you need that kind of setup for these AI systems. You need you need AI models to be generating large amounts of data, and then humans who can kind of take over and nudge the models when necessary to make sure that you get really high quality

Yeah. Yeah. Trainers is one term. AI trainers or contributors is another term. For what it’s worth, I think this process of of contributing data to AI is actually one of the highest leverage jobs that humans can have. And the reason for that is, let’s say I’m a mathematician. I can either go into my hole and, you know, do pure math and try to do pure math research. That’s one trajectory for my life. The other trajectory is I use all my skills and talents and intelligence to help make these AI models smarter.

Let’s say I make GPT-four just like a little bit smarter on math. If I take that little bit of improvement of the model and I sum that up across all the times that g p d four is gonna be called and used across every math student who’s gonna use g p d four, every company that’s gonna use g p d four, every developer that’s gonna use g p d that’s a huge amount of impact. And so as a human expert, you have the ability to have society wide impact by producing data to help improve these models.

What we see is for scientists, mathematicians, doctors, human experts in the world, it’s an incredibly exciting proposition to be able to I can transmit my capabilities, intelligence, training, all of that into a model that’s going to be able to have society wide impact. I mean, it’s an incredibly exciting proposition.

So again, I think this is a case where there’s two parallel efforts. One is mining existing data, which by all means is going to be a one time hit. There’s going to be a one time benefit that you get from mining all your existing data, and it could be really meaningful. Do you think in five years time, everyone will have mined their largest data sources internally? Don’t I think everyone will, but certainly the most sophisticated companies will. Okay. And then and then we’ll be at a point where we still need to make the models better.

Yep. At the end of the day, it’ll all boil down to data production. What are the means of forward production in the same way that, you know, you need the means of forward production for chips and and all the other things that you care about.

So there’s data mining, and then there’s forward data production. Yep. These are the two core core directions for where we need this data to come from. And I think kind of taking a a broader step back, I think that a lot of AI progress at this point is fundamentally more data bottleneck. If we were able to produce compute and data in lockstep with one another, so as NVIDIA continue to manufacture hundreds of billions dollars worth more of chips, if we were able to produce a proportional amount of data as we got more and more chips and we were able to produce these two together, then we would get astronomically more capable of

So there’s probably two main pieces. One is like this effort from Limitless or other efforts, which is basically much more longitudinal data collection. Collecting more of what’s naturally happening in the world. Mhmm. There’s a bunch of forms of this. So one is like, in a workplace, I think you’re gonna want, you know, as creepy as it sounds, some kind of constant data collection of what apps are you using, what order apps are you using, you know, where do you copy paste one thing to another thing?

You have a

Yeah, yeah. So so process mining, which is one of the terms in SaaS. But basically, like, the continued collection of existing enterprise processes. Mhmm. Then there’s the consumer version of that, which looks kinda what you’re referencing, or maybe it’s with the Meta Ray Ban collaboration, or whatever device ultimately does it, but sort of something that collects, you know, the the longitudinal view of your own life. And then there has to be a real investment towards human experts collaborating with models to produce frontier data.

So both things I referred to before, both enterprise process mining and, for lack of a better term, consumer data collection, those are all gonna produce valuable data sets, but they’re not gonna produce the data that’s actually gonna push the models forward. Because to push the models forward, you need really highly complex data that’s going to be able to push the frontiers of what the models can do. So this is where you need the agentic behavior. This is where you need the complex reasoning chains. This is where you need advanced code data or maybe advanced physics or biology or chemistry data.

These are the things that really needed to push the the boundaries of the models. I think this is a global kind of infrastructure level effort that needs to happen. Like, I think we need to think about it as how do we get the world’s experts to collaborate with the models to help produce AI systems that are going to be the world’s best scientists or the world’s best coders or mathematicians.

Yeah. Well, I actually think, to your point, if you think about the competitive playing field of these different model providers against one another, there’s three pillars. Right? There’s algorithms, compute, and data. And data, I think, is actually the primary pillar that you can imagine a real durable competitive advantage emerging. So if you think about where are their moats in this LLM race or where are their moats in this foundation model game? I think data is one of the few areas where you can produce a sustainable moat.

Because the issue is algorithms, that’s IP that at some point, the rest of the industry will learn about. You know, you can have more compute than other people, but other people can just spend money and buy that compute. And data is one of the few areas where you can actually produce a a long term sustainable competitive advantage.

Yeah. Exactly. And I think this is the it’s the start of this form of thinking of sort of data as a moat. You know, there’s the Feet, there’s Accel Springer. These are the first indications of this. But in the future, these labs are gonna be thinking a lot about, okay, what’s the data that I’m gonna use to differentiate relative from my competitors, and how am gonna produce that data, and what is the long term durable advantage created by that? I actually expect that, you know, everything we’ve been talking around data around model commoditization, we’re gonna see companies start building data strategies that drive more differentiation in the market over time.

I mean, another way to think about this is, you know, right now, in San Francisco, researchers and the big CEOs brag about how many GPUs they have. The biggest indicator of how serious they are about AI is is how many GPUs they have. But I think in the future, they’re gonna brag about, you know, what data they have access to, how much data they’re producing, and what are their sort of unique rights to to different data sources. And I think that’s actually gonna be the primary plane of competition in the future versus just, okay, Jensen’s giving me however many hundreds of thousands of GPUs.

There’s two futures. One is that even data strategy becomes something that very quickly commoditizes and and different labs sort of copy one another, or they all they all end up converging to the same direction.

Yeah. Different labs need to have strategies to produce their unique datasets. Let’s say Anthropic, for example, has focused a lot on enterprise use cases. And maybe they need to develop a data strategy that enables them to have a very differentiated access to new data to support those enterprise use cases. Or maybe OpenAI with ChatGPT needs to develop, you know, a unique data strategy that lets them leverage the fact that they have all these users and all this reach. The various labs are gonna need to lean into where they’re gonna be able to get proprietary and differentiated data going into the future.

This is a super interesting question. I think when we talk to Thank you, Alex. I think when we talk to these large enterprises and the leaders within these enterprises, they are very quickly realizing this fact that you stumbled on, which is that their enterprise data might be their only competitive differentiator in an AI world. They’re extremely, extremely cautious about, you know, if they do a deal where all their data somehow a model developer gets access to it or they share it in some way, know, they could be mortgaging away their entire future.

And so I think they’re they’re very, very cautious about that. And this is actually why I think there’s a very big sort of opportunity for whether it’s open source models or the LLMA models or the Mistral models or whatnot. Basically, these models that that can go on prem and that enterprises can take and then customize on top of their own data, and then it never has to go back to a model developer or cloud or anything like that. I think that there’s huge unmet need there.

And I think that’s actually where most serious serious enterprises enterprises are are gonna go towards, which is I need really, really strong guarantees that my data is not gonna be used in any way to improve my competitors.

First of all, I think you’re right. There’s so much value to be generated from AI for sure. But then there’s this natural question of where’s the value capture gonna be. Right? You know, it’s this fascinating thing. You know, if you go back and read High Output Management by Andy Grove, there’s like there’s these chapters around, like, oh, you know, for Intel, like, first we thought that, like, this is where the value capture is gonna be. But then we realized it was gonna be in this other part of the stack, and so we had to migrate to that part of the stack, and then we had to migrate again.

And it’s this incredible case study. You know, I remember reading that, and I read it maybe a decade ago in a different era of tech, and I was like, this is weird. This doesn’t feel very relevant. And then now in AI, you’re you’re seeing it once again, where it’s so new and so nascent, where exactly where in the stack value will accrue feels like it’s constantly moving. And I agree with you. I think that the models themselves, there’s so much competition there. I don’t know how much value accrues at at literally the model itself, but everything above the model and everything below the model, I feel very confident there there will be value occurring.

So for the infrastructure, I mean, NVIDIA is the biggest company built on AI today. Like, they’re they’re the third most valuable company in the world. NVIDIA is more valuable. You know, their market cap is higher than Meta and Google and Amazon and Saudi Aramco. I mean, it’s really staggering. Like, NVIDIA is an incredible, incredible company. And that’s below the model. And then above the model, you have all these apps and these services that are gonna be built on top. So

Yeah. So our thesis on this is there was this article that that flew around the end of software. Right? I saw this. Chris

You’re from Chris Pike. Yeah. It was an intentionally provocative point of view, I think. Mhmm. Sorry. For those that haven’t read it, what was the core premise just so they understand it? I thought it was, like, a brilliant comparison. But he basically drew this comparison of software companies today to media companies, pre social media. The rough comparison was in the older days of media, you had all these incredible media companies. There were, these high end shops where there were, like, all these experts producing this very differentiated content, but then it got disrupted by social media and the internet broadly, because all of a sudden you just had the content distribution costs came down dramatically.

The world of media consumption turned into this like very broad constellation, where you would consume whatever media was produced by anybody that was interesting to you and was sort of, like, much more on demand versus being this sort of walled garden of large media producers.

And basically, this comparison to that’s what’s about to happen to software, which is now the enterprises live with this walled garden of some small number of software providers, and what’s gonna happen now with generative AI and all these other trends is they’re going to have this constellation of all these different apps and point solutions, and this sort of portal to that constellation of various software providers, and this sort of like, we’re gonna move from this current world of like smaller number of walled garden SaaS apps to this sort of much more decentralized universe.

Do you agree with that? It it it’s intentionally provocative. Right? But I think one thing that is true is I do think that enterprises and the world writ large are going to demand greater levels of customization. They’re going to demand greater levels of personalization and stuff that is that is really purpose built for their business like a glove. The first tech company that ever did something in this direction was Palantir. You know, they got a bad rep for a long time, because everyone thought that Palantir, oh, they’re just a consulting company.

But Palantir’s point of view, which is provocative as well, was like, no, what we’re gonna do is we’re going to go into enterprises, we’re going to understand exactly what their problems are, We’re gonna help them build the perfect application for them that’s built on top that that connects all their data and all that stuff. And if we can do that, then we’re gonna build something that’s that’s far more valuable for them than what any other software provider is going to be able to produce. And they did this, obviously, before generative AI, before all these, you know, tools that are gonna make this this motion a lot more feasible.

But I do think there’s an element that this is, like, the way the world is moving, which is now, especially that the software production costs and software creation costs are going down so dramatically, we’re going to end up moving towards a world where more and more of the software that enterprises consume are going to be customized and custom built and purpose built for exactly their problems.

Yeah. Well, I think software engineering in general is gonna change dramatically. A lot of what developers spend a lot of time on today, they will not need to spend time on going into the future as the models get better and better at coding. But there’s certainly big parts of what they do which are irreplaceable. And over time, I think that the part in particular that’s very, very valuable is this sort of like general process of going from what are my customer problems or what are the sort of like problems I need to solve, and like translating those into engineering problems.

And scoped sort of like tickets almost that can be solved by an AI engineer.

The reason that per seat pricing doesn’t make sense going into the future is that at an enterprise today, certainly, most of the productive work is done by by their employees, done by people. But in a future where you imagine more and more of the work is done by AI agents or AI models, then perceived pricing doesn’t really make sense. As a provider of software, provider of of solutions, you wanna make sure that you’re capturing the value that you’re providing to the people, but also the value that, you know, your agents or your AI systems are producing.

That shifts shifts a lot of the world towards consumption based

It’s a really important point. And I think that that certainly what we’ve seen in the EU is a very restrictive approach to data. My personal belief, I don’t think that more permissive regulations around data are at odds with being a liberal democracy. More sort of liberal data access provisions are in fact very compatible with being a liberal democracy. And I think that we as a society need to figure out what the right balance there is and and how we sort of square the circle.

But I think I think this is a very important question because it’s almost like I think in in The United States, there’s been a huge amount of effort and real regulatory effort on terms of how do we ensure we do not slow down chip production? How do we make sure that we can keep manufacturing huge amounts of chips and The US won’t be disadvantaged from that perspective? We need to take a similar lens to data. So how do we, from a policy standpoint, both in The US and in The UK, frankly, how do we think about ensuring that as countries, we’re not tying one hand behind our backs for future data production for these models?

Do you think The US is currently tying one hand behind its back in terms of that? I’ll put it this way. We’re definitely not taking a pro data regulatory. What would a pro data regulatory stance look like? I think there’s a few things. I think first, there’s large data sets that do not lend proprietary advantages to specific players that need to be sort of centralized and made accessible to whole industries. So simple examples, safety data in, let’s say, aerospace, which is a which is a hot topic, obviously.

But safety data in aerospace should be collectively pooled for the purpose of advancing the entire industry forward. Or the data I mentioned before, or the the example I mentioned before, fraud and compliance in financial services should be pooled together and should build forward capabilities. So I think there’s, like, entire industrial sectors where there should be some degree of data pooling to just push forward the overall industry. And I think what you need is, in a lot of consumer facing areas, you need to we need to work through a lot of the existing restrictions to make sure that those don’t prevent AI progress.

So one great example here is actually HIPAA in healthcare and all the PII and other other, limitations. Right now, HIPAA and p I and all the PII regulations will more or less prevent patient data from being used to train AI models. But I think we can agree, as a civilization, as a human race, we really wanna learn from all of the existing medical data on how do we cure human diseases going forward. And so we need to figure out, like, how are we going to make it so that, like, there’s very clear anonymization provisions or there’s a there’s a very clear and obvious way in which you can use existing patient data to improve future health outcomes.

Two years ago, they were probably more than two years behind. When OpenAI first produced GP four in the lab, China were nowhere near that. But just even the past few months, there’s a Chinese company, Zero One Zero One dot AI, that produced a model yi large, y I dash large, that is one of the best models in the world. I think it’s just behind so it’s behind GPT-four o, and Gemini, and Claude three Opus. And it’s the next model right behind that in the leaderboards. So it’s one of the best models in the world.

So we’ve already seen them meaningfully catch up. They are like Chinese LLM and AI capabilities are, I would say, now, pretty close to neck and neck with US capabilities. And I think if you plot the path ahead based on everything we’ve talked about with data, they have a clear shot at racing forward and racing ahead of us. It comes down to, at its core, the CCP system is incredibly good at taking very aggressive centralized action and centralized industrial policy to drive forward critical industries.

And what we’ve seen even in the past few years on or the past few decades, frankly, on solar, how the CCP has been able to make take industrial policy to the point of, like, being, by and large, the world’s leader in solar, and then most recently, EVs, and how the CCP system and approach has been able to create very, very cheap EVs. You’re seeing this pattern play out over and over again, where the CCP approach to industrial policy is not the most innovative. But once an industry has been established, and it’s about, you know, turning the crank, they are better at turning the crank than any other economy in the world.

worries me a lot. You know, one of the elephant in the room topics, which I think, you know, as an AI community, we rarely discuss, is that at its core, this AI technology has the potential to be one of the greatest military assets that humanity has ever seen. Let’s say you had AGI, and you have one country with AGI, and another country without AGI, which one will win in a war?

Well, probably the one with AGI is gonna, like, figure out all how to produce all the weapons, or will figure out a brilliant military strategy, or will people hack the other country’s systems, and it is potentially one of the greatest military assets the world’s ever seen, potentially even more of a military asset than nukes. And so if you think about this, you know, we’re in a geopolitical environment that is increasingly tense. Know, the amount of conflict in the world is has been monotone and increasing the past few decades.

You’re seeing these multiple wars being fought in the world, and some of them without very clear paths to resolution. And there are totalitarian leaders right now in the world, many of them, for whom, like, let’s say China or Russia had AGI today, and The United States didn’t, I would imagine they would use that to conquer. That’s a really scary outcome for the world writ large. And I think it’s one that the Western world needs to spend a lot of our thought and effort

I think there’s a bit of a dichotomy that must emerge. I think we need to think about the most cutting edge and the most advanced systems, those we will want to ensure are closed for geopolitical reasons, for military reasons, for whatever reasons. Like, as we develop systems that are that are genuinely so so powerful, we will wanna keep those closed. That doesn’t preclude us from making open, less advanced versions of technology that frankly just have the ability to produce a lot of economic value. And I think that’s where we are with LLMA right now.

Like, LLMA three in and of itself is not a military asset yet. And I think that there’s clearly a line underneath which I think it’s totally fine to have open models. So that’s what we need to be thoughtful about, is where that is and when are we getting close to it.

I think at its core, what we’ve seen about the foundation model race is that it is incredibly expensive. And it is expensive to the level of, you know, these models have gone from costing hundreds of millions to $1,000,000,000 to maybe multiple billions. I think in ten years time, maybe they’ll cost tens or hundreds of billions. There’s just not very many entities that have that much discretion and capital to invest into these AI models. So naturally, what will happen over time is AI effort, the foundational efforts will coalesce around nations or the large tech companies over time.

Basically, you will see the most all these hyper profitable business models, whether that’s a nation state or one of the hyperscalers, those will be the only entities that could possibly subsidize or underwrite these massive AI programs. In the future, already, it looks like a battle of giants, but you but at that point, it’s even more a battle of giants.

Yes. Within maybe an asterisk that there’s some of these partnerships that I think it’ll be interesting to see how they play out. You know, the OpenAI Microsoft partnership or the Anthropic Amazon partnership. One of the most interesting questions of this technology era is how do these partnerships actually end up playing out long term?

At its core, the traditional press industry is not particularly conducive to great companies being built. And let me be more specific around that. A lot of traditional press, you know, is oriented around generating clicks. And so the traditional press engine will it’ll build you up, and it’ll generate clicks on the way up, and it’ll tear you down, and generate clicks on the way down. This is in contrast to, I think, 20 VC and other direct outlets, so to speak, where founders and companies have a direct channel to get their message out and and explain what they’re working on.

Exactly. Yeah. You’re you’re there to to genuinely educate and explain what’s going on to to your audience.

Yeah. It’s pretty stark. You know, I feel like I’ve received more fair treatment testifying in front of Congress than I have from various media outlets over the years. It feels like this totally ridiculous statement, but I think we’re in this perverse state of of a lot of traditional media where the system itself, you know, because of this sort of, like, very click oriented approach versus a genuine educational approach, it almost has no way of being fully fair to the companies. And so I think it’s the imperative is on the companies themselves to properly tell their story through direct channels and through podcasts and through avenues where their their message won’t be altered.

It’s kind of a shocking state of the world, but I Has that changed your

Yeah. I think for us, we think a lot about how do we get the direct direct message out there, and how do we develop, to your point, like, what are what are the purest ways that we can transmit and explain what we’re doing? And this is this is a great example. You know, you’ll ask me a question, I will answer exactly how I believe that I think, and this will go out to your to your listeners and your viewers. One of the purest

I think this is true. I think there’s like so few companies that can OpenAI is one where like, I think OpenAI is an entity has like a lot of has a lot of meaning as a brand. But very

That’s fascinating. I mean, it it probably speaks to a deep a deep human need. I think we, as people, we have a lot of circuitry to understand individuals. We have this ability to understand individuals. It’s very hard to understand what an organization means. There’s no intuitive So

I would argue no. I would argue we’re in an era now where they they shouldn’t. They should think about what is an interesting point of view they can have? What is the most pure way to get that point of view across? When do you feel the press tried to tear you down unfairly? We’ve had I would say that, like, almost precisely, we’ve had this this story where we had an incredible rise up and an incredible come up. Maybe we initially became a unicorn back in 2019, and for the few years after that, you know, it it felt like smooth sailing.

And then starting in about 2022, right, when it was or the entire media narrative, let’s say, was tearing down tech companies. Because, I mean, in some ways, it was very fair. Many, many tech companies received very high valuations. There was an incredible amount of excitement in tech, and then the markets all crashed. That, and starting in 2022, was when I noticed, for us specifically, the tone entirely shifted, where it was the media engine pointing itself towards, know, pointing out all the missteps from companies like us or a lot of our peers versus trying to point, you know, trying to take a balanced perspective.

And there were even, you know, another example of this is, so starting in about 2020, we began working with the US military and the US DOD. This is obviously long before the current defense tech hype wave and and long before all that, but it was it was driven by an intrinsic belief that we had as that I had and we had as a company that it’s important for the United States DOD to have access to incredible AI technology. That was like a fundamentally important thing for the future of the world.

And in the years after that, by and large, the traditional media engine actually tore us down for supporting the US government and supporting the military, versus taking a broader view that maybe this is a positive thing, actually, to to support, support the US military. This almost goes to what I was saying about the dichotomy in treatment and, you know, testifying in Congress versus with the media. You know, I testified before Congress about AI’s use in the military. I would say my The treatment I got there, I think, was a properly broad one.

Was, hey, obviously, is powerful technology that we need to be thoughtful of, but it is so important that America leads in this, and, you know, thank you for for everything that you’re doing. That that, I felt, was the response. Whereas in the media, it’s this incredibly scornful perspective around, is this a good thing? Do we trust this company? Like, what does this mean?

Yeah.

If

people who we say give a shit internally, but who really, really care, You know, they really, really care about the their work product. They really, really care about the quality of their work. They really, really care about the organization. They care about making sure that the company has an impact. They just really care. And what that means, how that manifests is, they’re willing to sweat every single detail, and if they get roadblocked or there’s like something in their way, they’ll spend the extra They’ll go the extra mile to make sure that they get through those things.

That’s how startups work. These small teams of people who each care 10 times more than the average employee, 10 or a 100 times more than the average employee inside a big company, you end up just solving so many more problems than than the big company. How many people do you have in scale

I think the answer is yes. You know, what we say a lot internally is how do we hire the Navy SEALs? Not not the Navy. Not that there’s anything wrong with the Navy, but how do you have, you know, a really small elite group where you’re really hiring the cream of the crop? And this goes down to process. You know, for us, at this point of the company, still, I approve every hire. I will either indirectly interview or look at the interview feedback and look at the understand every single person who we hire to ensure that we’re keeping an exceptionally high bar.

And that way What

Maybe on average, 25 to 30%. Like a lot. Wow. Like a lot. And I think usually it’s due to, you know, maybe there’s a new hiring manager who, you know, needs to get calibrated, or it’s like an edge case of various forms. But to me, the way I think about this is like, I, as the founder of the company, I have seen everybody who’s come in and I’ve seen who succeeds and who fails. I have, as an algorithm almost, like developed the most fine grained data set of understanding what it looks like for people to be successful at scale, and what it looks like to have the Navy SEALs versus the Navy.

And it’s my job as a founder to help ensure that we as an organization are actually utilizing all this knowledge, and utilizing all this learning that’s been happening over the past eight years in the organization, and carrying that forward.

The biggest one was actually, you know, in the same era of like 2020, 2021, was thinking that hypergrowth as a company meant that you had to hypergrow your team. In those few years, we did what a lot of tech companies did. We, like, doubled, tripled the team year on year. In 2020, we were about a 150 people. By the end of 2022, we were over 700. It was this insane amount of hiring and this incredible amount of of hyper growth as a team. What I found out is when you hire that quickly, it is impossible to to do what we’ve just been talking about, which is maintaining this high bar and maintaining this feeling of excellence within the team.

Did you see the reduction of that bar in real time? It was kinda subtle. It was something where you would hire all these people in, and then you notice it, like, the next year or the next six you know, six months later. You would notice it slowly, and that the organization, you know, there were challenges that the organization used to be able to to deal with and solve that slowly just calcified, and we weren’t able to we weren’t able to get around. And so you’ll notice, you know, from the end of twenty two, where I said we were 700 people, to now we’re 800 people, team has mostly kept the same size.

But the company the revenue of the company has grown dramatically.

This is a fascinating thing, actually. I actually asked Patrick Collison this question as well, and Stripe obviously is an incredible company that has, for a lot of their lifetime, I think, been one of the iconic Silicon Valley companies. And I asked him whether or not he thought that the fact that they were such an iconic company was beneficial in all the hiring they did. And he made an interesting point, which was that the best people they hired, he thinks would have been people who would have joined whether or not they were the hottest company in Silicon Valley.

It was these sort of like off the beaten path people who were actually the the best hires they could have gotten. And a lot of the people who joined because they were the hottest company in Silicon Valley, for one reason or another, weren’t necessarily the most valuable employees. And so there’s this there’s this element where I think the common belief and the common narrative is like, you wanna be the hottest company so you can track the best talent, so you can hyper grow, so you can then go keep growing.

And I think that’s often so so difficult, and it’s much more about like, how do you develop an ecosystem of talent that is like very self preserving, keeps a very high bar, and always seeks out and searches for the best people, and then independent of whether the company’s hot or not hot? Because you will have, to your point, you’ll have moments where you’re hot, moments where you’re not hot, moments where you’re hot, not hot. And you need that talent ecosystem to be self preserving independent of that to drive the best

Yeah. I think that’s right. Yeah. Yeah. But then if you look at core technical development at OpenAI, a lot of that is still driven by people who have been at OpenAI since before they became the hottest company ever. You know, another company that I think experiences or went through this is Airbnb, Brian Chesky. Right? And I think that, you know, he’s talked about this publicly after the pandemic, he all of a sudden realized like, hey, I have to He had to kinda like rebuild the entire company.

And he massively shrunk the team. He invested a lot more into talent density, and then he built the team to to remain small. And I think that they’re even now the most or one of the most profitable companies per head in all of tech, and that’s because of this sort of this realization that he had that he didn’t need to keep growing that team to see the financial gains or the financial output.

I think it’s actually everything about this hypergrowth stuff that we’ve been talking And it’s really around divorcing team hypergrowth from company hypergrowth and extra investing into quality and excellence. What’s the biggest misconception you hear most often about AI? I think the biggest one today is all that’s between us and AGI is compute, and I think we need data to get there too.

is a great question. You know, I think obviously, I don’t think this is practical, but I do think Satya Nadella has has been one of the most brilliant business strategists of the modern era. What he has accomplished at at Microsoft is just staggering, and I think any board would be very lucky to have him.

That’s an interesting one. The interesting one is, like, how my perspective on AI has changed in the successive eras. And I mention this because I started the company in 2016. The first three years of the company were just full focus on autonomous driving and autonomous vehicles. And then in 2019, we actually started working on generative AI. We started working with OpenAI on GPT-two. And so we are one of the few AI companies that I think has seen multiple eras of the technology and has seen the sort of the first boom and bust cycle with with autonomous vehicles.

I think it’s a it’s an interesting one, which is like, what’s the same in these successive areas, what’s different? That’s an interesting question. How has your view changed? Are you most excited now? I’m quite excited, but I think there’s also reasons to be cautious. In autonomous vehicles, one of the things that happened in the autonomous vehicle craze, there were a lot of promises that were being made that were divorced divorced from the technical reality. And so a lot of the prominent autonomous vehicle companies, a lot of the prominent organizations were making bolder and bolder promises to be able to raise money.

And those were at first, they weren’t super divorced, but over time, they became more and more divorced from the technical realities. And that resulted in this very painful trough, where it’s sort of the promises weren’t met, and so it felt like the entire industry was falling apart. And actually, at the end of the day, you know, now we have Waymo’s driving around San Francisco, perfectly proper l four autonomous vehicles driving around. Tesla autopilot’s gone really good. If we had more measured promises along the way, I think now we would feel amazing about autonomous vehicles, whereas instead we went through this huge up, this big down, and maybe it’s sort of like on the upswing again.

I think this is one of the big concerns I have about generative AI, which is, I hope not, but the same thing might happen again, which is that we have these really big promises that are starting to get made about the technology that get divorced from technical reality. And then that creates this sort of gap that is bound to cause a hangover. Will Trump win penultimate one? I still think it’s a toss-up, actually. US elections are so strange to think about because it always gets decided by the swing states.

Frankly, I don’t trust anyone on the coasts to have any fine grained understanding of how the swing states will play out. I have no idea. I don’t think anybody should listen to anybody who lives on the coast to understand what’s gonna happen. It always boils down to the swing states. Final one for you, my friend. What does Scale in ten years time? You know, hopefully doing something very similar to what we’re doing now, which is continuing to be the data foundry for AI and serve the data pillar for AI progress.

Would you like to go public? For sure. Yeah. Well, one thing I think a lot about is how do you solve problems that will never go out of style? Like, would

There’s clear benefits to being a public company, for sure. But I think Stripe is an incredible company in that they they can be incredibly profitable, and they can accomplish all their core financial