Scaling to $1BN+ in Revenue with No Funding: Surge AI

Thanks for having me. It’s wonderful being here today.

Yeah. So I think the biggest lesson for me was that you can build a completely different kind of company with 10% of the resources and 10% of the people, but you’re still moving 10 times faster and building a 10 times better product. Like, imagine you could just magically remove the 90% of people who aren’t working on on interesting problems. What what would happen then? Well, if you have a company that’s one the size, you don’t need to hire as many people, so you spend less time interviewing.

You spend less time in meetings. You spend less time giving people updates for the sake of updates. And if it’s one tenth the size, that means everybody has a better view of what’s going on around the company because there isn’t all this clutter masking important stuff. And because the talent density is higher and the teams are smaller, that means the communication is a lot higher and the iteration speed is a lot higher and better ideas just percolate around more quickly.

Yeah. I mean, I think a big thing about being small is that when you’re smaller, that means that I, other people around the company, would just have a much better view into the customer problems themselves and what everybody’s working on. And so it’s kind of like at these bigger companies, a lot of your priorities, a lot of the things that you’re building, they’re simply you’re simply building them to impress someone. Like, hey. I need to impress my VP. I need to impress my manager. I need to impress my director so that I can get promoted.

And you’re not really building things or prioritizing things because they’re good for the end customer, and they’re good for the end product. It’s more like, okay. I have this priority to like, let me think about it. It’s like, have this priority to improve an internal tool. Okay. Why are you improving the internal tool? Well, it will make people 5% more productive. Why do I want them to be 5% more productive? Because they’re spending 20% of their time interviewing. Why are they interviewing?

Because they like, they’re growing for the sake of growing, and it just leads to this perpetual cycle where a lot of your priorities are just divorced from, like, the end customer to end product, and they’re almost, like, priorities just for the sake of internal company machinery.

people don’t realize, again, from the outside, it’s that how much of what you’re building, again, is for this internal company machinery, and how much of the internal company machinery is simply because a lot of people within these organizations, their goal, again, their goal isn’t to build a product. Their goal is to tell their friends they’re a VP of a thousand person org, and that sounds impressive. And so their goal is to think about, okay. So how do I how do I grow my org even faster?

How do I find more teams that I can hire? How do I have these monthly performance reviews where, again, now that I’ve built this thousand person org, I need to prove to my VP, my CEO, that the thousand person org I’m building is efficient and useful. And so, like, basically, a lot of the work that goes on in these large companies, it is simply to kind of perpetuate and grow even even further a lot of this, like, very, very big company machinery that exists purely purely for, like, internal horizons.

think a big part of it actually just boils down to the kinds of questions they ask me. Like, some people, when I bring them, they will ask really interesting questions about our product. They will brainstorm about ideas to make our product even better. They’ll be like, okay. Yeah. Like, I went to your web page. Why don’t you improve these things? I tried signing up as a worker. Why did these things happen in a flow? I tried working on this project. Like, what if you guys did this instead?

And other people are like, if I join in a year, will I be able to be a manager of a company? If I join, will you be able to hire like, will will I be able to hire 20 more people to support me? And so it kinda just boils down, I think, a lot a lot of times to the kinds of questions that people even have at the forefront of their minds.

Yeah. So I I’m a big fan of that. So, like, I for example, personally, I actually have no one on one meetings. And it’s kind of funny because oftentimes people will ask me, well, how often do you meet with your reports? How often do you set aside for, like, for for these meetings? And I I just don’t have them at all. Like, oftentimes, like, I will just give people my calendar, my Calendly, and they’re just surprised at how blank it is because I try to avoid filling my meetings all day.

And so I will actually go out and I like, sometimes when people join, they’re like, okay. I need to go and have one on one meetings with these 10 other people that I’m collaborating with on a weekly basis. That’s just because that’s so used to when you come from Google or Facebook. And I tell them, like, why are you having these standing one on one weekly meetings? Like, did you not talk to them every day during Slack? Are you just, like, unaware of what they’re doing?

Like, it’s almost like a negative sign if you’re having a one on weekly meeting because means that you just don’t know what’s going on with these people. You’re not you’re, like, almost waiting for your weekly meeting to raise raise interesting questions or raise interesting problems. And so I think we’re pretty ruthless internally about killing meetings when you’re unnecessary.

Yeah. I mean, I absolutely believe that that company was this one day. You think about it, like, I’ve always believed in 10 x engineers, even a 100 x engineers, and already you have a lot of these single person startups that are already doing 10,000,000 in revenue. And so if AI is adding all this efficiency, then then, yeah, I can definitely see this multiplying a 100 x to to get to this $1,000,000,000 single person company.

I mean, like, even today, you see how we are, like, honestly so much more efficient than some of our peer companies. Right? And so even for that reason alone, you can you can already see the fact that a 10 x engineer or a 100 x engineers exist. Like, let me even just break it down. Some people are simply two to three times better, two to three times faster than anybody else. Right? They just code faster. There are some people who simply have two to three times more better ideas.

There are people who simply work two to three times as hard. There are people who have two to three times fewer meetings. There are people who simply have ideas that something other people can’t think of. And so if you just multiply all these things together, right, and, like, two to three x is often actually an underestimate. Like, I I know people who, yeah, literally are five times more productive coders than anybody else. And now add in, you know, all the AI efficiencies that you get. Like, you can you can just, like, multiply all those things out, and, yeah, you you get to a 100.

I would say that or maybe both today, but definitely even more so in the future. It’s like good people have so many ideas that they just don’t have time to implement. And if you think of AI today as something that isn’t necessarily coming up with the greatest ideas, although it can, but it often just removes a lot of the drudgery of, like, your day to day work, a lot of your day to day coding.

And so if you don’t have to spend that time on the drudgery, but you just have, like, these endless ideas that are just bouncing around your head and AI just helps you put them to paper, then I I do think it kind of disproportionately favors people who are already, the the the 10x engineers.

I mean, I think it’s both. I mean, I think a lot of the other companies in our space, they’re just not technology companies at the end of day. They are either body shops or they are body shops masquerading as technology companies.

I think the way to think about it is a lot of comedies in this space, so they don’t have any technology. And when I think about technology, it’s like they don’t have any way of measuring quality of the data that they’re producing, and they don’t have any way of improving the quality of data that they’re producing. They are literally just body shops in a sense that they sometimes literally have no technology at all. They don’t have a platform where workers are doing work, and so what they’re doing is they’re simply finding people.

Like, they’re recruiting warm bodies. They’re looking at resumes. Like, anybody with a PhD, they’ll just, like, instantly hire them, and then they’re just passing them along to the AI companies, to the frontier labs. And so, again, they have no technology. They have no way of measuring what any of these workers are doing. They have no way of knowing if they’re doing a good job or not, so they have no way of doing things like, hey. What if I AB tested this algorithm for improving quality? What if I changed this method of allowing workers through?

What if I tweaked our tools in order to change these questions around? Would it make their workers more efficient? Would it improve their quality? Or would it actually make it worse? They just had no way of doing these things because, again, at the end of the day, what you’re passing to, like, their customers is just the body itself, the person, as opposed to the data. And so what that means is they just, like like, again, they they just have no technology to measure or improve anything.

Yeah. Again, like, if I think about the way we think about it, it’s maybe the following. So we have always started out with quality of the data as our number one principle, and as a result, we need to build a lot of technology in order to to measure that and improve that. If I think about, like, what what goes wrong, it’s that people often just don’t realize how difficult quality control is. And People often think that humans are smart, and so if you just throw a bunch of humans at the problem, you’ll get good data.

And what we found is that’s that is completely untrue. For example, I I went to MIT, but, yeah, I think half of the people who graduate with a CS degree, they they can’t even code. So it’s a really challenging problem to detect high quality. And second, if you actually take the folks from MIT who can code, they’re actually just going to try to cheat you. They’re going to sell their accounts to somebody in a third world country. They’re going to try to use LMs to generate the data for you.

They’re going to come up with all these crazy methods to cheat a system. So it’s also this really, really challenging problem to detect low quality. It’s it’s actually really adversarial. And so what we found is that when you want to get the highest quality data to train LMs that are already, you know, super intelligent, you actually need to build a ton of really sophisticated algorithms. You can’t just take warm bodies or, you know, try to improve your methods for resume filtering and then throw people at the problem and get good data and results out of it.

Like, the teams I know who try this, they actually end up moving 10 times slower than anybody else without realizing it.

Yeah. So I I used to work as an ML engineer at a bunch of the big companies, and the problem I just kept running into was that it it just kept on being impossible to get the data that we needed to train our models. For example, I used to work on our ad search and ad systems at Twitter, and one of the thing first things I wanted to do was build a sentiment classifier. Yeah. It’s a super simple problem. All all you need is 10,000 tweets labeled as positive or negative to train your models, but our human data system at the time was literally just two people we’d hired off of Craigslist working nine to five.

Even just in order to get started, we had to wait a month. Then we had to wait another month for them to label the tweets inside the spreadsheet because the tools of AI were just terrible. And when when we finally got the data back, it was actually just completely junk. They didn’t understand slang like, she’s such a bad bitch. Like, they were they were actually labeling this negative when, you know, it’s actually really positive, and they didn’t understand hashtags and all these other aspects of the tweets.

And so I actually ended up just spending a week labeling tweets myself because that was so much faster and better. And at the same time, this was actually really simple stuff, but the bigger problem we wanted to solve was how do we optimize our ML systems for the right objectives, and how do we build feeds that are engaging in a positive way for users? Think about, again, about Twitter. This was the old days when it was a purely chronological timeline, and so one of the things we wanted to do was just make it easier for for our users to discover the tweets that they really cared about.

So the question was, how do we train our recommendation algorithms? And the obvious choice was clicks and retweets. Like, you just train your algorithms to produce as many clicks and retweets as possible. But the problem is we try doing these things, and it turns out to be this incredibly negative feedback loop. Like, once you optimize for clicks, the most clickbaity content starts rising up to the top. You get lots of racy content, lots of lots of girls in bikinis, lots of listicles about 10 horrifying skin diseases, and and so on.

And so we wanna train all of our models on all these deeper principles instead, where we’d ask our human raters to label tweets and recommendations with product principles, like whether this is a top voice connecting somebody with their interests or somebody just had this really interesting insight of a particular topic. If we couldn’t even get simple sentiment analysis right, like, again, like labeling whether a tweet was positive or negative, we definitely couldn’t get this more complex data, the quality of SCAW that we needed. Like, we basically started Surge in 2020 right after the launch of GPT-three, and I think it really is because there was just so much more that you could see the industry moving moving towards.

And if we really wanted to progress it in in obviously really, really big ways, we just need a different kind of data data solution to to help the industry.

Yes. So the way it worked was so I’ve always been a really big fan of MVPs, and so I’ve already just built myself or v one in a couple weeks. But, like, I think the really nice thing was so, again, I had worked in this space for a really long time, so I already had a very clear vision of what I wanted to build. So as opposed to feeling like I needed to go out and hire 10 engineers in order to build a product, instead of feeling like I needed to go out and fundraise, you know, $10.20, $30,000,000 in order to hire, you know, more people.

I just wanted to build it myself, and I wanted to talk to customers myself. And so that’s what I did. I I’ve already built the v one in a couple weeks. I posted about it on my blog. I told people about it that I met. And, yeah, there actually was this giant demand for the data already, so I think we were we were very lucky early on.

So I think one of the things that’s always driven me crazy about Silicon Valley is that it really is just a status game for most people. People are just raising for the sake of raising. Their goal isn’t to build some great product that solves an idea that they fundamentally believe in. The goal really is to tell all their friends that they raised $10,000,000 and they get a headline on TechCrunch. I have a lot of friends who’ve worked at Google for ten years. When you think about starting a company, they actually often tell me they don’t even have a problem that they wanna solve.

They’re kinda just bored, they wanna try something new. And at the same time, they can actually definitely pay their own salaries for a couple months, but the first thing they tell me is, yeah, they’re they’re going to go out and raise some money. And so they might try talking to some users, and they might try building an MVP. But the only reason they do that is just to check off some checkbox on a YC application. And then what happens is they will just constantly pivot around random ideas until they get something that happens to get a little bit of traction, and that sounds impressive to VCs.

And so they spend all their time tweeting and tweeting hot takes and networking and going to all these VC dinners, and it’s all just so they can get this highline about raising $10,000,000. I I really think that people’s first instinct should instead be to find some big idea that they fundamentally believe in that could change the world. And I don’t really care why they believe in it. It could be because they have a lot of experience in this space. It could be because they talk to a bunch of users, but it really has to be something that they believe in that they’d double down on for the next few years.

Like, if think about startups, startups are all about big risks. Right? You have to believe in something enough that you’re gonna take a risk loading it. If all you’re doing is jumping around idea to idea every week until you land on something that gets you a thousand retweets, you’re you’re not taking any risks. You’re just somebody looking

Yeah. For 90% of companies, no. Like, there are sure. There are some companies where you actually do need a lot of capital in order to build hardware or, like, whatever it is for for a couple of years. Like, you really need a lot of investment before before you, before you can get to your your, like, actual MVP. Probably 95% of products that are out there, for 90 to 95% of startups that people are building, no. Just go out and build your MVP and see, yeah, see if it gets any traction.

I actually do believe in it. Again, if you think about the idea of a startup as something that a place where you can take big risks, where you can build something that nobody else can, and you’re willing to just go all out to, again, create something that literally nobody else could. It does have to be something unique to you because, again, like, otherwise, you’re like, sure. You can get to, like, a decent and medium sized company with a commodity idea. But if you really wanna go big, if you really want to build a generational foundational company, I think it really should be about an idea that is

I think it’s kinda funny. So if I think about the things that have made me happiest in the past few years, I can think of, like, two things off off the top of my head. So one is sometimes our customers, whenever they launch their next big model, one of their first things that they’ll do is they’ll reach out to me, and they’ll be like, hey. I just wanna send you a note that we couldn’t have done this without you. And I think that’s just so amazing to hear.

Like, again, if you think about, like, how often do you get to play a role in building some of the most important technology over time, and then right after their launch, like, these very, very top people who are very, very busy, one of their first thoughts is to thank you because of how critical you were to the operation. I I just think that’s so cool. So, like, that that is one of the things I often think about. And then I think the other thing that I often think about is, again, in many ways, Surge is embodiment of me and my interests.

And what I’ve always loved doing is analyzing data and figuring out how to use that data to make models better or to make products better. And so every now and then when I just get the chance to write an analysis myself of the latest frontier model, or I get to read some of the analyses our internal employees are creating based off of the data that we’re providing, I just think it’s so cool that a lot of the data we’re providing, it’s just so insightful, and it, like, helps people build models in ways that they just wouldn’t know how to otherwise.

Sorry. I I think I say it nonchalantly because it felt very nonchalant. I think what would end up happening is so I would find all these people who really were desperate for a lot of really high quality data. I mean, way it work is they would just email me with their request, or we would just jump on a live meeting, and we would just get started. And it might take a week or a couple weeks to negotiate some sort of, SOW or contract just because, you know, a lot of this does have to live within the confines of their company.

But, yeah, I mean, I think we’re really lucky in that, again, I had a lot of experience in this space, and so I had a lot of experience working with ML engineers and research scientists in the ways that they wanted to get data and the way they wanted to look at it. And so I think things just moved very, very quickly.

It was both because I mean, obviously, we need a talent supply in order to make our product work. So there are some companies in this space who will simply think of it as a pure supply problem, and they don’t give any consideration to the technology. Like, what’s the technology? The underlying technology? Like, how do you identify these people? How do you make sure that they’re doing good work? How do you remove the bad quality work? Like, they’re just literally not thinking about any of the technology aspects at all, and you’re also not thinking about the product at all.

Like, how do you present the data to the customers? Like, one of our principles like, one of the principles that I’ve always had, even prior to Surge when I was just an ML engineer or data scientist, one of the things that I’ve always tried to encourage is what we call this visceral understanding of the data. Like, I really just want you to go in and get your hands dirty and look at the data. Like, historically, a lot of ML engineers, they kind of just don’t take the time to look at the data, and maybe that’s because the data just isn’t all that interesting.

Like, when all you’re doing is drawing bounding boxes on cars, sure. I don’t need to look at a thousand bounding boxes. But what you’re doing is, yeah, creating poetry, creating mathematical equations, creating new research, like, want to get your hands dirty with the data to see what it is that you’re producing, what you’re teaching your models. And so I I think it actually really is important to this aspect of viscerally understanding the data that you’re getting.

I mean, I I think it was very, very obvious that there was just huge demand for this product, and there were just so much more that we that we could be doing.

I mean, I would say there was nothing that raising would help us with. We were very lucky to be profitable from month one, and so we didn’t need the money. We didn’t need a sales team. Like, I didn’t actually didn’t want a sales team going out and selling our product. Like, I wanted people to buy us precisely because they understood the value of high quality data. They saw all the gains that our data was producing. I didn’t want them to buy us simply because they heard about us in some tech crash article because that would almost put them at odds with the kind of product that we were building.

Like, one of the things that I think is actually really important, especially early on, you want customers who believe in your product and not people who are simply giving you a little bit of money. Because your early customers will shape the kind of product that you’re building, because, yeah, you’re building for them. You’re you’re building for their needs. Like, they’re giving a lot of really, really great feedback. And so you almost want customers who share the same overall vision, and so that that was actually very important for us.

Like, I I didn’t want sales teams who would email 10,000 people and be like, hey. Any any thoughts on on getting good data? It was just very, counter to the kind of kind of product that we wanted to build.

So I think this is where we actually have a really, really strong vision of where our product should be. So, again, like, going back to what I said earlier about how most companies within our space, but maybe also at large, they don’t have product principles that they try to adhere to. Again, like, we had very strong product principles from the start. We wanted to focus on quality above all else. Like, if whoever thought that we couldn’t give the quality that we wanted, we would just say no, as opposed to these other companies where they’re almost, like, desperate and racing around just trying to get any traction that they can.

They’re trying to prove to the VCs that their numbers are always going up. They’re almost, like, focused on getting $10, a $100, a thousand dollars, wherever they can. And so as soon as some customer comes to them, even if that customer is counter to the kind of product that they’re building, if they’re offering money, they’ll just say, sure. I’ll do it just because they’ll give me another logo from a website. They’ll give me another case study to show another customer. They’ll give me another talking point with my VCs.

I think we’re very lucky to not have to worry about that because we could build for the long term vision we had as opposed to, again, like, opposed to pivoting every few months. We just wanted to double down double down on the idea that we actually believed in.

No. I think we’ve we’ve never let quality slip. It’s such a principle ingrained into everybody at the company. Like, one of the things that we simply tell everybody when we first join, quality is the most important thing. It’s more important than anything else. If you have to make a deadline slip because for whatever reason you don’t think the quality is there, if we have to say no to a project because we just can’t handle it right now we can generally handle a lot of things, but we just want to ingrain this principle that it’s okay to say no.

It is okay to kinda, like, let other things maybe slip just because we we care about quality at the end of the

Yeah. I think the funny thing is, like, again, I’ve been at all of these other companies. Oftentimes, when people are saying, like, yeah, I my hair is on fire, and I really need this engineer, so I know they don’t meet the bar. I’m gonna lower the bar to hire them. Like, actually, the the engineer didn’t like, what are they doing? They’re building probably a feature that nobody cares about. They’re building an internal tool to improve the productivity of everybody around a company by 2%, while at the same time having so many meetings with them that they, like, take up 5% or 10% of their time just talking about the feature.

Like, a lot of the things that people hire for just actually aren’t all that important. And so, again, like, when you don’t feel like you have to hire for the sake of hiring, like, when you have the mentality that, okay, if your company only grows by 10% or even 0%, that’s actually positive. Like, I think people right now, they have this view that if someone were to tell you, oh, yeah. My my engineering org only grew by 2% this year, your initial reaction is going to be, okay.

You you guys must not be doing well. And so there’s, like, this negative incentive where people feel like they need a hire just in order to prove to other people that

I honestly don’t pay enough attention to to, like, these kinds of Silicon Valley Twitter discussions for me to have a sense of whether this mentality is is becoming more pervasive. I can believe in it. I I can hope for it. I I don’t know if it’s true right now.

It’s kinda funny because, again, I used to work at Twitter, and I I love Twitter back in the heyday, but I actually really am glad that I’m not surrounded by the default ways of Silicon Valley thinking. So every now and then, if something is important enough, like, maybe there’s some big new product that is actually really, really cool or there’s some really, really interesting new research paper. It’ll be, like, big enough that even though I’m not monitoring Twitter every day, it will just reach me in some other way.

Like, yeah, one or more employees will tweet will, like, post it in our Slack channel, or somebody will email it to me. So, like, the really important stuff will manage to percolate itself to me in other ways, but I actually am really glad that I’m not worrying about what people are saying

Yeah. Let’s just say we’ve been we’ve been doing really, really well from the start.

Yeah. So we’ve always been very, very successful from from literally month one. Things definitely hit an inflection point with ChatGPT because I think people just saw how incredibly valuable human data and RHF was. Definitely ChatGPT was an inflection point for us, but even before that, we have a very strong growth.

So it’s interesting because I think it was an open secret where a lot of top researchers already knew who we were. Like, they already knew that we were the biggest and the best in the space even though we’ve been pretty under the radar, and so most people were already working with us. There are a lot teams who are using Scale for legacy reasons or they just didn’t happen to know about us, so we’ve been getting a lot of new interest for them too. I think the more interesting thing has been it’s kind of been really fun seeing how we’ve opened their eyes to what really amazing, really high quality data can actually look like.

Like, a lot of them have tried getting human data from other teams, and they tell us it’s been this slog. They’ll spend months trying to improve the data quality for really basic stuff. And it will look like it’s better for a month, but then it will just quickly regress. We have this concept where we just wanna get started immediately. We wanna show them really, really high quality data immediately. Like, one one of the big concepts for us as a company is we always want to be producing data that you simply couldn’t get anywhere else.

Like, there’s so much richness and complexity in types of things that we do that we just want to open up new avenues of research and open up new avenues of new types of products. I think a lot of these new companies or these new teams who who’ve been coming to us, it did help. It’s just been a breath breath of fresh air for now.

Yep. I mean, so I would say I’m pretty sure that a lot of these other companies like, at the end of the day, people want high quality data, and they don’t want to be working with body shops. And so I think we’ve seen, like, a massive wave of interest because, like, yeah, like, the space is really large, and there are lot of teams who are still using Scale for legacy reasons. But it’s like, at the end of the day, we were already the biggest to invest in this space.

And so even when there were teams at some of these larger companies who weren’t working with us already, they kinda, like, knew who to turn to.

I mean, I think for some people, like, they have a price, but I I think we don’t.

No. I I mean, I definitely wouldn’t sell for 30,000,000,000 or even a 100,000,000,000. If you think about us as a company, I I already have everything I want. We are profitable. I have complete control of our destiny. And so I’m really lucky to have all the resources I want to already do anything that I want. And there aren’t many companies who can say that.

I mean, I I think it really is to help achieve AGI. Like, if you think about every every well, like, what do kids dream of? Like, yeah, when you’re a kid, you literally dream of building AI that can do all these amazing things. And now we have the chance to do it. Like, I I really do think we are such a critical aspect of what all these companies are building. Like, again, a lot of our customers at these Frontier Labs, they would just often tell me they wouldn’t be able to build what they’re building without us, and they’re just amazed at what we do.

And so being able to be this critical part of what is literally the greatest technology of both our time now, but also maybe one of the most important things we can ever build, that’s amazing. And so why would you get acquired and stop doing that? Because, yeah, getting acquired would be really limiting. It would be this admission of failure and jumping ship because you can’t make it on your own anymore. When we’re the opposite, we’re incredibly successful, and there’s literally nothing else that I’d wanna do instead.

So I think there are two reasons. One is that there will always need to be more breakthroughs, whether it’s breakthroughs in, yeah, how you leverage all this data or breakthroughs in different types of algorithms that you’re building. And then another one is just how you gather that data. Like, order to cure cancer, how will you gather the data that’s needed to make those breakthroughs? Maybe you’re gonna have to run real world experiments, real world studies, and those studies will simply take time. Will there be a way to speed up those experiments through various kinds of simulations or just other forms of gathering data?

I I don’t know. But there’s some of the question, how how do you get the data even faster,

So a lot of people talk about the shift to PhD level data. And, yeah, I think it’s important. Like, we basically have the biggest group of the smartest people in the world working on a platform. We actually have Harvard professors and Stanford PhD students and prints and computer science theorists working on all these really interesting problems with us. Like, it’s kinda crazy. Like, if you think of all the PhDs even at Google or Meta or Microsoft, we have way more than all of them combined doing work for us in a single day.

And it’s also true that they’re not just writing random JavaScript code to improve ads. They’re actually pushing the frontiers of science when they’re collaborating with these models. But I think what people underestimate is that having a PhD isn’t enough. Like, lot of PhDs, they just aren’t good at this type of work. There are a lot of body shops and recruiting shops in our space that basically just look whether you wrote down that you have a PhD on your resume, and they’ll just instantly give you work if so.

But a lot of PhDs just aren’t very good. Like, I I think 80% of the computer science PhDs I know, they write shitty code because they’re only good at math and algorithms. And you think about people like Ernest Hemingway. He didn’t have a PhD. I don’t I don’t think he even went to college. And so I think there are two things that are important. There is this underestimated aspect of our space where you actually need a lot of technology in order to make sure that you’re delivering fully high quality data.

Like, I think it’s like a lot how, like, Vimeo has a lot of so called high quality videos. But, yeah, they don’t have any algorithms, and so YouTube’s videos are way higher quality and more engaging in the end. And then the second is it’s just that a PhD isn’t enough. Just because you have a PhD doesn’t mean that you can make some breakthrough in physics. What you also need is street smarts. Like, you need the creativity and the mental 42 to think of really interesting problems and find these problems and probe LMs and see whether they can solve them today and then teach them in really interesting ways.

Because otherwise, if all you’re kind of doing is throwing PhD cells a problem, all you’re doing is teaching models how to hack SIDI benchmarks and get good at basically the equivalent of SAT problems.

Again, I think this boils down to technology that we build. Over time, it’s simply true that people are going to be trying to solve more and more problems. Like, we have hundreds of thousands, millions of people working on our platform. And when you do that and you have a thousand projects, like 10,000 projects that are literally running in any given week, How do you make sure that you are building technology to identify who are the the top 1%, top 2% of people who can really push the boundaries of physical problems with these models?

Or how do you identify the top two or 3% of people who are writing the most amazing poetry? How do you find those people? And then also, how do you remove the the worst of the worst? The people who will inevitably try to cheat you and spam you, and they will basically regress the models if you allow to tear data through. Like, it actually is a really, really prevalent problem, and you just need a lot of technology to build this. And then at the same time, like, these are researchers who wanna move really fast, like, to all these frontier labs.

Again, like, all the algorithms are changing every day, and so they want to try out new projects every single week. And so if you’re not moving fast enough, like, if you’re unable to create a new template or you’re unable to find the expertise that you needed, like, literally within the next day or the next week, it’s just gonna be too slow for these researchers. And, again, if you don’t have the technology to manage these 10,000 projects and automatically create them and automatically identify the really high quality data, it’s just gonna be too slow for them.

I would definitely rank data quality first, followed by compute, followed by the RLMs.

I mean, I actually just fundamentally don’t believe that you can throw more compute at the problem. Because if you’re not getting the data that the computer is essentially trained on or if you don’t have the right objectives and evaluation metrics that, again, your your computer is optimizing towards, you’re just going to fall into this trap of seeing progress that actually isn’t there. I can give you some examples. So let me talk about why I think data quality is such a problem. So I think data quality issues have already been a huge setback for a lot of frontier labs.

Like, one of the things that we often hear from teams over and over is that before they used us, they tried getting data in other ways, and so they’d train their models. They’d evaluate their models, and their metrics kept going up. But after six months or even a year, they realized that their training data was shit. Their evaluation data was shit. And so all the progress that they thought they were seeing was actually completely misleading, and they either made no progress or the models after six months were even worse than when they started.

Like, for example, we see this a lot with LLM Arena. So LLM Arena is this popular leaderboard of LL models, and it’s basically the equivalent of clickbait. What happens is that you have people going on to what’s called a chatbot arena. They’ll enter a prompt. They’ll see two model responses, and then they’ll vote on which one’s better. They’re not taking the time to really read or evaluate the model responses at all. Like, one of the models could have made completely made everything up, and these participants, they’ll vote on it because it has emojis and nice formatting.

We’ve literally seen this in the data ourselves. Like, one response will just be a complete hallucination. But because it has an emoji and because it has a couple words bolded, people will just like, okay. Yeah. Like, that looks good. That looks much better than this other thing than that I didn’t take the time to fact check at all. And so one of the things that we’ve learned is that the easiest way to improve in this arena is simply to make your model responses a lot longer.

Like, one of the funny things is that if you actually take the top model on its leaderboard, the number one model, and you ask it, when did a pope die? It’ll give you a really long response that seems impressive, but it gets the answer completely wrong. It tells you that pope Francis is still alive. It’ll even tell you that there are search results that indicate that Pope Francis died in April, but, actually, these were just rumors and misinformation, he’s still alive. It’s wild that this model will say this.

So, again, like, what happens is that there are lot of companies who are trying to improve their leader board rank. And so they’ll see progress for six months because all they’re doing is unwittingly making their model responses longer. They’re adding more and more emojis. They’re adding more and more forerunning. And so they see their models cladding on this leaderboard, and so they think they’re making progress when all they’re doing is training their models to produce better clickbait. And they may finally realize six months or a year later, but it means they basically spent the past six months making zero progress.

This is what happens when you kinda, like, throw compute at the problem without understanding the underlying training data that you’re, like, again, like, throwing the compute towards.

If you watch the Grok four launch, the Grok four livestream, I think you would even have heard Elon himself saying, like, yeah, these models are really good at I forget the word he used, but, like, they’re really good at homework problems. They’re really good at these academic, very narrowly scoped problems. It’s basically the equivalent of making them really good on SAT problems, but I’m making them good at problems that people are actually facing.

I am not. It’s kind of funny. So before we worked with the team, I didn’t really have a conception of what an Elon company was like. But, yeah, I mean, we we work really closely with the the xAI team, and it’s actually just incredibly refreshing to see how they operate. They are all very, very mission oriented. They’re all incredibly smart, and they work incredibly hard. Like, it will be 11PM at night, and I’ll DM them, and someone will want to jump on a meeting. And, yeah, I I jump on a meeting with them, and I see them.

They’re in the office, and there’s a ton of people behind them. So, like, they’re all they’re just, like, crazy hacking together on on all these problems. And so I I actually think it’s incredible, and it’s this kind of embodiment of what a startup can do when you really believe in something and are kinda, like, willing to do whatever it takes to achieve it as opposed to living within the confines of this giant bureaucracy. So I I think

that you know what you’re getting into when you work at Grok or when you work at xAI or any of these other companies. You know when you interview that these people are incredibly mission oriented. You know when you interview that everybody works super hard. You know that if you want to work there, you’re gonna have to be the kind of person who has the same values. Otherwise, you just shouldn’t you just shouldn’t join because you’ll be miserable. It’s this fact that it has such a strong culture and such a strong belief in what they’re doing.

It just attracts

So I think synthetic data is actually really useful in some places, but I think people overestimate what it can do. I’ll give a couple of examples. So right now, there are a bunch of models that have been trained really heavily on synthetic data. But like I mentioned earlier, it means that they’re only good at very academic, homework style, benchmark style problems. They’re actually terrible at real world use cases. So, yeah, synthetic data, it’s made models good at synthetic problems, not not real ones. And we actually hear from a lot of companies who tell us they spent the past year training their models on synthetic data, but they’ve only now just realized all the problems that’s caused.

And so they’ve spent actually months throwing a lot of it out. Like, a lot of them tell us that even a thousand or a couple thousand pieces of really high quality human data that we generated for them, it’s actually been worth more than 10,000,000 pieces of synthetic data. And so a lot of the work that we do is simply cleaning up all this synthetic data. If you think about, like, why this happens, it essentially is because the models collapse on this very, very narrow scope of, like, similarity that the synthetic data creates, and so it just doesn’t give the models the kind of diversity and generalizability that they need.

And then then one other point is that there’s also this interesting phenomena where models simply make a lot of mistakes and have certain misunderstandings that humans never will. Like, I was actually just playing with one of the frontier models recently, and it kept on just randomly outputting Russian characters and Hindi characters in the middle of its responses. This is a mistake that would be obvious to any human, to any second grader, but a model just didn’t know. And it’s, like, shocking that a model in 2025, a French year model in 2025 would do this.

And so it’s almost like you always need this external value system as a kind of safeguard to make sure that the models are working properly just because the models themselves have such a different set of way of thinking.

I think there’s an opportunity for both. And the reason I think that is it’s because on the one hand, you have these giant, all powerful models. And, sure, they can be really, really good and really, really powerful. And in, like, a raw capability sense, I think they’ll be able to encompass all of these different use cases. But in the same way that a company so take a company like Google or Facebook. There are simply some products that they can’t build because building those products would be counter to, like, culture or the business goals of, like, the overall parent company.

And so in the same way, sometimes you need to be able to move faster and to take big bets on certain kinds of products, and all powerful model just can’t kind of let that happen. Because if you let it happen within just, like, one small domain, it’ll kind of almost, like, pervade the entire model. So sometimes you do need, like, the smaller models to break through if they have, like, a really unique view on how they’re operating.

area where I’m not great, which is kind of funny, but, one area where I’m not great is I’m really bad at understanding financials. So sometimes people around a company, they’ll try to tell me, like, hey. Have you been paying attention to our revenue numbers? Have you been paying attention to our costs? Have you been paying attention to our margins? Like, do do you even know what they are? And I don’t. Like, they’re just, these financial metrics that, like, I could not tell you what EBITDA is.

I mean, I know what the acronym stands for, but the difference between that and revenue and profit and net margin and operate like, I actually just don’t know any of these terms. And it’s just like this blind spot. Like, no matter how much I try to try to understand these things, I just can never remember.

Like, if I could paint my perfect North Star, and this is something that I think we we wanna work towards, like, it’s something that we actually we actually wanna build for the industry. It’s like, are models progressing in fundamental ways? Like, are they actually getting more intelligent? Like, are their capabilities improving? Again, as opposed to simply climbing up a meaningless clickbait leaderboard. So are these models progressing? And then how much of that is kinda, like, due to us? Whether it’s due to our training data or whether it’s due to the evaluations we provide or whether it’s due to the insights that we provide all these researchers for for ways that they can improve their models.

Like, there are a way to measure that, I would love it. I think the the closest proxy we have for it today is just, like, the variety of projects that we’re creating. One of the things I really, really believe in is we want to make it easy for all of these researchers to come up with new ideas and to not be blocked by data. So the more complex, the more diverse, the more creative projects that we can provide, that is, like, almost a proxy for for for that overall North Star.

So I would say, I think you have to be willing to work hard. Like, you have to be willing to jump on a call at 2AM. And, yeah, a customer like, think one of the things that I love is that sometimes customers will call me. Although they call me at 2AM, 3AM, and they’re like, hey. Our models are freaking out. I need a bunch of data to fix it by 6AM. Can you do it? And, maybe going back to the the question of things that make me happy.

Like, nothing makes me happier than knowing that, yeah, we can we can deliver this. Like, yeah, we can deliver 10,000 data points to you in the next few hours even if you call us at 3AM to fix some critical bug, critical fire that you’re facing. That is actually something that that makes me incredibly happy. And so I think you have to be willing to wake work hard. I think a lot of people do confuse working hard with creating value. Like like, again, it’s maybe a trope to say, but you have to work smart and not just hard.

Like, if I think about a lot of what I’m doing, like, oftentimes, the the best ideas come to me when I’m just walking around, not necessarily when I’m at my computer. I mean, I think we all work really hard, but I wouldn’t confuse the number of hours we spend with actual progress. What

Good question. So at least the thing that I really enjoy is like, I I’ve always really enjoyed writing down insights in written form, and I think I’m pretty good at it. And so this ability to deliver some novel insight about a model or deliver some novel insight about an algorithm or deliver some novel insight about a dataset and communicating that to to our customers, I I think I’m pretty good at it, and it’s something I really, really enjoy.

I think a lot of people think AI safety is overblown, but I think they ignore the paper clip maximizer problem where you have AI models that are accidentally trained towards the wrong objectives, even though this is a big problem that all of the models face today with with all the issues around LM Arena and benchmark hacking. So I actually think it’s a really important problem that people should be thinking more about.

Both dangerous, but that it can be accidentally maximized towards the wrong objectives. Like, today okay. Sure. Like, if you maximize towards these LM arena objectives or benchmark hacking, the worst that will happen is that your models will regress in progress a little bit. But the, like, the more fundamental problem is that people don’t realize this. And so in the future, when the models are more powerful and, yeah, you’re you’re basically accidentally maximizing AI models towards the wrong objectives and you just have no idea what will happen, it’s almost like a similar phenomenon to what’s happening today.

But because the AI models are so much more powerful. Like, yeah, they’re literally building the code for an insurance company or they’re literally building the code for some trillion dollar company. Just the the consequences

So I think it would be twenty twenty eight if you’re talking automating the job of the average engineer, and then twenty thirty eight if you’re talking about curing cancer.

I don’t think we’re at that stage yet. At least if you’re again, if you’re if you’re working on deeper problems that aren’t just random features, like, again, if you’re concentrating your company on the 10% of problems that are most important, I don’t think models today can write 50% of the code and, you come up with 50% of ideas that are actually gonna be meaningful to your company. Sure. If 90% of your company is writing little features that nobody cares about or improving the efficiency of your code base by 1%, then, yeah.

But I don’t think we’re at a point if you’re

So if you’re a frontier lab, the question is, are you actually improving your models into raw intelligence, or are you just hacking benchmarks? If you’re a product company, yeah, the question is why the frontier labs won’t be able to instantly replace you.

go after everything, but there are so many things where, yeah, you literally just want to chat with the model in this very simplistic, universal interface that again, like, think about Google search. Like, I actually do feel I mean, I have felt that maybe 50 of the things I used to Google Google search, they are replaced by ChatGPT, or they’re even better with ChatGPT. There’s a very pleasing aspect of a universal all intelligent interface that I I

So the difficult problem, I think, for Google is they have to be willing to take a short term hit to all of your advertising revenue in order to build something better. That’s just really hard.

is how I see a world where there actually will be multiple, multiple frontier AI companies, frontier AGIs, just because every one of them will be able to go in a different direction. Like, you see it already you see it today playing out already with the differences and the the strengths and weaknesses of OpenAI and Anthropic, and I think that that trend will will continue.

I don’t know if there’ll be 10 more of them, but I can certainly see even, like, three more of them. And I just think each one will have different trade offs that they’re willing to make, different focuses that they’ll have. And, like, even today, like, Claude is really, really good at coding. Claude is really, really good, I think, at enterprise and, like, instruction following. Whereas ChatGPT is, yeah, it’s, like, more optimized for consumer use cases. Like, I think it actually has a really, really great and fun personality right now.

And then Grok, like, yeah, Grok is willing to maybe answer certain questions that maybe it should, maybe it shouldn’t, but it’s willing to, be a little bit transgressive in ways I actually think are very, very interesting. And so I actually think that just like this willingness to have different personalities and different boundaries and different focuses on your models that leads the models to be good at different use cases, just in the same way that, like, yeah, there’s like, I think analogy is there isn’t a single poet.

There isn’t a single mathematician that is, like, the greatest mathematician of all time. They all have different focuses. They all have different ways of approaching these problems. And I think that richness, like, what we often call, like, richness of human intelligence, that will apply to to models as well.

I don’t think so yet. I can actually see big, new, even more powerful model

I think it’s because it depends on what you view the long term vision for AGI to be. Despite all the immense progress that we’ve made, if you believe that there’s only we’re only like, I don’t know, 1%, 5% of the way towards AGI, because, yeah, we literally want AGI systems that can in the future cure cancer and send rocket ships to Mars and, like, design entirely new philosophical systems. Like, these are big massive problems as opposed to simply automating away the job of the average l three or l four software engineer.

And so if you believe that we’re only 2% or 5% of the way there, there’s just so much more headroom. It’s like almost like asking, do you believe 10 ago that Google was gonna be the final search engine in the world? Like, sure. If you if you’re, like, only looking forward to the, like, the next five years. If you just think of the immensity of what AGI could do, there’s so much more ahead of us than behind us, that there could be these serendipitous, very, very creative breakthroughs that just nobody’s expecting, in part because, like, maybe it’s gonna be created by by some of the AIs themselves or AIs in concert with humans.

Just so much opportunity ahead of us that it would be almost a miss to think that we’ve already solved it.

So I think it would be to focus always on the 10 x improvements that you can make as opposed to worrying about 10% trivialities.

Thank you. It’s been great chatting.