Spending $2M to Train a Single AI Model: What Matters More; Model Size or Data Size

Yeah. That was like the first project that I worked on at Google. Yeah. I guess at the time, you know, Google had a spelling corrector that was some third party software. It was, you know, based on maybe what you’d find in a word processor at the time. So there was, like, some human compiled dictionary of maybe about 50,000 words, and any word that wasn’t in the dictionary that was in the query, it would say, you know, did you mean such and such? And this worked great for spelling correction.

It was, like, absolutely terrible for web search because people search for such a wide diversity of things on web search, like, most of them are just not in the dictionary. So, like, you’d search for TurboTax, and it would say, did you mean turbot axe? And people just learned to ignore the thing. So, yeah, but first project, like, we were just looking at, like, why are people, like, not happy using Google, and, like, spelling correction was, like, the number one issue. So I was like, okay. Let me help out with this.

And, you know, there was someone working on this, Paul who’s, you know, got on to do a lot of illustrious things in his career. He’s also one of our investors here at Character.

Okay. Google’s been just an incredible company. It’s brought so much value to, like, billions of people. And so I I’d say one of the big takeaways is that, you know, if you have a technology that is, like, really, really general and have billions of use cases and, like, ordinary people can use, like, launch it to billions of people. I remember when I joined Google, there were, like, a lot of people working on this enterprise search appliance, which, you know, it was okay. Like, I think maybe somebody had this conventional wisdom that B2B is the only way to make money, but what it actually turned out was, like, the much bigger thing was B2C, you know, like, serve something to everybody.

Yeah. Well, right now, I started this company Character, Character, and we’re taking this large language model technology, and we are just direct to consumer first. Like, here is something that is even more versatile and more easy to use than even web search in that, okay, you can use it to be your friend or do your homework or, like, a billion things, and we haven’t even thought of the best use cases yet. And then it’s massively usable. Like, all you have to do is talk to it.

So it has these two properties. To me, that means go launch it to, like, everyone in the world and let everybody in the world use it, where I think some of the other companies are taking a more, like, b to b sort of approach with a, you’ll have a foundational model company and then, like, verticalized application companies on top of it. So I’m really inspired by the Google model of full stack end to end all the way from, like, basic research to launch a product directly to consumers.

It’s super fun. Engineers like building stuff and then launching it and having, like, everyone use it immediately. And then it also lets you do all this co design of, like, you get to affect every part of the stack, which is hugely powerful and fun.

I I guess, yeah, why did I start working on artificial intelligence? Like, partially because it’s just, like, fun and what I do for fun anyway. Like, what could be better than try to get the computer to do something that it currently can’t do? But then, you know, the other thing is just to push technology forward. You know, there are so many, like, technological problems in the world that could be solved. You have, like, fifty million people a year, like, dying from stuff like old age and cancer and heart disease and, like, all kinds of stuff that we could potentially find cures for.

So rather than directly working on, say, medical research or something, I think I’ve got a lot more leverage, like, let’s push AI technology, and then, you know, that can help with a lot of the rest of it.

I think we just need to have a lot of humility that we’re not in charge of the world. You know, we’re not even in charge of the government. We’re not in charge of what individuals do. Like, basically, you know, I think our place is to provide useful tools to, like, everyone on Earth to leave people in control.

up

what do you chastise as the mission? I like this sort of motto of a billion users inventing a billion use cases. Like, because that’s sort of the superpower of this technology, and it puts our company in the right place. We can’t really guess what are the best uses of this technology. We’ve just observed time and time again, like, you put one thing out there, and that’s not really what people want, and somebody else out there, like, finds something better to do with it. Like, we put up as an example, like a psychologist character.

Like, maybe you want to talk to something and feel better. That gets a little bit of use. But then what we hear a lot more from users is, like, I’m talking to a video game character who’s now my new therapist, and this makes me feel better. We we had no idea that was going to go on. And then there’s this huge use case in, like, some mix of, like, entertainment and companionship and emotional support. We were totally not experts in this stuff. Like, our job is just to put out something general, just respect the agency of our users, of everybody out there to do what they want with this stuff.

one is that we launched. Like, that’s definitely been a frustration. You know, in the past, things seem potentially too much brand risk at larger companies to, like, actually launch and get it out there. Another aspect is we launch something general. We let people find the use cases. And then the other is there are, like, massive needs out there in the world. Like, okay, there are billions of people who, you know, feel like they need someone to talk to. Combine those elements provides people with something general that they can use, and there are people out there with needs, they’re going to find it.

Yeah. I think there’s huge value in the connections between people and, you know, moral value as well. Like so last thing I wanna do is take people away from human connection. In a lot of ways, we wanna help with human connection. A lot of the people who don’t have friends and who are not as well connected, one big source of that is just social anxiety. There are huge numbers of people who are, like, uncomfortable, and we’ve gotten testimonials of people who said that that they were uncomfortable talking to other people and, like, this is great practice that actually helped them build up practice in either social situations.

Do

You get used to talking to someone who’s not of human. I would very much like it to be the former. That is gonna be ultimately up to the users.

The main things we need to do, make it very general so we’re not, like, cutting down on the use cases, make it usable. People think of those two things as being in opposition to each other, being versatile and being usable. You know, we talked to, like, some potential product managers early on, and they all say the same thing. Oh, yeah. Pick your verticals. Narrow it down to make it usable. And, like, no. We’re not going to hire these people. That’s, like, the opposite of what we want to do.

We want to build something that is usable but very, very general purpose. So there’s sort of that dichotomy.

That’s been the magic of neural language modeling. You know, the previous systems were all these rule based systems, like fantastically complicated systems with millions of handwritten rules. Really, really complicated. It requires knowing something about linguistics and about state of mind and, like, all kinds of stuff. The new way of doing things with neural language models has none of that. Like, I could know, like, zero about language in particular, other than it’s like a sequence of words. So it has nothing to do with understanding language at all, and there are not millions of rules.

It’s actually relatively simple, kind of like a big black box. It all boils down to this one beautiful simple problem of you have this sequence of words that’s like the beginning of your document. Guess what the next word is. Give me odds on what the next word is in the sequence. And that problem is called language modeling. Just guess what the next word is based on the previous ones. You know, so I got involved with this, you know, around, like, 2015. There were some other folks at Google, you know, working on this problem.

They’re like, how good can we make it? And this struck me like, hey, this is the best problem ever because it is so simple to state, and there’s a huge amount of free training data. You can just, like, download the text of the web off of, like, whatever you want. Common crawl. You’ve got, like, billions to trillions of training examples of guess the next word. And then if you can do it well, then this thing can just talk to you. It can be, you know, the better you do it, the smarter it gets.

It’s hugely, hugely general and useful, super simple to state, and now we just have to do it well. And people started building better and better neural networks, which I guess got renamed deep learning as some sort of rebrand. You know, neural networks had a bad name because the hardware wasn’t good enough. But now that the hardware is good enough, they switched the name. But anyway, so people were using deep learning for language modeling, and the bigger and better you made these things, the smarter it got.

And, like, around 2016, the most useful application, like, killer app for this was machine translation. It was about smart enough to take English and translate to French. You know, massively, massively useful. It lets everyone in the world communicate with each other, but, you know, still not smart enough to, like, carry on an interesting conversation or do your homework or, like, any of those things. But there seemed to be a pretty clear path. Hey. Let’s just make this thing bigger, better, smarter, and it’s going to get these capabilities.

I think there has been a lot of technological progress, both quantitatively and qualitatively, in that the models that were there in, like, 2016 were too dumb to be fun, the the neural models. Then there were all of the chatbot stuff you heard about back then was these rule based systems that were just highly fragile and not going anywhere, that you just needed more and more rules, there’s, like, no way to think of, like, all the things that could come up, and they just don’t generalize. That wasn’t going to work, but at the same time, we were progressing on the neural network solutions, which were going to scale.

It took some amount of time. I’d say around 2020 was when really impressive stuff was sort of in the lab, but not launched. So my cofounder, De Freitas, he’s like on this lifelong mission to do chatbots. Since he was a kid in Brazil, he’s wanted to build like open domain chatbots.

Yeah. Probably the size of the model is the bigger challenge. We can get a lot of data, but actually the number one thing that’s important is how much computation you do to train it. So you want to train a bigger model, you want to train it for longer. So the two things are both important, but the real constraining factor has been how many operations of computation it takes to train it. Because if you make it bigger and you train it for longer, both of those multiply into how long the thing takes to train.

So people have been building better and better, essentially, supercomputers to train these models.

biggest constraints on your models today, do you think? Computation. So, you know, the model we’re serving now, we trained last summer and spent about $2,000,000 worth of compute cycles doing it. We will do a lot better, you know, in the near future. But if we get a lot more better hardware, which we are getting, and spend longer training the thing, we can train something smarter. Back in, say, 2016, you could train something that was, like, smart enough to, like, translate languages, but not smart enough to, like, answer questions or be fun.

Both are useful. Like, data that you get from users is great because it tells people, like, what users like or, like, what users like in some particular application. It’s kind of like a training a human. Like, most of what’s important is you have decades of experience training your brain on stuff that is not really specific to your task at hand or your occupation, but you’ve kind of gotten a generally good understanding of the world and gotten generally intelligent. And then you can improve on that dramatically by getting a smaller amount of training in the task you’re doing right now.

But both will contribute, and we do have a huge amount of data flowing in from users.

experience coming from Google is that a startup can move way faster than a big company and can launch products in ways that large companies are just going to move too slow because they’re worried about compromising their existing product.

I’d say the users win. The users are going to have a lot of options. But, you know, on the business side, I think there can be a lot of winners. There is just so so much value about to be created that there’s going to be room for multiple players in there. Big companies doing what big companies are good at, startups doing what startups are are good at. We’re we’re gonna try to move our company from being a startup to being a big company as fast as we can there.

And then a lot of just individuals and universities and such in that the hardware is progressing so fast that what you could do at a big company one year, a few years later, you’re gonna be able to do, you know, at a university lab or in your garage.

have said closed wins. The ability to mess around with things at a small scale is going to lead to way more research being published even if some of the larger entities are no longer publishing research because they’re trying to maintain competitive advantages. And obviously, though, there’s also, like, economics of scale of both training the best models and serving. Like, if you want to serve a product, you can do it maybe a 100 times more efficiently if you’re serving many, many people at once and kind of batching things together versus, you know, serving, like, an individual or, you know, or you’ve got your own rig to run your language models in your basement or something.

I think the one message I have is that the best applications just haven’t even been invented yet, that we’re still at, like, invention of electricity kind of moment or invention of the computer where we don’t really know what the coolest things are going to be.

Our strategy is like launch something general. Let people do what they want with it. If these models are hallucinating, which they certainly are and we advertise that they are, then the use cases that emerge first will be ones for which hallucination is a feature. So I’m happy to have entertainment and emotional support and fun be the first use cases. I’m happy to have productivity be the first use case, but let that happen naturally based on what the technology is good at.

I think you could ask the same thing about any company that is selling some very general tool, like a company that’s selling computers or electricity for that matter. What is electricity for? Is it for fun? Is it for productivity?

We believe that individuals should make that decision. You know, we we want we wanna respect everybody’s agency. I think that’s one of the big fears around that AI will take away your agency. We wanna come come down on the side of we love humans. We respect humans. This is a tool for people. Have you enjoyed that transition to CEO and scaling CEO? I have. I still do a lot of technical work and leadership, which is big. I’m gonna stay CEO because I want the company to make the right decisions.

I don’t judge what I do by how much fun it is. It’s more like what’s the most useful. So very, very happy to be doing what I can.

Yeah. It it wasn’t like a matter of am I going to be having more fun being a startup CEO than an ML researcher at Google. It’s it’s more like I want to push this technology forward. Like, what’s the best thing I can do?

Yeah. I think this is, like, a really interesting concept there. After sort of becoming a parent, like, early twenty tens or something, we I mean, one thing I re remembered not being a lot of fun was, like, getting woken up in the middle of the night and getting, like, way, way less sleep than I would have liked. And, you know, a lot of things about parenthood are absolutely terrific and super fun. I think it made me more religious. It sort of decided to take a change of attitude from, like, what is fun right now to I should be thankful for having the opportunity, you know, to do something important and meaningful.

So I I think that it’s it’s kind of been a big attitude shift in in growing up.

Beautiful. Yeah. Yeah. Not everything in the world is your responsibility, that you should understand what is your responsibility and what isn’t your responsibility. And I think that works really, really well in marriage and and parenting. I mean, I think religion’s a lot about that as well, like, sort of beliefs about what’s your responsibility and what’s not. Like, that’s the question that it’s answering because you don’t have a solution to just staring at your face in nature. Like, what should I be what do I need to do?

What what should I can be concerned about, and what should I not?

I mean, like, luckily, we’re recording a lot, and you could like the crazy parts.

really fun.

This technology is just gonna get way smarter. I think we’re at a Wright Brothers first airplane kind of moment in AI that there’s, like, a lot of momentum going on both in building better hardware and in research. So whatever really amazing applications you’re seeing now, it’s probably nothing compared to what’s gonna happen in the future. What do you think that adoption timeline is? I think things are gonna move very fast. Like, I think we’ll see, like, a lot of very cool stuff happening in the next one to three years.

I think, like, externally, it looks like entertainment app. Yeah. But, you know, really, we are a full stack company. We’re like an AI first company and a product first company. Having that is a function of picking a product where the most important thing for the product is the quality of the AI. So we can be completely focused on making our products great and completely focused on pushing AI forward, and those two things align. What

Yeah. I mean, there’s so much stuff being published. It’s hard hard to know what’s good. And I think a lot of that has to do with the fact that this field is kind of alchemy right now. Like, no one knows exactly what is going to work. You know, you have a lot of people trying lots and lots of different things. You you can come up with hits by having, like, a good intuition of what will work on the ML side combined by, like, a good mathematical understanding of what will run fast on hardware that you can buy or that you can build.

So there are some hits that come out and, you know, people will adopt, and it’s combined with a lot of noise. Negative results are not useful because they could be negative because somebody just made a mistake, and there could be a bug, you know, that it didn’t work for some other reason. What is interesting is positive results that are proven out by experimentation. If somebody can say, hey, I did something and did better at this well known problem, then that gets interesting. Then everyone tries to figure out, okay, why does this work?

How can I adopt it? Penultimate one, Noam. What did you believe that you turned out to be wrong on? Well, when I started getting into deep learning, I, around 2012, had a bunch of early failures trying to do sparse computation. You know, I was like, okay, you must be able to do something better and more efficient by building a sparse network. That was so wrong because I did not understand that the reason this whole field is working so well is because now we have this magic hardware that’s great at these dense matrix multiplications, and so you can do them, like, orders of magnitude faster than you can do anything that involves poking around in memory.

And there was no one there to, like, explain that to me when I got started with deep learning. So, okay, as soon as I sort of understood that part of it, it’s like, okay, let’s do sparsity, but let’s build it out of these dense building blocks so that it’ll run fast and then publish this sparsely gated mixture of experts idea that’s only now getting, like, a lot of adoption. But, you know, that that was back in 2016 and that have had, like, a string of hits ever since, which I will attribute to divine intervention, but also to understanding, like, the hardware mechanics and sort of quantitative computation aspects of the field.

have absolutely no idea. Like, we will see what technology is like then, but, you know, it’s just important for us to be agile. If you were in 1900 and asking where some company would be in 2000, there will be such technological improvement before then that it’s roughly impossible to predict where any company will be.