# Does Value Accrue to Incumbents or Startups in the AI Race Why Model Size Matters More Than Data Size, Why Artificial General Intelligence is Far Away, Why Carpenters Will Be Paid More Than Software Engineers & Future of Jobs with Richard Socher 20VC · Aug 18, 2023 · 46 min · 8,785 words Speakers: Richard Socher, Harry Stebbings Source: https://www.996.fm/episodes/20vc--ep-63f0074e/ ## Cold open **Richard Socher** [0:00]: The future is already here. It's just not equally distributed. And I think it will meaningfully change so many jobs. The more data there is about a job, the more that job is likely to be automated. But at the same time, there are still a ton of jobs where no one collects data. What that means is that the tasks that are physical are getting more and more expensive, and they're gonna become the new bottleneck. And then they're gonna slow down overall progress so that the GDP can't, like, a 100 x because of AI, it can only increase less than we think it might when we're deep in the bubble. **Harry Stebbings** [0:33]: This is 20 VC ## Intro **Harry Stebbings** [0:34]: with me, Harry Stebbings, I'm so excited for the show today as our guest is a true AI OG as being widely recognized as having brought neural networks into the field of natural language processing, inventing the most widely used word vectors, contextual vectors, and prompt engineering. I'm so excited to welcome Richard Socher, founder and CEO of you.com. Previously, Richard served as the chief scientist and EVP at Salesforce. Before that, Richard was the CEO and CTO of AI startup, MetaMind, which was acquired by Salesforce in 2016. And check this out, Richard has over a 150,000 citations. Bloody hell, that's more than I've done podcasts. But before we dive into the show today, ## Sponsor read **Harry Stebbings** [1:13]: you know all those mind numbing tedious tasks that seemingly take up half your day? Well, Coda is here with their new AI powered work assistant that helps you and your team not just finish tasks, but make progress. So your product team can bring a feature to market faster by using Coda AI to tag customer feedback, draft PRDs, suggest target audiences, summarizing product discussions, and more. Your sales team can engage more customers by bringing in data from sources like Salesforce, and then use Coda AI to suggest action items, meeting agendas, or lead scores. And your marketing team can drive an impactful launch with Coda AI summarizing user insights, creating briefs generated from notes, and writing more press releases to visualize taglines. With Coda AI, you can reimagine your to do list and how you collaborate, so you're not only finishing tasks, but really making progress. If you wanna work a system that lets you get back to work, you can get started with Coda AI today for free. Head over to coda.io/20vc. That's coda.io and get started for free. And speaking of amazing tools we cannot live without, travel and expense are never associated with cost savings, but now you can reduce cost up to 30% and actually reward your employees. How do you do this? Well, Navan rewards your employees with personal travel credit every time they save their company money when booking business travel under company policy. Does that sound too good to be true? Navan is so confident you'll move to their game changing all in one travel corporate card and a spend super app that they'll give you a $250 in personal travel credit just for taking a quick demo. Check them out now at navan.com/20vc. And last but by no means least, we need to talk cash. As of the June 29, you can get 5.5% yield on your cash with twenty six week treasury bills. But buying treasury bills, it's not that easy and you have to navigate a website that looks like it was made before I was born. Enter public.com. Their treasury accounts make it simple to earn a high yield on your cash and it takes twenty seconds. Here's how it works. Sign up at public.com, easily purchase twenty six week treasury bills that automatically roll over at maturity for compounding yield. Plus, there are no minimum hold periods. You can access your cash at any time with the flexibility of a bank account, of course, to receive the full guaranteed yield, of course, to receive the full guaranteed yield, you do have to hold to maturity. But here's the thing, these are t bills, which means your investment has the complete backing of the US government, making one of the safest places to park your cash. Go to public.com/20vc to lock in a historic 5.4% yield on your cash. You have now arrived at your destination. ## Conversation **Harry Stebbings** [3:58]: Richard, I am so excited for this. I have been looking forward to this one for a while. So thank you so much for joining me first. Thanks for having me. It's a great podcast. You are very kind. I I appreciate all ego inflation. But I'd love to start today with a little bit of context, because you've been in the world of ML and NLP for a long time. How did you make your first forays into the world of ML and NLP, and what did that look like? Boy, that goes **Richard Socher** [4:21]: back to 2003 is when I started linguistic computer science at Leipzig University, kind of the forays and the early days of natural language processing. But then I actually felt like there wasn't enough math in it, and I switched to computer vision. Did my master's in medical computer vision and a lot more sort of statistical and pattern recognition, statistical learning. And then during my PhD, I saw some folks work in deep learning and neural networks for very small images, you know, 32 by 32 pixel images of digits and things like that. And I thought, couldn't we use these ideas that they're using in vision for natural language processing? And that sort of started down the road of contextual vectors and inventing prompt engineering and and trying to train a single model for all of NLP. Given how steeped **Harry Stebbings** [5:11]: you are in the community, in the technology, we've obviously seen the recent hype cycle around AI. And I really wanted to start with a landscape evaluation of with the historical context you have. Is now a fundamental shift in AI, or is it the result of the hype cycles doing their work? It's **Richard Socher** [5:29]: a great question, and it's actually so difficult to navigate for a lot of people who are not deeply in it because we are at the beginning of an exponential improvement in a lot of different capabilities. At the same time, some people think the exponential will just keep on going, and we keep having these major breakthroughs. And I do think there's some inflated expectations also. People think they have to use a chatbot for every single thing out there, just like they used to think we have to use speech recognition for every single thing out there. And, you know, back in the day, when speech recognition finally started to work, people thought, oh, I'm gonna have, like, this restaurant recommendation engine in speech. That's actually probably not a good user interface. Right? So long story short, the entire level of AI capabilities is rising massively, but then on top of it, there's sort of small waves of inflated expectations. **Harry Stebbings** [6:17]: You mentioned before something that was really interesting, and I wanna start that. But it's you've been on a quest for a single pre trained model. I just wanna make sure everyone follows along with us in this conversation. For those that don't know, what do we have today, and why is that maybe inefficient? **Richard Socher** [6:32]: So today, it's finally changing. We've made progress towards that single model, but just even last year or two or three years ago, the prevailing idea was that every task in natural language processing should have its own model. You have a sentiment analysis model that just classifies tweets as positive or negative. Then you have a summarization model that takes in some long input and then summarizes it in a fewer sentences. You have a translation model that just translates German to English. You have a question answering model that takes a context and says, who's the president in this Wikipedia article that's mentioned? And then you just give that. So there are all these different sub models that people have worked on, and some people have built their entire careers on just sentiment analysis models. And so the difference and something I've been very excited about for pretty much a decade now is to have a model that you keep making better, that you keep adding to rather than restarting every new training run and so on. Imagine Wikipedia, and everyone just keeps adding to Wikipedia and keeps making one dictionary better rather than everyone who wants to build a dictionary just starts their own dictionary company and then builds it from scratch. It just doesn't make as much sense. It makes sense when humanity and people and researchers work together and keep making a single model better and better. **Harry Stebbings** [7:53]: Richard, I think one of the reasons why this show is successful is because I'm not afraid to ask the slightly more basic questions. When you explain it like that, it seems highly logical that you would improve and improve and improve upon a single model versus start again. Why is that not obvious and why has it not been that way? Oh, man. Yeah. I **Richard Socher** [8:11]: had some funny rejections on that. Actually, the paper when we submitted this prompt engineering paper, it it got rejected, and the reviewer said, oh, this is such a misguided effort. There's not a single model in the world that could do any of these things together, to which I thought the brain. It's like, we don't replace our brain. We use the same brain. So there is an existence proof for a single model for all of NLP. But it was just a very different way of thinking about the world. There's this debate, and people done a lot of research and were very stuck in in one way of thinking about it. There's sort of academic preconceived notions about how things have been and how certain people have worked on certain problems for a long time. And the notion that you should have a single model for each specific task was prevailing for a long time because it's also really, really hard. You needed the ability to train massively large models. You needed the ability to incorporate world knowledge through a large language model to really make that happen, and you needed attention mechanisms and fast GPUs and hardware. There there's a lot of things to make it work to then eventually show that that it was better. And when you had very small models ten, twenty years ago, it would have been impossible to make it work. How important is the size of the model, Richard? It is super important. You just cannot train a single model for all of these different tasks with a small model. That's exactly why and how it would have always failed in the past. **Harry Stebbings** [9:36]: I've had guests on the show before, and they say the model size isn't so important, but it's the data size that is. Is that wrong? **Richard Socher** [9:44]: It's totally not wrong. It's just not mutually exclusive. You need a large model, and you need a lot of training data for that model. Either of them in isolation, like, you know, just like imagine the simplest neural network will just predict a single one dimensional output line. Right? That's like a regression analysis. You have some input x, some output y, and you try to model where it goes. You can model that with a handful of neurons. And the simplest one is just a line. Right? A linear regression. And that model has even fewer parameters. Long story short, if you now give this linear regression model billions and billions of training data, it's not gonna learn magically anything but a simple linear line. But if you give the model billions and billions of parameters, it can learn all kinds of very complex predictive functions and abilities. So concretely, the big breakthrough on top of this idea of prompt engineering of being able to have a single model was to also use language modeling as one of those tasks. It was actually on our to do list, but we didn't get to it before others did. And the idea of language modeling is you just predict the next word, which is very easy to get a lot of data for because you can use anything on the Internet and so on, but it's actually incredibly hard. And to do it really, really well, you have to learn so much about the world. Right? If I'm just I have this sentence like, I'm in New York City, and I'm driving north too. Right? And now you wanna predict what's the next word. Maybe it's Boston, maybe it's Montreal, maybe it's Yale, but you probably wanna put more probability on the word being Boston than Yale because there are more people probably driving to Boston because it's a bigger city. And so not only do you learn about geography, but you also learn just to predict that one word really, really accurately from that one sentence. You need to learn everything about the geography of the Northeastern United States. And now if you do this billions of times across the Internet on all the chemistry articles and biology articles and so on, you learn world knowledge. You infuse world knowledge into that large language model, but only if you have enough parameters **Harry Stebbings** [11:45]: to learn it all. I need your help. I think that straight away about the access to this training data. And some people say, ah, this is where incumbents really thrive. They have the customer data. They have transacted. They have all this data. They can utilize that. And others say, no. There's so much open data today that actually access to data is highly democratized. Which side would you sit on and how would you think about that question? Because I don't know the answer. It's so funny because it's **Richard Socher** [12:12]: another question where they're actually both right. So it's complicated in the sense that unsupervised data, just raw Internet text, is easily accessible. But there's still a lot of datasets out there that are not out there. They're actually stored in private databases, and indeed, if you want to answer customer emails automatically, it's very helpful if you're Salesforce and you already have all those emails. You already have labels that people this knowledge based article answered this email or answered this question. Right? If you have that data, you can then train that particular AI for that company to answer its questions from its customers automatically. You can do that much more easily than if you're a small startup and you need to just get access to that data, you need to get permissions, and so on. And so that part is true. At the same time, the state before large language models, and we call them foundational models because you can often build on top of them very easily, before that, it was even harder. It would have been impossible for us as a small company to build a search engine that understands all of these different things in many different languages. It would have been unthinkable, but now we can actually, because of these large language models and foundational models, we can have a general sense of understanding natural language. And you can, as a small startup nowadays, build in, like, 80% solution very quickly. You layer more and more specific data for your task on top as after you have an MVP, a minimum viable product, and then you make it really, really good. Now the big incumbents, they can make it really good much more quickly, but you could get to at least an 80% solution, thanks to large LMs, like, more quickly also. **Harry Stebbings** [13:57]: A lot of people suggest in the venture community, so you can poo poo this one. But, like, when they're denigrating kind of a lot of AI startups, they they say, oh, it's a thin value layer on top of foundational models. And, actually, really the value accrues to the foundational model layer because it's this thin line on top. Is that fair? Or do you think that's total bullshit? **Richard Socher** [14:16]: You're asking a very good question that often have a more subtle answer than what would fit in a tweet. I think there are some very thin rapper companies out there that probably have very little moat, but there are also companies that people don't realize the complexity to make it really work, and they underestimate all of a sudden all the other stuff that companies need to get right to build a viable business. So concretely, you know, you could think about Instagram. Right? Instagram is like, what's the moat of Instagram? It's certainly not their AI and their back end and the brilliance of engineering. Right? It's just like a fairly simple photo sharing app with some fun filters back in the day. None of that was rocket science. Turns out you can have mode other than your back end AI model. Right? It's distribution, it's partnerships, your sales funnel processes, and so on. So there's a lot. And then there are also areas where the default large language model will not do as well. So for instance, if you wanted to be more factual, more up to date, and have citations for the facts that it tells you, you need to have a search packet. And so at you.com, for instance, we've had to build this very complex search back end with a ton of data in it and knowing when to retrieve what facts from the Internet so that your model, you can ask about Messi's Miami switch or something, and it will just talk to you about that even though the LM itself could be trained. If anything, you can think of these LMs as, like, reasoning engines, but you still need to feed them with the right facts and information so that they can reason over the right things rather than just sort of reason with what they remember. And their memory is a little bit like maybe your uncle who sometimes exaggerates some idea, like stories from the past and doesn't remember all the details exactly. He often still gets it right, but not always. And so you wanna infuse the fact into the LM and then reason over it, and that whole retrieval back end is also highly nontrivial, but it's something that some VCs don't appreciate and understand the complexity of. And then they say, oh, like, you.com's just a thin wrapper around a large language model, which is very far from the truth. **Harry Stebbings** [16:27]: You mentioned that about your uncle sometimes kind of being hyperbolic or exaggerating. It was Iman at Stability who said on the show that it's like I think it was like a crazy smart student who sometimes goes off their meds is what he described a model. And he said that hallucinations are a feature and not a bug. How do you think about hallucinations as a feature, not a bug? **Richard Socher** [16:49]: Yeah. You know, it depends again on the context. Your questions aren't perfect in that they help people understand it's neither black or white, but it's just some complexity in between. And the complexity here is that indeed the LM doesn't know your intentions and your background, and so it takes some time for them to adjust to understand and for us to train them and then to balance out, like, what they can and cannot say. And I think some cases, we even overshot a little bit. You know, there are some folks who say, oh, you cannot tell any dirty jokes anymore, or, like, a murder mystery is, like, unethical to write, so we should not ever write about a murder mystery. But no one says Stephen King or a Gothic Christie are, like, unethical people for writing murder mysteries. People just think it's great entertainment, but they're scared when an AI slash large language model predicts a token given that prompt to, write about that kind of stuff. And so in some cases, we almost overshot what we can and cannot say or hallucinate about. But hallucinations aren't a feature and are a problem for search engine. Right? We do want to be most of the time, when someone asks us about a fact, we need to bring in the right fact and then be very accurate. **Harry Stebbings** [17:59]: You mentioned that they don't know your context, I think is important. We had another guest on the show that said no models used today will be used in a year's time. Do you agree with that? And how do you think about the longevity and lifespan of model usage? **Richard Socher** [18:14]: That's a great one. That's sort of right in the sense that we're gonna update all these models. Like, we update our model every other week. And so it's not that exact model with those exact weights. But my hunch is we will still have a lot of large language models that are running in production that are that general model architecture. I mean, there'll probably still be some transformers. Even though, you know, transformers are mostly amazing because they can be trained on GPUs, there are lots of other models that we're currently not exploring because they're not trainable on GPUs. **Harry Stebbings** [18:44]: I'm, like, taking a collection of wisdom from other people and throwing it at you to hear your thoughts. I had Alex Nabbler on the show, and he said that, actually, a company's ability to transition between model is what will determine their success moving forwards. How do you think about companies transitioning between models as a differentiator and advantage mechanism? **Richard Socher** [19:06]: Interesting. You can almost start to, in the future, think about the LMs as akin in some ways to a database. You don't really care about which database people are using nowadays. Right? You if you wanna go really big, you might use an Oracle database. If you just wanna build a smaller thing, use some MySQL open source database, and there's a bunch of others in between. And I think it won't matter that much which database you use, just like it won't matter that much which LM you use. But it matters what you do with it, how you tune it, what kind of trained data you add onto it to fine tune it, how you retrieve facts into it so it can reason well over all of those things, how you may be prompted to run multiple chain of thought steps to get to a conclusion. All of those things, I think, will matter more and more. And so I would argue that it's even more important than switching between models is to be able to incorporate multiple different models because you may have a predictive forecasting model, and that is important to include into an LM. Like, an LM won't be as good in doing a financial forecast because that's not what they're trained on. Like, LMs help a ton for all things natural language because they understand so much about natural language and world have so much world knowledge, but that doesn't help you if you just have a huge sequence of numbers and you need to build a very, very accurate statistical predictive model for, like, forecasting or something. **Harry Stebbings** [20:32]: Final one before we do open versus close. But on the foundational model side, do you think all the foundational model companies that have been created today are the incumbent layer? You think there will be a new incumbent foundational model layer, or have the winners been created already? **Richard Socher** [20:48]: I mean, certainly, you can't deny that OpenAI is ahead by a lot. I predicted that we'll have a GPT-four equivalent model before the end of the year that's open source. Of course, GPT-four keeps getting better and better, so my prediction was for the version we had, like, a few months ago. But I actually think that with models like Llama two from Facebook and everyone there, I do think open source will take over a lot of use cases. Right? It's already getting close to GPT 3.5 when there's this much excitement and so many careers depending on understanding these models. Imagine all the researchers in all these universities. Right? They're all of a sudden kind of out of a job unless they have an LLM that works really, really well and they can do useful things with it. They're not gonna just say, oh, let's just, from now on, run our entire research agenda on some closed API that we cannot analyze and understand and improve and publish papers on. So they need to have a model to exist, and those are all very, very smart people. Now they don't have as many resources usually. They can train a single model for, like, 20 or $50,000,000 because they're in universities, but they're finding ways. They're collaborating, and and they're probably going to work on foundational models that are fully open source. And we now see this with, like, surprisingly Facebook being at the very forefront of it. People will layer on top of that, make it better, and then there will be open source versions you can run on your phone, and those will get better and better over time. And and so I'm I'm quite bullish on the LMs in particular getting more and more commoditized. And, yes, there will be a few foundational companies. You know, there's Cohere and Anthropic also behind OpenAI working very hard to catch up with them, and they did raise a lot of money too. And it's good to have some competition in that space, but my hunch is a lot of people will be okay with an open source model too. **Harry Stebbings** [22:42]: On the show from said that actually it was the lack of alignment between open ecosystems, which would lead to closed winning, lack of alignment around, bluntly, goals, timelines, decisions, and that actually to think that open would win would be naive. Can you see his point, or do you think that's not right? **Richard Socher** [23:03]: I mean, I certainly see that there's a lot of complexity in open models, but you also have already seen a lot of open models. Right? Like, Llama two, this was probably, like, a week or two ago, Llama two came out. Right? And Llama two is incredibly good, and it made even more progress than the past open model. And so we might not see for a while until some really strong, smart engineering project manager comes along that can herd all these incredibly smart and ambitious cats to come together and build a single model. But that would be my dream, is that we actually have a single model. And just like Wikipedia, the whole world collaborates and adds to it. And then we get ultimately an AI that anyone on earth can talk to just like anyone on earth can ask facts on Wikipedia. And, you know, like, Wikipedia takes a lot of coordination. These articles have lots of layers and hurdles you have to jump through to make an edit to that article. You have to build trust with folks and whatnot. But it has happened in the past that when there is enough interest and excitement and a strong structure that people can collaborate on open models. **Harry Stebbings** [24:11]: I love that vision, and I love that excitement for that view as a world. Why would it not happen? Coordination is hard. Right? Someone **Richard Socher** [24:17]: needs to come through, needs to get funding. There's a lot of complexity. Do you think that's the role of governments to provide the funding? I think governments can help. The problem is it's very hard for governments to fund uncertain research and science projects because when it doesn't work out, some taxpayers will for sure complain. They're like, oh, you guys wasted my taxpayer money. It's hard to build that. It'll then inevitably have to become even more bureaucratic than where, like, actually has some necessity for and so on. But I do think governments should fund science and and especially open science. **Harry Stebbings** [24:53]: When we chatted before, you said to me that search was the highest impact of NLP technology. Why? Help me understand that. **Richard Socher** [25:02]: Yeah. Obviously, it's a massive market. Right? We have a trillion dollar plus company value in that. But more importantly, intuitively, we ask search engines questions every day to learn something. They ask in their phone search engines a lot. And so it's an incredibly impactful technology to help people learn. That's sort of another really big foundational block for the Internet. Some people even think, like, Google is just sort of a yellow pages. Right? I mean, there's so many examples and use cases where you can do so much better than what we had in the last fifteen years. **Harry Stebbings** [25:37]: Richard, how do you do this without killing those original providers? And what I mean by this is say, I type in a question into you.com. You.com scrapes it for the best information it can, answers incredibly well. I now don't go to the original site because you provided it for me instead. That site often a news site or an information site or a content site relies on clicks and traffic. How does the next generation business model of the Internet work when that attribution and that throughput is not to the provider, but it is to this chat search? **Richard Socher** [26:10]: Yeah. It's a great question. We need to have that search engine that we've built be an open platform where you can actually contribute your apps to and your content to. And then if it actually brings up the context from your app and we make money with it, then you can also participate in making that money. Or you have some subscription model. Right? And you have some data that you show publicly, but then you have to subscribe. You can have that subscription happening right within your search engine by virtue of this being a much more open platform than Google. Now we've launched this open platform last year, but to be honest, we haven't had a ton of really amazing apps being added to this platform because we just don't have hundreds of millions of users. And so if your app lets you book a kayaking trip in Indonesia or something, by the end, like, you you only get, like, five people, and so been slow in an uptake on that open platform. But I think that is philosophically the right solution, and I hope people will start collaborating with us. Because if we don't win and others win, then they'll be actually fully left out. **Harry Stebbings** [27:15]: You mentioned slow on uptick. It just makes me think to a question that I ask myself so often in in this world, which is, will the incumbent acquire innovation before the startup acquires distribution? How do you think about that in this case here? **Richard Socher** [27:28]: Yeah. It is the main question. The truth is distribution won't be ever fully solved, and it's a constant uphill battle. You know, we just got deranked in all our sites from Google and saw, like, a drop right away. So we have seen that drop from getting delisted out of Google. We're working on a lot of different partnerships and, yeah, constantly thinking of of clever ways to do it that because we have some competition, small and and large that, you know, have been copying many of our features in in fairly quick succession, we have to be a little more careful there and not share all the details. **Harry Stebbings** [28:01]: I mean, delisting, that's quite petty, isn't it? Where do you think value most accrues if you were to bet on one in the next five years? Does the next wave of AI create more value for incumbents or more value for startups? **Richard Socher** [28:12]: I think it'll be a mix. I think there are companies that have been fully activated. I see, like, Salesforce having launched a bunch of really incredible features and made incredible announcements in their AI day that will make it harder for AI for service automation, for instance, because it's so core to their business. We've also seen Bing and Google copy what we have launched late last year with You Chat. They've copied us in the sense that we've launched it earlier, and then they launched something very similar three to four months after. At the same time, we don't see Google change and become a chat first search engine. They have some features somewhere else, but the main Google experience is the same. That kind of big change will be hard for Google too because they make $500,000,000 a day with privacy invading advertisements on that page. And so you don't just willy nilly change most of that page, and you get rid of the five, six ads that are on top of that page, followed by a bunch of SEO and microsites that are not as good as the ads so people click more on the ads. Like, you don't just replace all of that with a chat. Right? You just lose hundreds of millions of dollars a day. And so there is still some innovator's dilemma that will not make them change their main experience overnight. It's been very carefully tuned. Every shade of blue has been AB tested to death. It's very hard to make a massive change and improve that entire revenue overnight. So they'll slowly move carefully in their default experience, and that's sort of one opening that we have. But it's not gonna be easy. You have to keep innovating, and you have to keep working on partnerships too, try to stay clever and and a little bit paranoid as a startup CEO in in a space where you have these huge incumbents, trillion dollar something companies that would love to crush you. **Harry Stebbings** [29:54]: 500,000,000 a day. My word. Okay. Final one before we do. One final segment on AGI, which I'm super excited for, I have to say. Which incumbent do you think has done the best, and which do you think has actually lagged behind? **Richard Socher** [30:06]: I'm a little bit biased because I still love Salesforce and my time there. I think they have done a phenomenal job. You know, when we invented prompt engineering, we actually did it at Salesforce Research when I was chief scientist there. And so they've been at the forefront for a while, thanks to the research, and so they've been very, very quick also in in incorporating that technology into its products. So that's been really great to see. And then, of course, in in MySpace and search, you can't deny the fact that we have awoken the the giants. Right? Like, both Microsoft, Bing, and Google are much more actively working and innovating than they had been for the last ten years. Right? Like, since we've launched YouChat and had the first LM with a search back end and citations and web links and so on in the search context, which we launched last December before anyone else in the world. Since then, search has changed more than the last ten years combined. That is exciting to see. And in some ways, at a very abstract level, our mission has succeeded. We wanted to improve the state of the art in search. We did it. But company wise, we have a lot more work to do to actually financially benefit from those changes even more. **Harry Stebbings** [31:15]: I do wanna discuss one final element though, and we we touched on it before the show actually. But it is around AGI because there's a lot of people who are very excited and optimistic about AGI. But you mentioned to me that people might be overly optimistic. Why do you think that people are potentially overly optimistic around AGI, Richard? **Richard Socher** [31:32]: I think it's very natural for people to look at a type of progress and then extrapolate it further and further. And I think there's a little bit of overly strong optimism because we have, again, made a ton of progress. Right? Like, it's undeniable how much better AI has has gotten. But just like with flight, for instance, human flight, we went from the first motorized human flight, and then literally thirty, forty years later, we could fly loopings with machine guns and full metal airplanes high up and the speed of sound. And you're just like, wow. I mean, at this rate of progress, we're gonna have vacations on the moon, and we're gonna have flying cars, and, like, everyone will just fly everywhere all the time and and so on. And then in the fifties, the whole thing just stopped, and we're flying slower often now than we did before. People realize all kinds of issues. They burn a lot of gas, like airplanes and so on, the way they're structured right now, and we're slowing down. And, like, there hasn't been that much improvement. Like, if you think about the space shuttle, it used to be that you could land coming out of space. You land like a a plane in the space shuttle. Now we just drop people in a bucket with a parachute, and they fall somewhere in the ocean. Doesn't look like we've made that much progress in terms of returning from space, but progress is not as linear or as exponential as a lot of people think. **Harry Stebbings** [32:56]: What extent do you think that is science and the laws of nature, physics, gravity versus human reasons, skill, funding. **Richard Socher** [33:05]: It's all of the above. Right? But maybe in terms of AI and the progress, right, you think, like, we have this keep ex having exponential growth. But I'll give you an example where we already see the flattening out of that exponential growth into an s curve, and that is an image generation. It's photorealistic now. Look at some of these mid journey images. You can just say, want this exact thing, and it'll just give you a photorealistic image of the scene you just described. Where do you go from there? You can't be, like, hyper super duper photorealistic. Like, people only can see an image. Right? And that's how good it is. Now you can, of course, still make a ton of progress in video generation and, like, make the moving. But image generation itself is basically maxed out now. It's hard to go. And then in some ways, think, oh, the AI has this superhuman capability. And you could argue, yes. Like, one AI algorithm can translate decently well into 50 different languages. No human can do that. No normal human can translate that many different languages. But it's also hard to imagine what superhuman language is because language is a human construct. And if it was superhuman, it would just AIs be talking to each other with, like, 50 parallel streams or thousands of parallel streams, but humans can only register and understand language sequentially. Right? You can't have five conversations at the same time. You can't read five books at the same time. So language being a human construct and being how humans communicate thought and ideas has a limit on how superhuman it can be. It doesn't make sense to have superhuman language because humans couldn't understand it anymore to some degree. Right? Again, you can produce it more quickly. You can generate a ton of stuff, but humans can only consume it at the rate that they have over the last several years. **Harry Stebbings** [34:47]: It seems like there's this mismatch there between the asymptotic point of development there, as you said, with image generation aligned to the distribution awareness. And what I mean by that is if you go to most people on the street in a normal town, not San Francisco, they've got no idea what mid journey is. There's no question here. I'm just like, it's weird that we've reached plateauing technological development with zero awareness of it in 6,900,000,000 people. **Richard Socher** [35:13]: You're a 100% right, and I'll just jam on that. But there's a famous saying, which is the future is already here. It's just not equally distributed. How long does that take? I mean, that's the interesting thing, whereas, like, I'm so bullish and excited about AI. I have been for over a decade, and I think it will meaningfully change so many jobs. The more digital they are, the more data there is about a job, the more that job is likely to going to be automated and improved massively in its efficiency. But at the same time, there are still a ton of jobs where no one collects data. No one collects data about how my needs cleans and what are exactly all the inputs and the outputs and the actions and so on. Every house is slightly different. It's hard to constrain the environment. Full self driving on off road, dirt road, like, small roads, nighttime, fog, like, all of these things won't work for a long time. You know? But the more constrained the environments are, the more the AI can do it. The more data we can collect, the more the AI can help us automate and make things more efficient. But what that means is that the tasks that are physical are getting more and more expensive, and they're gonna become the new bottlenecks. Right? So your carpenter, people, like, who built your house, all of those kinds of jobs are going to be more and more expensive, and then they're gonna slow down overall progress so that the GDP can't, like, a 100 x because of AI, it can only increase less than we think it might when we're deep in the bubble. **Harry Stebbings** [36:37]: Richard, I'm giving you a warning that this is a shit question. Okay? So warning ahead of time. Are you concerned by the job displacement question and just the awareness that when you look at industrial revolutions and technological revolutions, there's always decade long, actually, transition periods. Whereas here, it seems like the transition period is years, like a couple of years, not decades. Are you concerned, or do you think we're overwiring about this? **Richard Socher** [37:03]: A 100%. I do think, you know, past industrial revolutions, they usually happen for an individual at a surprising rate. When you're a weaver and you got some big machine now and it just, like, totally automates making clothes, you will hate that machine. Right? And the Luddites tried to destroy those machines, and and there are people who wanna slow down and destroy certain kinds of AI that change their their jobs. And I feel empathy with those people. And it would be nice to have social systems that catch some of those people and help them learn new kinds of skills, help them incorporate those new technologies into their workflows so that, you know, if you're an illustrator and you used to be able to charge a thousand dollars for an illustration because it took you three days, now it takes you three minutes. If you don't use it and you still expect to get paid for three days, but now there are thousands of people who can do it in three minutes, that if you're not adapting to it, it will change your job landscape. And so to try to help people in that transition is incredibly important. I think it's also hard to say, let's not make things more efficient. Like, let's not have more art. Let's not have, in health care, like, more healthy people cheaply. Let's make more jobs. Feels a little bit like you'd wanna slow things down, and some people feel like they wanna go back to the mountains, right, and just live a simple lifestyle with no technology. You you see that now come up sometimes. But I I think overall, a middle class person now lives in along many dimensions, a better life than a king did a thousand years ago or even a few hundred years ago. Most of the time, people appreciate the end state of that those transitions. Like, we can have more cheaper food now. There are fewer people who have to starve. You know, a hundred and fifty years ago, over 90% of people worked in agriculture just to put food on the table. If you told them, hey. We'll have these massive machines. If you stand in their way, they will crush you to death, but, like, they'll do all of this farming automatic automatically, and now only 5% of people need to work those kinds of jobs, they would have said that's really scary, and what are we gonna do? But people always find new things to do when there are efficiencies created in in existing things. **Harry Stebbings** [39:09]: I do have just one final thing before we do a quick fire. And it's just we've mentioned before AGI and kind of why it's potentially over hyped or overexcited. What are the three barriers you mentioned to me before that prevent AGI or will put a pause on it? **Richard Socher** [39:22]: We do have a lot of research breakthroughs we still need to make in order to achieve AGI. Just like it's very hard to know when someone would invent prompt engineering or when someone would have the idea to scale up language models and transformer networks just massively and not have other cute little models and modify those, but just scale up existing models massively, like OpenAI and and Ilya, and others ideas. We don't know when those breakthroughs will happen. I find it hard to call something artificially super intelligent or or generally intelligent if it all it does is predict the next tokens. And you say, oh, predict this next token. It'll predict that next token. I think an intelligent existence probably needs to have some of its own goals and, like, some of its own mind of what it might wanna do. Maybe it doesn't wanna generate tokens. If you just wanna maximize the right token prediction, you just make sure everyone says, a a a only ever, and then you predict the right token a 100% of the time. But that wouldn't be very interesting, fulfilling intelligent ex existence. And so because companies need to make money and governments wanna have a productive economy, no one is working on AI just doing whatever it wants to do. Right? Because that doesn't make any money. So no one's working on AI setting its own goals. And until that happens, like, it's gonna be hard to think about general intelligence of almost life form that it has some some intelligence even if it is not a biological. **Harry Stebbings** [40:48]: Richard, I'm gonna do a quick fire with you. So I give you a short statement, and you ping me a quick thought back. It's as fast as possible. Does that sound okay? **Richard Socher** [40:56]: I'll try my best. As as you may have noticed, I'm not really good with the short form. **Harry Stebbings** [41:00]: No. Neither am I, my friend. So what do others not know that you know to be true? **Richard Socher** [41:06]: Knowledge is similar to the future, is already here, but not equally distributed. How important retrieval augmentation is for LMs? A lot of people underestimate that. **Harry Stebbings** [41:16]: Do AI founders need to be in the valley? It certainly helps a ton. What single element would you most like to change about the AI community? To have more folks talk to **Richard Socher** [41:27]: each other about real risks and let a small set of folks continue to think about existential risks, but not scare people so much with with very interesting sci fi general fiction scenarios that would make fun action movies, but are not the real issues at hand right now. **Harry Stebbings** [41:45]: Ten years time, what role does AI play in society then? An even bigger one than now. Were you against Elon Musk's petition to pause development? **Richard Socher** [41:54]: I did not sign it. I don't think it makes sense to pause the training of models just like I wouldn't want Tesla to stop updating the AI in my Tesla car because I would assume it gets better and better, and I don't assume at some point the car will just take off and watch the sunset by itself because it got too intelligent. **Harry Stebbings** [42:11]: Final one for you. Ten years time, where's you.com then? We have this conversation in 2033. Where's you.com then? Well, **Richard Socher** [42:20]: I think and hope it will be the default in many hundreds of millions, if not billions, of people's computers so they can find better answers. **Harry Stebbings** [42:29]: Richard, I love this. We've gone off topic many times. It's been a fantastic discussion. So thank you so much for doing this, my friend. Thank you, Harry. You've **Richard Socher** [42:36]: been one of the most engaging podcasters I've ever talked to. That was super fun. **Harry Stebbings** [42:42]: I mean, have to say, I think that I showed that I actually know a little bit about AI in that podcast. I can do more than just venture. I'm quite impressed with that knowledge. I hope you enjoyed the show there. You can find out more and see everything from that episode on YouTube by searching for 20 VC, that's two zero VC. Huge thanks to Richard for being such a fantastic guest. But before we leave you today, ## Sponsor read **Harry Stebbings** [43:04]: you know all those mind numbing tedious tasks that seemingly take up half your day? Well, Coda is here with their new AI powered work assistant that helps you and your team not just finish tasks, but make progress. So your product team can bring a feature to market faster by using Coda AI to tag customer feedback, draft PRDs, suggest target audiences, summarizing product discussions, and more. Your sales team can engage more customers by bringing in data from sources like Salesforce, and then use Coda AI to suggest action items, meeting agendas, or lead scores. And your marketing team can drive an impactful launch with Coda AI summarizing user insights, creating briefs generated from notes, and writing more press releases to visualize taglines. With Coda AI, you can reimagine your to do list and how you collaborate, so you're not only finishing tasks, but really making progress. If you wanna work a system that lets you get back to work, you can get started with Coda AI today for free. Head over to coda.io/20vc. That's coda.io and get started for free. And speaking of amazing tools we cannot live without, travel and expense are never associated with cost savings, but now you can reduce costs up to 30 and actually reward your employees. How do you do this? Well, Navan rewards your employees with personal travel credit every time they save their company money when booking business travel under company policy. Does that sound too good to be true? Navan is so confident you'll move to their game changing all in one travel, corporate card, and a spend super app that they'll give you a $250 in personal travel credit just for taking a quick demo. Check them out now at navan.com/20vc. And last but by no means least, we need to talk cash. As of the June 29, you can get 5.5% yield on your cash with twenty six week treasury bills. But buying treasury bills, it's not that easy and you have to navigate a website that looks like it was made before I was born. Enter public.com. Their treasury accounts make it simple to earn a high yield on your cash and it takes twenty seconds. Here's how it works. Sign up a public.com, easily purchase twenty six week treasury bills that automatically roll over at maturity for a compounding yield, plus there are no minimum hold periods. You can access your cash at any time with the flexibility of a bank account. Of course, to receive the full guaranteed yield, of course, to receive the full guaranteed yield, you do have to hold to maturity. But here's the thing, these are t bills, which means your investment has the complete backing of the US government, making one of the safest places to park your cash. Go to public.com/20vc to lock in a historic 5.4 yield on your cash. As always, I so appreciate all your support, and stay tuned for an incredible set of episodes coming next week.