# Are Foundation Models Becoming Commoditised? Do OpenAI and Anthropic of the World Have a Sustaining Moat? Why Smaller Models May Work Better? Why Incumbents with Data Power Win the AI War with Christian Kleinerman SVP Product @ Snowflake 20VC · Sep 22, 2023 · 45 min · 8,564 words Speakers: Christian Kleinerman, Harry Stebbings Source: https://www.996.fm/episodes/20vc--ep-6b5c5921/ ## Cold open **Christian Kleinerman** [0:00]: We've seen companies that with seven employees have creating models that are comparable for some use cases to what OpenAI or Anthropic do. At the end of the day, it's a data problem. And model, they're getting commoditized until the next big innovation comes and you allocate some more value to the model. If anything, model size will influence things like cost and latency. So smaller may be better. Welcome **Harry Stebbings** [0:24]: back. This is 20 BC ## Intro **Harry Stebbings** [0:25]: with me, Harry Stebbings. And joining us in the hot seat for this deep dive on generative AI is Christian Kleinerman, SVP of product at Snowflake. Before Snowflake, Christian spent close to five years at Google as a senior director of product management at YouTube working on their infrastructure and data systems. Before YouTube, Christian spent over thirteen years at Microsoft serving as general manager of the data warehousing product unit where he was responsible for a broad portfolio of products. Before we dive into the show's ## Sponsor read **Harry Stebbings** [0:51]: day, you know all those mind numbing tedious tasks that seemingly take up half your day? Well, Coda is here with their new AI powered work assistant that helps you and your team not just finish tasks, but make progress. So your product team can bring a feature to market faster by using Coda AI to tag customer feedback, draft PRDs, suggest target audiences, summarizing product discussions, and more. Your sales team can engage more customers by bringing in data from sources like Salesforce and then use Coda AI to suggest action items, meeting agendas, or lead scores. And your marketing team can drive an impactful launch with Coda AI summarizing user insights, creating briefs generated from notes, and writing more press releases to visualize taglines. With Coda AI, you can reimagine your to do list and how you collaborate, so you're not only finishing tasks, but really making progress. If you wanna work a system that lets you get back to work, you can get started with Coda AI today for free. Head over to coda.io/20vc. That's coda.io and get started for free. And speaking of amazing tools we cannot live without, travel and expense are never associated with cost savings, but now you can reduce costs up to 30% and actually reward your employees. How do you do this? Well, Nabla rewards your employees with personal travel credit every time they save their company money when booking business travel under company policy. Does that sound too good to be true? Nabla is so confident you'll move to their game changing all in one travel, corporate card, and a spend super app that they'll give you a $250 in personal travel credit just for taking a quick demo. Check them out now at navan.com/20vc. And last but by no means least, we need to talk cash. As of the June 29, you can get 5.5% yield on your cash with twenty six week treasury bills. But buying treasury bills, it's not that easy and you have to navigate a website that looks like it was made before I was born. Enter public.com. Their treasury accounts make it simple to earn a high yield on your cash and it takes twenty seconds. Here's how it works. Sign up at public.com, easily purchase twenty six week treasury bills that automatically roll over at maturity for a compounding yield, Plus there are no minimum hold periods. You can access your cash at any time with the flexibility of a bank account. Of course, to receive the full guaranteed yield, of course, to receive the full guaranteed yield, you do have to hold to maturity. But here's thing, these are t bills, which means your investment has the complete backing of the US government, making one of the safest places to park your cash. Go to public.com/20vc to lock in a historic 5.4% yield on your cash. You have now arrived at your destination. ## Conversation **Harry Stebbings** [3:36]: Christian, I am so excited for this. I've heard so many good things. So I would love to start with your entry into product. How did you come to be SVP of product at Snowflake? Let's start there, Christian. **Christian Kleinerman** [3:47]: Thank you for having me, Harry. I born and raised in Colombia in South America. I did a startup there. I learned what not to do. I did another startup in The US. I learned what else not to do. So at some point, like, I need to learn from from the guys that really know how to build software. This is 1999. I joined Microsoft, did a long stint in data all the time, SQL Server, appliances when appliances were the thing to build, and then cloud services. And from then I went over to YouTube at Google where I was responsible for the infrastructure, including data systems. And I think all of that set me up for understanding data, a data junkie. When the opportunity opened up for Snowflake, I'm like, I I could appreciate the technology and the type of company, so I'm like, I'm ready to be there. **Harry Stebbings** [4:32]: It was the charm and charisma of Frank. I don't blame you. I had the same feeling when he looked into my eyes. I do have to ask. You mentioned that you learned what not to do. If there were one or two things that you really learned what not to do, what would they be, Christian? **Christian Kleinerman** [4:45]: I would say talent being the driver of truly great outcomes. I would say don't ever compromise on talent. Don't ever say, yeah, this person doesn't have the background, but maybe has the right intention. Just take a bet. No, I think that nothing substitutes talent. That's a very clear lesson learned. And the other thing that has been very clear is building a scalable business is difficult. One of those startups, built some scheduling software for airlines. The thesis was you build it once, works, then you just resell it, and you can be the next Microsoft. It was not the case. There was a lot of customization. It would turn out into more of a services business. So scalability and building platforms was another lesson learned. **Harry Stebbings** [5:29]: If that's some lessons learned from respectfully the startups that maybe didn't go to plan, we could say, when you think about Google and Microsoft, they're such symbolic institutions in our environment. What are one to two of your biggest takeaways from thirteen years of Microsoft? I mean, shit. It's a long time. And then, you know, the four to five years that you had at Google. What are one or two of those big takeaways? **Christian Kleinerman** [5:49]: For Microsoft, I I think I got the ease of use and value of simplicity in products. At the time, SQL Server was coming from way behind competing with market leaders that were Oracle and IBM. And the way in was not to have every single feature and capability that those technologies had. It was just simplify things. If you turn something that is a subset of the capability, but dramatically easier to use, that gets a following. And that was a very, very clear lesson learned and I've seen it over and over. And by the way, the Snowflake story follows a big part of that journey, which is if you simplify things to a point that it is delightful to use, people adopt. That's from the Microsoft time. Let me think from the YouTube time, maybe the biggest lesson learned. Consumer products have many more elements beyond just technical difficulty. There's a lot of timing, what are the trends with consumer behavior. Sure, you need to have a good business model, need have good technology, but there are things that you can't control. Is simple always better in product? I would say yes. You'll say all things being equal, there are points where you will oversimplify, but I do think that make things as simple as possible and no more. **Harry Stebbings** [7:03]: If you could call yourself up the night before your first role in product, before Microsoft, before Google, what would you call yourself up and advise yourself knowing all that you know now? **Christian Kleinerman** [7:13]: The fundamentals of making something work as advertised make a very big difference. So simplicity is part of it. Even things like latency make a big difference. So there's nothing like the magical experience of you try to use a product and then whatever technology or device or an appliance at home. And if it just simply works and does what it's expected, I think that is a magical thing. It's very hard to make happen in all situations, but that would be the big word of advice. **Harry Stebbings** [7:41]: I I do wanna focus the show on a joy and passion for both us. It's funny. We we discussed a little bit in terms of topics to discuss today, and I was very excited when I got your suggestions. And I wanna start from the top, is kind of ecosystem level, and then we'll move down. But is generative AI is kind of the biggest buzzword of today? So when we think about gen AI as a buzzword, like top down, how do you analyze the ecosystem today? Is the hype overblown as a starting point? **Christian Kleinerman** [8:07]: I would say that for sure there is hype. For sure there's a FOMO on what are you doing on on Gen AI and people are rushing to somehow stitch some Gen AI to their products. But if you look past that, there is fundamental innovation there. There is fundamental disruption. There is the opportunity to change pretty much every single interaction between humans and computers into something that is friendlier and simpler. So yes, there's fog and there's noise around it, but it will clear up and at the end of the day, it's gonna be a different world. **Harry Stebbings** [8:42]: Do you think mobile is a good analogy to AI or do you think it's actually more significant? **Christian Kleinerman** [8:48]: I think it's comparable. I think it's of the scale of the Internet. I think it's of the scale of mobile. One of those things where anything and everything will get better. **Harry Stebbings** [8:56]: Can I ask what do you think is most exciting? When you look at the different verticals, use cases, opportunities, what do you think is most exciting? **Christian Kleinerman** [9:04]: I would say that this is a real shot in the arm to the creative businesses. That's where we saw the initial use cases and stable diffusion or mid journey and those things. In many ways because what in other industries we we would call a bug or a hallucination, in the creative world, there are features. There's a it's a goodness to come up with something that has not been done before or it's some mix and match of existing things. So creative industries are probably the sweetest spot. I love what Adobe has been doing with their products and how they've integrated Gen.ai. A massive kudos to them. Then there's the opportunity in every other industry and every other vertical. It's only that requires a little bit of understanding, correctness, understanding data maturity, things like that. But every customer that I talk to these days, they're looking into doing something. They're all trying to figure out how to get started and how to get there, but I think it applies to all sorts of businesses. It's interesting **Harry Stebbings** [10:00]: you said there about kind of they're all looking to do something. You know what I'm finding there? Because I speak to these enterprises too. They've got no freaking clue how to do it, Christian. They're all like, yeah, so it's exciting. How do I do that, Harry? And I think the biggest businesses in AI will be built in the implementation services businesses, helping large enterprises implement AI in a meaningful way into their enterprise over the next decade. Do you agree with me in terms of this lack of enterprise education on implementation and the potential opportunity for implementation services given that? **Christian Kleinerman** [10:30]: I agree part of it is services implementation education, but I also think that the stack and the way to think about this is evolving. I don't think that we have a very clean, this is the three or four components that you use and this is the type of use case that you apply. Companies are playing with, hey, the LLM is magical and you send it send it to questions and magic answers come back. Some others have combined vector retrieval with LLM's. But within that, there's a lot of variability. Which model? Which database? Or which vector database? How much do you prompt versus fine tune? So I would say a lot of that is still being figured out. I think the stack continues to evolve, will continue to mature, and in parallel, okay, then company need to get educated on when to use what for what use case. **Harry Stebbings** [11:19]: I get you. In terms of kind of verticals that are fastest to adopt, I think this is the other thing that we will get a little bit overexcited about, which is like speed of adoption always takes longer than people think. Many, if you can believe it, Christian enterprises in Europe still have no idea what Slack is. And so my question to you is, when we think about the different verticals, who do you think is first to adopt and the fast movers? Who do you think is slow to adopt? **Christian Kleinerman** [11:41]: I think it completely correlates with data maturity. You may have heard some of us at Snowflake talk about there is no AI or Gen AI strategy without a data strategy. It's not just a line. It's a truth that we strongly believe in. From that perspective, I would say financial services are at the forefront. Most of the financials have figured out for a long time how to organize data and leverage data for competitive advantages. Go look at the sophistication of hedge funds as an example. Retail and CPG companies have also been very wise at using data. Who would be on the slowest to adopt? Maybe I would point at public sector. Not that they're not data savvy, but they have a lot of regulation and constraints that may make it harder for them to adopt. How **Harry Stebbings** [12:27]: do you think about incentives being a problem? Point here is like, if you think about healthcare, healthcare is largely dictated by, you know, government budgets by the NHS in The UK or alternative in The US, which is run by politicians. You don't want to replace nurses with AI driven robots procedures because then you'd be firing nurses. And then the front page of the newspaper is Biden or Rishi Sunak fires nurses. So the incentive mechanism is misaligned to increasing efficiency in healthcare or in a lot of public services because it likely leads to job loss. How do you think about the incentive function being the problem for adoption versus implementation speed? **Christian Kleinerman** [13:07]: I think it may be true that once we're ready to wholly replace nurses and other functions, probably there's an incentive problem. But I don't think that we're there yet. I think of most of the use cases right now are around productivity boost or assistance, co pilots as opposed to replacement. Think of it as I want to help people be more productive, not wholesale replace. So at this point, I would say the incentives should be working. Maybe a year, a couple of years from now, what you're saying becomes true. **Harry Stebbings** [13:37]: You said there about there's no gen AI without being incredible data strategy. You said to me before generative AI is democratizing data access. You left me with that as a cliffhanger, Christian. How so and why do you believe this? **Christian Kleinerman** [13:51]: Yeah. If if you think of the role of traditional business intelligence technology, it was to sort of bridge the impedance mismatch between the business and business terminology and business users and a technology that frankly is just for a few people. Like writing SQL statements is not something that most people in a company do. And that was the role of that technology to do that translation, but it still requires some mapping and curation and then effectively how do you inform that impediments mismatch. I think Gen AI has the opportunity to turbocharge this type of translation where the language is natural language and the answers come in natural language, but along the way there's traditional database lookups, traditional retrieval. And has the opportunity to democratize data dramatically more than where we are today. Now that BI has not done a really good job, but I think it's it's gonna be now data for everyone. **Harry Stebbings** [14:45]: When we think about kind of data as a competitive moat, I've had a lot of people on the show say before, bluntly, that data is so freely accessible today. It's no longer this prized possession that incumbents can hail and use to their advantage given how freely accessible it is. To what extent do you still place a premium on data ownership to leverage versus the freedom to access data? **Christian Kleinerman** [15:06]: Yeah. I I would separate. There's both public data and private data, privately owned by enterprises. But the premise of your question is based on public data. And I would say if things did not change, probably the language models or the models in general would all converge towards they're all training on the same data and at some point this what mix of data you use, but you'll trend towards the same answer. The interesting trend towards there is the notion of many companies realizing that their data is being used and monetized by these models. So there are companies rethinking and changing their data policies. Are you allowed to crawl me? Are you allowed to train models with this? I think all of this will shift in the next six, twelve months. This is all starting already because in the same way that search changed the rules of engagement with public data, Gen AI is doing the same thing and and the companies that are behind the data are doing so. **Harry Stebbings** [16:01]: You mentioned that companies realizing that their data is being used bluntly and they're not being able to monetize it and they're losing eyeballs because of it on their sides. How do you think about the business model of the future to ensure that they don't lose this revenue despite their loss of data control? **Christian Kleinerman** [16:16]: I don't know if they're gonna be able to keep all the existing revenue, but for sure they should be able to capture some amount of revenue. But clearly, their data is valuable. Right now it's not being paid for. That says that there's a value gap there. And you see it in some of the license terms that are being floated around. I think it will shift so then it changes the economics of all of this. **Harry Stebbings** [16:36]: If you were to place a value on data versus model, what would you place a 100 as your total pie size? Is it $80.20? Is it $50.50? How do you weigh the importance in terms of data and model? **Christian Kleinerman** [16:50]: The vast majority goes to data. 90 plus percent. Certainly there is a lot of IP and technology that has gone into how do you build these models, but that is becoming less a differentiating aspect and what's becoming bigger is data. If anything, hear folks doing the math on, are we gonna run out of public data to improve these models significantly? **Harry Stebbings** [17:11]: Why are models as little as 10? And why is such value being placed on the likes of OpenAI, Bard, Anthropic, if actually models are 10 and not a significant chunk more? **Christian Kleinerman** [17:23]: If you look at it where where things are today or where they were six months ago, maybe models would have taken a bigger edge because they paved the way on how do you model the data. You could make the case data has been there all along. But I would stick to the 10% or the smaller number because now you see how many foundation models are being created. We've seen companies that with seven employees have creating models that are comparable for some use cases to what OpenAI or Anthropic do. At the end of the day, it's a data problem. And model, these are strong words, I think they're getting commoditized until the next big innovation comes and you allocate some more value to the model. Kass, what's the next big innovation do you think? What has happened for for language models is coming for computer vision for images. Demortization, how do you simplify it? Then there's the intersection of those true multimodal languages, which there are there are many of them out there. But how do you turn into it's completely seamless to go and intersect images, speech, text, all of it into a richer human computer interface. **Harry Stebbings** [18:28]: And do you think it is the existing model incumbents, your OpenAI, your Anthropics of the world, who chase down those innovations, or do you think it's net new platforms? **Christian Kleinerman** [18:38]: I'm pretty sure all of them are chasing that. I don't know for a fact, but I'm willing to to venture that they are definitely chasing those innovations. What we're seeing right now has generated such a big spur of creativity. Most new startups getting created, somehow they all wanna chase some aspect of the generic revolution. So maybe it's a hybrid of both. You **Harry Stebbings** [18:57]: mentioned their startups chasing it. So many VCs say, they're just a wrapper on top of a, you know, GPT model, and it's kind of brushed off in that way. It's just a GPT wrapper. Do you think that's fair and actually these models and the providers will create and kill off all the startups with their verticalized use cases? Or do you actually think that VCs are being shortsighted with the kind of hand off of it's just a wrapper? **Christian Kleinerman** [19:21]: I think there's many categories. There are some very shallow wrappers on top of GPT four. I don't place much value on them. I I usually ask, hey, how long did it take you to build this? Oftentimes, it's a week or two. I don't think there's a company there. I do think that there are some very deep wrappers on top of GPT-four that apply domain specific, legal or other domain that I think you end up with a true way to bring GPT-four to a given market or industry. I think those are value. But my comment on startups was there are actually many startups chasing the different aspects of innovation on the core technology. I I I was at a dinner a few weeks ago and someone was saying, we're trying to blend fine tuning with prompting. Someone else was saying, we're trying to address the limitations of the transformer model. Someone else was trying to look at how to do computer vision better. So once you start looking at the core tech, not just application of existing models, I think there's innovation everywhere. There will be a number of new startups creating new things and probably the OpenAI and Anthropic are continuing their innovation, those are effectively research companies. **Harry Stebbings** [20:29]: And they absolutely are in many ways research companies. The thing I want to ask is like often the size of the model is quite hailed. How important do you think model size is today, Christian? **Christian Kleinerman** [20:39]: I think it depends on the use case. For a generic consumer product like ChatGPT, where anything is fair game, the model is supposed to know about every possible topic. It speaks every language, etcetera. I would say that for those use cases, a model that is large and has lots of cumulative knowledge is very valuable. For specialized use cases, which is what I see more in the enterprise, model matters less. Model size. If anything, model size will influence things like cost and latency. So smaller may be better. And now there is plenty of examples that have been run where a smaller model fine tuned where a specific purpose or a specific dataset produces results better than a generic model. So it's use case dependent. **Harry Stebbings** [21:25]: Do you think we actually do that? We downsize model size to increase efficiency, you know, to reduce latency, to reduce cost? Or do you think we actually just increase efficiency of compute to be able to ingest and work with larger model sizes more efficiently? **Christian Kleinerman** [21:40]: Both. And and as I was mentioning startups, one of the companies that that I talked to was working entirely on model compression only to improve latency and cost. So I think there is research and and and development on on both. Sure, the computer is getting faster and cheaper, but smaller models are better. Like, if if you want to have one or more model calls in the serving path of a consumer product, you have a budget of a few milliseconds to go make things happen and size will matter. **Harry Stebbings** [22:07]: It's interesting that kind of thinking about kind of the efficiency, the cost, the latency. I I had the norm from character on the show and and he said the biggest problem that we have is the cost of training. It was like $2,000,000 to train one single model. How do we think about the cost of training changing over time? Will we see a democratization there which will allow startups to fine tune and train themselves? Would it continue to be high? Help me understand that. For sure, **Christian Kleinerman** [22:31]: like all of this, the computer is trending down. But the other piece is there's a lot of reinvention of the core training. If you think about how many of the data sets and data sources are common between all the foundation models out there, and they're going through the exact same processes or very similar processes. Are there ways to take a common subset and then use fine tune on top of it and avoid the cost? I think we've seen reasonably good results under that path. I do think that both approach wise and compute cost wise, cost of training is gonna get lower. **Harry Stebbings** [23:04]: Cost of training gets lower. There's also another thing which is like the longevity of models. I I have one guest, email at Stebbity, and he said that no models we use today will be used in a year. Do **Christian Kleinerman** [23:14]: you think that's true? Depends on how you define a model. If you say LAMA and LAMA two are the same model, then yes, all of them will continue to be used for a while. If you pin a specific version, yeah, for sure there will be new versions of any of those models, Claude, Claude two, etcetera. But the reality is there will be new models and new refinements on an ongoing basis. **Harry Stebbings** [23:37]: When we think about kind of those refinements, companies that are able to transition between models obviously have the most flexibility. I think it was Alex at Nabla, another incredible founder we had on the show. He said the best companies will be able to transition between models at ease and those that can will win. Do you think that's right in terms of the importance of flexibility's transition between models being a core determinant of success? **Christian Kleinerman** [24:02]: A 100% agree. There is so much innovation in the landscape of models that anyone that builds too tightly coupled to a given model is sort of giving up optionality for the future. I just came back from Japan two days ago and a big recommendation I was doing in large forum was make sure that you build optionality into which model you're leveraging. Even though it's easier to learn the ins and outs of a single model, I think the optionality matters especially there's way too much change and innovation going on. **Harry Stebbings** [24:30]: What does it mean to make sure you have optionality? What do you do differently if you're thinking with optionality in mind? **Christian Kleinerman** [24:37]: Oh, you'd want to be able to plug in different models, but it's not just replace the model and use everything else the same. Certain models respond different to a specific type of prompt. In the same way that you have a hardware abstraction layer or a cloud abstraction layer, you should have a model abstraction layer that knows how to translate a specific request that your application needs to do to a model with its intricacies or or specific characteristics. **Harry Stebbings** [25:01]: In terms of the transition between models, implementation is not that easy. You're not just handing over data with ease. What are the biggest challenges to adoption do you think for startups and for companies moving forwards when they think about working with new models and data migration to them? **Christian Kleinerman** [25:18]: There are a lot of issues. Probably the most obvious one is around the correctness and dependability of answers. If you ask folks one of the key concerns, they're like, these models make up stuff. So that is the obvious one. But then there are second order issues, maybe less obvious for people which is security and privacy of data, the data that is used for the question. And then there are even more complex questions on the rights to the answers. A Wall Street company asked me, if we feed a number of portfolio trading strategies into a model and it makes a recommendation and the recommendation makes money, could anyone have claims on that answer? And it gets very complicated very quickly. **Harry Stebbings** [26:00]: How do we solve the security side? I totally get you. You've got customer data, you've got transaction data, but you do want access to the LLMs. What do enterprises do to get the benefits of access to them without the security issues that come with just sending thousands of customers data. **Christian Kleinerman** [26:18]: Evolve, there are platforms where it's easy to bring LLMs to the data as opposed to send large data volumes to where the LLMs are. That's where you see Amazon has new new services to do this. Microsoft, believe, has it with with Azure OpenAI. The trend is create private and secure endpoints that can run close to your data and by implication, not only don't have to move a lot of data, but more important, there are some assurances on what is done with your data. **Harry Stebbings** [26:48]: So we bring them closer in terms of the endpoints. You mentioned the hedge funds there and like the right to the answer and whether it's proprietary or whether it's spread across 10 hedge funds that also wanted that information. How do you think that plays out? Does anyone have rights? Do you buy rights, nuances? **Christian Kleinerman** [27:04]: I do think that some of these statements about public web data owners changing licenses or potentially licensing the data gives one path forward. There's been interesting developments in the last week or so. Microsoft saying that they'll stand by customers from a copyright perspective. That was meaningful. **Harry Stebbings** [27:22]: For for people listening, why is that meaningful that Microsoft will stand by customers for copyright data? **Christian Kleinerman** [27:27]: Because enterprises are worried about if they incorporate Gen AI into any of their products or services in a way that they truly depend on it. And at some point, lawsuits start to fly everywhere. They're exposed and because of those concerns, it has held back enterprises. So Microsoft statement is material in alleviating those concerns that are very real. I talk to people every week and those are real concerns. **Harry Stebbings** [27:51]: Do you think they will set a precedent now, which for the next year or two, while there is ambiguity in confidence from enterprises, that actually these large providers must provide a backstop to their enterprise customers to allow them to onboard with confidence? **Christian Kleinerman** [28:07]: I think it alleviates a faction of customers and a category of concerns. I don't think it takes care of all of it. There are opinions out there on the does copyright law even apply to Gen AI? And it's a fascinating debate, but I think the standing by folks from a copyright perspective is a step forward. **Harry Stebbings** [28:26]: Are there any other big regulatory intricacies or challenges that you think not enough people are paying attention to? **Christian Kleinerman** [28:33]: The use case conversation is difficult. Much has been said, oh, don't worry about it. Most of the bad things that you can do with Gen AI are already regulated and illegal, so nothing new. But I do think that there are entire categories of work products that we need to go think through. What does that mean? **Harry Stebbings** [28:49]: You not think there's, like, an inherent challenge just in terms of the opacity of models? Like, when we think about the regulatory challenges, we can't continue to have such opacity to get to outcomes. Will there not need to be more transparency in models to show people the pathway to answers? **Christian Kleinerman** [29:06]: So a 100%. Two examples. One, again, from last week, IBM was announcing their own gen AI models and and they were talking about being fully transparent on the data that went into the models. That's a big step in lineage. The other one is is talking about how they trained it. They're in tune with this concern and they want to surface it in a more open way. The other example is at Snowflake we acquired this company, Niva, that was doing consumer search based or augmented by LLMs. And a lot of the principles that they had when they created this product was you want to have citations and quotations and attribution of even if it's a summary, every snippet, what was the source that it came from, which is a way to control for hallucination. So that is the world that we're headed towards, at least in the enterprise. Forget the creative side of things, but at least in the in the world where you need correct answers, you need to be able to attribute where things came from. **Harry Stebbings** [30:01]: I totally agree with you in terms of that attribution. I also was one of first investors with neither. So All good. That was great to hear. Can I ask you that, you know, obviously, Snowflake is an incumbent? You've mentioned IBM there last week. We've mentioned also OpenAI and Anthropic, which I don't know which company you put them in, startup or incumbent, probably incumbent now given funding amounts. How do you think about the question of where does value accrue in this next decade of AI? Is it the startups building or is it the incumbents who have the distribution? **Christian Kleinerman** [30:30]: I would bias towards incumbents that have the data. Think of all the data that a company like Google has. That's public data or not semi public, but a lot of user data. Or think all the private enterprise data that a company like Snowflake has, and and that's obviously self serving comment. But I do think that data is what powers outcomes. Those are going to be the companies that at least are best positioned. But to make things harder is I think all of the folks that have data are creating platforms for startups and others to come and leverage that data and deliver innovation. So it's complicated, but I would bias towards the incumbents. **Harry Stebbings** [31:06]: I have to say I do agree with you. I also don't think you can underestimate, especially in the short term, the power of distribution. And I think like the copilot strategy is a very incumbent focused strategy, which will win in the next two to three years for sure. But I'm a VC, so we pontificate on stuff we don't know about, Christian, but you'll you'll learn more about that as you know. Can I ask the other question that comes up is open versus closed? We mentioned opacity. We mentioned transparency. How do you think about whether closed system versus open systems will be more dominant in the next ten years as the predominant way to build the best models? **Christian Kleinerman** [31:40]: I think I think there is nuance on what open means. Usually refers to the weights of a model are available. But beyond that, there are questions on are there use case restrictions? For example, LLM two is very significant as a development in the industry, but they were very clear, thou shall not use LLM two for training other models. Thou shall not use LLM two for, I think it's 700,000,000 user use cases. So you can say, is that fully open or not? And and sure, because the the weights are available, you can say meet the the bright line definition, but there are other items that define what is more or less open. I do think that open weights creates a lot of opportunity for research and further innovation. Much has been said of how much excitement and interest is happening around the open model. So from that perspective, I think there's a lot of value of those models. But I also think that commercial solutions will largely be hosted cloud services, in which case, in the same way that it is true with open source, I think it will be true about open weights. I have no idea if it matters or not. At the end of the days, who has the best answers or the best service integrated for customers. **Harry Stebbings** [32:48]: One final one that I am thinking about a lot is just like the speed of adoption. Everyone seems very worried about, you know, impact on jobs, impact on economies that AI brings. Are you concerned by the speed of adoption? I guess, how should businesses prepare for the shock wave of AI? **Christian Kleinerman** [33:04]: All of this is harder than people realize. The demos are awesome. The productization takes longer time. So I would say all of us should go forward as fast as we can because there's gonna be natural difficulties that will just throttle us. I also, as I mentioned earlier, I don't think that it's a mass firings happening next week. It's more hopefully incremental productivity boosts happening over the next six, twelve, twenty four months. And then over time, you decide whether you take those productivity gains and you turn into fewer employees versus more productively deployed employees. **Harry Stebbings** [33:40]: You're a product OG, Christian, and you wouldn't say it, so I can. I think that with AI, you see the reducing value of UI. You will see personalization and customization according to each user, and actually you see this kind of chasm between device and user in a way that wasn't before. Do you agree that UI importance will reduce with the increasing prominence of AI? **Christian Kleinerman** [34:03]: I would say that for certain use cases, that's entirely true. If I have a UI to, I don't know, configure a cluster, like, I don't need to know all the options. I just can specify what I want. But I would say there are many use cases where you may want a richer way to interact with data or with whatever is the problem space that I don't think Gen AI would would dramatically change it. It does continue to help with personalization and we've been on the journey of personalization for a long time. So I would say, yes, some use cases, it shifts the the value of UI, but many others has the opportunity to continue to enrich them. **Harry Stebbings** [34:36]: What do you think Snowflake's biggest challenge is in terms of embracing and getting your arms around this next wave of iteration around AI? I know it's a continuation given the data first strategy and mindset, but if there was a challenge or an internal, hey, we're gonna solve this or how are we gonna get around this? What do you think that is? Probably perception. **Christian Kleinerman** [34:55]: I talk to folks on a regular basis and many of them still think of us as data warehousing. We expanded from those origins six years ago. We need to make sure that organizations across the world understand that they can do AI and Gen AI close to the data within Snowflake without having to copy the data to a different platform. Why don't they already? When you're very successful with some positioning, that comes and and bites you later that you are too successful with that positioning. And for many years, we said Snowflake is the data warehouse built for the cloud, and that still keeps getting repeated over and over. And it's a journey. If you think about most companies end up stuck with their original use case for a long time. We see a little bit that here. **Harry Stebbings** [35:36]: You know what's funny? We can edit out stuff, but it really fucks me off, Christian, because I started this show when I was, like, 18 and a kid, and that was, like, ten years ago. And people still think of me as, like, a kid podcaster despite managing, like, $600,000,000 and not being a kid anymore. **Christian Kleinerman** [35:51]: Yeah. It's exactly the point. Go go look at Salesforce will be CRM for a long time. Like, ServiceNow is a ticket company, so it sticks around very well. **Harry Stebbings** [36:00]: Cursor, one final one before we move into a quick fire. I've so enjoyed this. But when we think about your leadership style as the product leader that you are today, how do you think your style of product leadership has changed over the years? **Christian Kleinerman** [36:11]: I've been more willing to push opinions in a slightly more top down way as more time has gone by. Earlier on, it was I need to be a great manager and listen to everyone and accommodate everyone's opinion. And I'm not trying to say that's not important, but in some instances where you want a product to come out with a consistent view as if it came from a single unified set of principles and individuals, Sometimes you had to go and push for something like, hey, this is what we're doing. I think that confidence comes or evolves over time. **Harry Stebbings** [36:43]: I had Gustaf, who's the CPO of Spotify on the show, and he says, talk is cheap, so we should do more of it. How do you think about the balance between internal debate on product and product ideas, iterations versus just speed of execution and getting it done? **Christian Kleinerman** [36:58]: I think it depends on the nature of the technology or the nature of the product. At Snowflake, have both types of technology. The core subsystem that does a clustering of data on disk, I think you want to design that thing really, really well. Measure a 100 times and cut once because nobody wants their data to get corrupted or their results to be wrong if that thing is not built the right way. But if you want the UI for a query editor and you have 10 different ways on how you could do suggestions for customers, there's no right and wrong. Might as well go quickly, iterate, learn from from users. Both are the right tools and the right approaches depending on what you're trying to do. **Harry Stebbings** [37:37]: I wanna move into a quick fire round. So I say a short statement, you give me your immediate thoughts. Does that sound okay? Sounds good. Okay. So what do others not know that you know to be true? **Christian Kleinerman** [37:47]: I don't know if others don't know, but I for sure know that it always comes down to people. **Harry Stebbings** [37:51]: In terms **Christian Kleinerman** [37:51]: of hiring, in terms of customer and The the results, relationships, how things are going. Everything is just people. **Harry Stebbings** [37:59]: Can one succeed as a PM today without being deeply technical? **Christian Kleinerman** [38:03]: For the most part, no. There may be a few types of products that you might get by, but I I like deep technical PMs. **Harry Stebbings** [38:10]: What's your biggest piece of advice to a PM starting a new role today? **Christian Kleinerman** [38:14]: Learn the product that you're a PM of. Go be a user. Go as deep as you can know the technology. **Harry Stebbings** [38:21]: Do all founders need to be in The Valley who are innovating in AI? **Christian Kleinerman** [38:25]: Absolutely not. What makes you say that? Well, there is amazing talent throughout the world. You even though The Valley has something special from the community and the ability to bounce ideas of what one another, it's very clear right now there's a lot of innovation happening elsewhere. **Harry Stebbings** [38:40]: What's the best product decision you've made and how did you learn from it? Focusing Snowflake **Christian Kleinerman** [38:46]: and ease of use. **Harry Stebbings** [38:46]: What did you learn from that focus and ease of use? **Christian Kleinerman** [38:50]: It is something a little bit counterintuitive that you may put out a product faster if you just say, I don't know if we should be used this way or that way, so you just surface choices to users. And sometimes it takes longer for us to take the automatic choice and simplify it for customers. So it may be counterintuitive that faster is not necessarily better if simpler is what is being traded off. **Harry Stebbings** [39:13]: What's the single biggest element that you have most like to change about the AI community? **Christian Kleinerman** [39:18]: I will double down on they need **Harry Stebbings** [39:19]: to know that Snowflake is a great platform for Listen. Always be selling, baby. Tell me, Quentin Clark said, were you right about Satya in the beginning? **Christian Kleinerman** [39:27]: I was super wrong. When Satya came into the enterprise business, this is before he was CEO, he came in in very short order and made a lot of really difficult decisions, how the org was structured, how contractors were hired, how we thought about the cloud versus the on premises products. And at the time, my thinking was like, I don't think that he understands all of this. And obviously with the benefit of hindsight, or actually shortly after, it was very clear. He understood it better than all of us, and he's proven to be a brilliant leader. **Harry Stebbings** [39:55]: What do you think makes Satya such a brilliant leader? **Christian Kleinerman** [39:58]: He is very clear what the needed outcome or desired outcome is. And so he's a clear thinker would be the attribute. And then he can relentlessly drive towards it and not get encumbered by all the 100 reasons that we all make up for ourselves. I love **Harry Stebbings** [40:15]: Frank Slootman. Okay? I love him because he's no bullshit. He says how it is in a world where no leader says how it is. What have been your biggest lessons from working with Frank? **Christian Kleinerman** [40:25]: He is also such a clear thinker. He has that commonality with with Satya. He becomes a clarifying force. Oftentimes, if you're just picturing yourself telling Frank about a problem and a couple of options, just in the formulation it becomes very obvious that you don't even need to get his opinion because you know where he's gonna stand. So that type of clarity that he has simplifies decision making, accelerates decision making. He's also an amazing, amazing leader. **Harry Stebbings** [40:51]: Final one for you, my friend. Now it's ten years. What role does AI play in society then? **Christian Kleinerman** [40:56]: Productivity boost on pretty much everything we do. Everything is gonna be simpler, easier, faster. What impact does that have on **Harry Stebbings** [41:04]: GDP? Is it like 2%? Is it like 10%? I don't know the magnitude, but for sure, net positive. Should we have a bet? What do you wanna bet on? Would you do 10 or two? Two. Yeah. I would too. Fuck it. Well, listen. I'll buy you dinner in London next time you're here to thank you for this anyway. Sounds wonderful. Christian, this has been fantastic. You are a star. Thank you so much for joining me today. Harry, thank you for **Christian Kleinerman** [41:23]: having me. It's been fun chatting with you. **Harry Stebbings** [41:27]: I love it when we can really go deep on a particular topic like we did there. If you wanna see more like that, you can head over to YouTube and search for 20 VC where you can find all of our interviews in video form. I always love to see you there. But before we leave you today, ## Sponsor read **Harry Stebbings** [41:41]: you know all those mind numbing tedious tasks that seemingly take up half your day? Well, Coda is here with their new AI powered work assistant that helps you and your team not just finish tasks, but make progress So your product team can bring a feature to market faster by using Coda AI to tag customer feedback, draft PRDs, suggest target audiences, summarizing product discussions, and more. Your sales team can engage more customers by bringing in data from sources like Salesforce and then use Coda AI to suggest action items, meeting agendas, or lead scores. And your marketing team can drive an impactful launch with Coda AI summarizing user insights, creating briefs generated from notes, and writing more press releases to visualize taglines. With Coda AI, you can reimagine your to do list and how you collaborate, so you're not only finishing tasks, but really making progress. If you want to work a system that lets you get back to work, you can get started with Coda AI today for free. Head over to coda.io/20vc. That's coda.io and get started for free. And speaking of amazing tools we cannot live without, travel and expense are never associated cost savings, but now you can reduce costs up to 30% and actually reward your employees. How do you do this? Well, Nabla rewards your employees with personal travel credit every time they save their company money when booking business travel under company policy. Policy. Does that sound too good to be true? Novan is so confident you'll move to their game changing all in one travel corporate card and a spend super app that they'll give you a $250 in personal travel credit just for taking a quick demo. Check them out now at navan.com/20vc. And last but by no means least, we need to talk cash. As of the June 29, you can get 5.5% yield on your cash with twenty six week treasury bills. But buying treasury bills, it's not that easy and you have to navigate a website that looks like it was made before I was born. Enter public.com. Their treasury accounts make it simple to earn a high yield on your cash and it takes twenty seconds. Here's how it works. Sign up at public.com, easily purchase twenty six week treasury bills that automatically roll over at maturity for a compounding yield, plus there are no minimum hold periods. You can access your cash at any time with the flexibility of a bank account, of course, to receive the full guaranteed yield, of course, to receive the full guaranteed yield, you do have to hold to maturity. But here's the thing, these are t bills, means your investment has the complete backing of the US government, making one of the safest places to park your cash. Go to public.com/20vc to lock in a historic 5.4% yield on your cash. As always, I so appreciate all your support and stay tuned for an incredible episode coming on Monday with the one and only, Benioff.