0:04Thanks for coming on. Eric Burn Hartson from Modal. Could you explain uh the business a little bit and maybe how you got here?
0:11Yeah, so we uh we build uh the easiest way to put it is we build AI infrastructure. So we we offer a bunch of compute a comput platform and a storage platform that enables a lot of customers like Decagon and Sunno and Runway and Cognition and many others to to essentially build AI applications. So what that means in practice is a lot of inference, a lot of sandboxes, sometimes training and fine-tuning, uh sometimes batch jobs and and other things. So we have basically built a um a software SDK
0:40that enables customers to move really fast and and build this uh we manage all the capacity underneath make it easy to deploy and and scale and and run these things in production.
0:49So you were on late in space like 2 years ago. A lot has happened since then. Like how big were you back then?
0:55Like whatever numbers you can share and comparatively Yeah. How big are you now?
0:59I actually don't remember where we were, but it's like it's probably like at least like 50x in terms of like revenue and users and usage and stuff like that. Yeah, I use it actually for like a side project that I have where uh Nice.
1:11Yeah. Yeah, it's like uh yeah, turn like 2D images into 3D models. So, yeah, just hosting like models there.
1:17What do you think? Any feedback?
1:21Um I mean, yeah, that's why I use it over like any competitors or anything.
1:24Good. So before you started modal, you were at Spotify and also like better.com I think. That's right. Yeah.
1:30Um yeah, what were you doing at Spotify?
1:33I actually did a lot of different things at Spotify, but my main contribution was I built a music recommendation system.
1:38The first version I I joined after school in 2008, 2009 maybe. uh and spent seven years at Spotify and and most of that time was building basically from scratch a music recommendation system and built up a small team and we launched a bunch of features like related artists and discover weekly and and a bunch of other stuff. So a lot of a lot of matrix factoriization a lot of compute intensive stuff and and and arguably like a lot of what sort of later became the genesis of modal was um you know just just building AI
2:07applications and realizing like the infrastructure sucks like there's got to be a better way to do that. So a a lot of sort of, you know, the rationale for for modal was like kind of building a tool I always wanted to have throughout my years at Spotify.
2:20Yeah. So modal was more of an offshoot of the stuff you learned at Spotify, not much from better.com.
2:27Better B better.com was a little bit of a side tour. It was kind of a roller coaster. Uh long story about the company, talk about it separately, but I was a CTO for for 5 years of Better and grew the tech team from I think, you know, two to three 400. So uh learned a lot about management and building teams and hiring and also learned a lot about developer productivity. Thought a lot about tools and things like that. Did a lot more like full stack stuff but yeah kind of came back to my what's close to my heart which is infrastructure and AI and machine learning after that.
2:56What was better.com doing again?
2:59How did you get into that? I mean we don't have to spend too much time on that. Just Yeah. No, it's fine. like this guy Zack Frankle uh who I didn't know at that time ended up recruiting me to better as the CTO and the company was 10 people and you know for whatever reason you know the the founder Vishal very interesting guy very very smart a little crazy in in mostly positive ways um we got very well along so we uh yeah we we started building it in in 2015 and I spent 5 years growing that company and
3:28then you know eventually turned out that doing mortgages is pretty hard And you know interest rates went up very quickly after the pandemic and then that that caused a lot of issues at the company but by then I had already left.
3:39That was like 2020 2021 when you left.
3:42Yeah something like that. Yeah.
3:43So okay you left before you had the idea for modal or you left and then you started it's hard to say. I I had a lot of random ideas when I left better and and and I I I actually wasn't sure if I wanted to start a company. I was kind of just, you know, during the pandemic, I I sort of almost took like a sbatical. I had two kids. I I mean I still have two kids but I I mean at that time there were there were I think two and four and so I decided to hang out and took some time off and was kind of working like
4:13part-time like hacking on certain things and wasn't sure maybe I should do some consulting and I started building stuff and I just got like super pumped about this idea that that eventually became modal but I actually built you know modal was like I think the third thing I built that I got you know that so so there were like a few attempts before then that that went nowhere but but modal was like the first idea was like this is kind of cool like I should do this. This is like this could be a really cool platform.
4:37Were the other ideas like also in a similar vein like developer tools?
4:41The first thing was like some sort of web scraping thing which went nowhere but actually you know in hindsight maybe could have gotten somewhere.
4:47And was this before the chat GBT launch?
4:51So Chad GBT was late 2022. Right.
4:53Then I I started building so maybe actually to rewind a little bit context like when I worked at Spotify I released an open source framework called Luigi which is like workflow orchestration.
5:02This is sort of the same category as airflow and and prefect and daxter. Sort of dated category to some extent like people don't talk about as much but for a while it was like you know got a lot of attention. Uh anyway I I never turned it into a business. Uh but a couple years later when in 2021 when I was thinking about business ideas or or things to work on I kind of came back to workflow orchestration and I started hacking on a bunch of ideas of you know running code in the you know like basically orchestrating code running in the cloud. And what I realized was like
5:30you know at the end of the day like core puzzle piece is not just like orchestrating the compute it's like executing the compute and I realiz like that's kind of a bigger problem actually to solve. So in a way I actually like you know left it on the shelf and then said like actually I'm going to like go to this other thing over here and I'll go back I'll get back to the workflow scheduling in a little bit but but like you know I never got back to it. Long story short and and and ended up building modal which is like the focus was more like let's just figure out a way to make it easy to run code in the comput in the cloud and and the focus
6:00was initially like fairly general purpose and actually in a way like it's always we've kind of gone full circle more recently but but we we focused on so many different applications it was kind of the idea was like let's just make it super easy to run code in the in the cloud could be GPU code AI machine learning could also be like data stuff um and uh focus a lot on you know how do we scale it uh how do we make the developer experience awesome? realized super early that like in order to do that we'd have to go very deep in the stack and build on file system built on
6:28container runtime uh and and did a lot of you know solved a lot of hard engineering challenges and then you mentioned chatbt that was actually not quite the killer app but like if you remember like that fall another thing came out around the same time which was stable diffusion and stable diffusion turned out to be a killer app for our for what we're building at that time we were only four or five people at that time so uh we started seeing a lot of people coming to us and wanted to run a stable fusion model and then we real that that that made us realize like
6:58maybe geni inference is actually a really good use case for us and how were they finding her I'm this like compulsive tweeter I was even more like now like I'm too distracted by building a company but I always like tweeted a lot I always blogged a lot I think I had a little bit of a following and so you know in the early days I like a lot of what drove the growth is just like me just compulsively just like tweeting about modal and like what I was building and getting people excited and writing long blog posts about where the cloud is going and all these things. And I call it a bit like founder le
7:27marketing. I always like feel like you kind of have to lean into like what you're good at and like what I'm good at is like I guess like just posting and being terminally online and so you know like work it it's it's still like you know I think a lot of our growth is we're just kind of good at social media and just like being you know online a lot.
7:43Nice. How long until you had your first customer?
7:48Also did you raise like early on?
7:50We didn't race for a while. I actually self-funded it for a while. Uh even with up until the first few employees uh which in a way was like kind of nice cuz it made me really like frugal about money and like wanting to see like ROI.
8:02I wasn't taking out a salary for a while for a long time actually. At some point it became a problem because I needed the money but we didn't race for a while and um uh until early 2022 raised a seed round. So at that point I'd already been working on it for about a year. uh Asha joined as the CTO and co-founder in in about about five years ago about now August maybe 2021 but then it took until like basically like late 2022 for us to see any meaningful uh revenue I guess that's when Chad GBT came out
8:32yeah and and stable fusion then in our case uh was the main thing that's sort so so we went from like you know $100 a month in like I think like November 2022 and the next month we did like $2,000 a month and then like January 2023 we did like 30,000 all of a sudden and you know so so finally I felt this like massive just like anxiety thing off my chest.
8:56I'm like maybe this is actually like we're on to something. It feels like we're on to something. And then 2023 we grew very fast and I guess we've been growing very fast since then. Basically that 30k a month moment was that just like a couple big customers or was it like very like diverse? It's just one and I don't I don't know if I ever talked about like who that was and if they even know it but I think they suspect it was a company in New York called Mage. They're doing like a stable diffusion in a company. So at some point you know and in the first I think first
9:24quarter of of 2023 I think they were like 80 90% of our revenue but but then after that you know it started diversifying a little bit and late 2023 we we got Sunno as a big customer. They were a big chunk of our revenue for a while too. But then, you know, it's over time like we actually have really good revenue diversification now.
9:42Yeah. For the stable diffusion stuff, I remember using it for like Comfy UI. I'm assuming a lot of people were using Comfy UI, like running Comfy UI workflows on it.
9:52Some people were, but but I I think the people that were using Modal, they tended to be more like low-level stuff, like they did like complicated stuff.
9:58What they like with modal is like it's always been pretty hackable. And so they they deployed their own custom workflows with adapters and custom inference and and a lot of optimizations and and all the all the stuff we had done in order to get the developer experience to be really good turned out to be really good for also running production workloads.
10:15Uh cuz autoscaling turns out to be actually a very similar problem like you want to scale up containers up and down very fast. Uh so we just happen to be like very well positioned to to capture a lot of those workloads in the early days. Did tools for autoscaling like not exist? Like why did you have to build those yourself?
10:30In general like my thesis is like you know every new sort of era of like you know technology kind of runs up against limitations of of tools uh you know whether that's you look back you know like the web or the cloud or you know mobile or PCs or or you know AI or whatever. So, so I I think you know it's often the case that tools exist but but they're often not like particularly good and and and often you you kind of need to rethink a lot of those tools in order to reap the benefits of you know and and
10:59really you know turbocharge and start building for this new world and and you saw this with cloud you know we had to invent things like Kubernetes and Docker and things like that and and in a in a similar way you know like with with Gen AI you could build these things on the sort of prior generation of tools like you can you can build these things on top of Kubernetes and Docker and in these things but but but those technologies are built for a very different type of workload. They're built for a very sort of steady work steady backend workloads. they have an even you know steady number of requests
11:27like running at pretty low utilization cuz CPUs are normally pretty cheap and so people don't really worry so much about cost and but now you're entering this new world and you have you know very expensive GPUs you want to manage capacity and and utilization you have very large models that you need to load over network into CPU into GPU and and and it turns out like a lot of these tools like you know like they were just not built for that so so we felt that you know we we had to rebuild a lot of that you building our own I mentioned
11:55the file system but we also spent a lot of time over the last few years like building you know some really cool stuff around like CPU snapshotting GPU snapshotting and auto we built our own autoscaler anduler and and and and all kinds of capacity management tools and we kind of feel like you have to do that because like you can't build you know for this new generation of of software on top of you can't build like a wrapper on top of the old stuff in order to to do the new stuff you kind of need to start over a little bit from scratch so I guess like the auto scaling doing
12:24stuff like did you did you see any like um weird behaviors on the platform like was there anything that like broke while you guys were scaling?
12:32I mean capacity is always from time to time been been a challenge like how do you get like you know hundreds and thousands and tens of thousands of GPUs.
12:39So that's that's always challenging. We had to go multi-reion very early. I think that was like kind of a fateful decision in hindsight like doing that and kind of just biting the bullet and building it like we always thought of like our our capacity management as like one big pool like it's like one big pool over the the entire world like we don't think of it as like separate clusters running in each region and so that that was like one of the like very consequential decisions we made earlier to kind of think about it that way. So I don't know if there's like been like you know specific issues. I mean, we've obviously had bugs and, you know,
13:08outages and I hate, you know, as much as I, you know, would love for modal to be 100% uptime, you know, and I think recently we've basically hit four or five nights, but throughout the history of modal, of course, we've had, you know, run up against scaling issues and things like that.
13:21The GPU market and well, actually before that, so you guys use AWS like GCP like all these cloud providers underneath like do you also buy any infrastructure or no? Let's talk a bit about like cloud capacity because I think it's actually changed a lot like in in the first few years we did we got almost all the capacity pretty much from from the hyperscalers spot and on demand capacity mostly spot capacity running in all the different regions you know using uh initially AWS but also GCP and and
13:51overtime Azure uh we we started renting some capacity from Oracle pretty early because we wanted um 810s and 8100s and we got really good rates on those uh and and over and we added a couple more clouds like doing reservations but but like still up until like actually end of last year like most of our capacity was either spot or or one-year reservations.
14:13Spot means you're just like renting it right there and then Exactly. Yeah. And it's also preemptable which we had to deal with. Um but but it sort of works in aggregate because you can sort of just you know aggregate capacity from many you know hundreds of regions basically. So that that that model worked really well until we started running up again. I mean first of all like modal started growing like very fast and so we needed more capacity but fundamentally also like the cloud market started changing a lot. Um so in the last year basically now in order to get the capacity we need we had to work
14:43with a much larger set of clouds. We actually work with almost 25 clouds at this point and in many cases um in order to get the capacity we want like we we have to go and do you know three plus year deals and you have to pay a meaningful amount of money up front. Uh and and you're locking in pricing on on a three-year term right. So you know like and and and so we had to spend the entire you know last 6 to9 months like you know kind of rethinking how we get capacity we had to build a whole procurement like desk like we have like
15:11four guys working full-time basically all they're doing is like procuring capacity calling neoclouds you know doing deals vetting the capacity we've had to build a whole engineering team basically 10 people just like managing like bare metal capacity from all over the world and dealing with network issues and bad drives and GPUs falling off the us and and all these things. We don't own capacity.
15:34We rent all the capacity so far. Um you know on on typically the three-year deals and that's like a big decision like should we buy should should we know you know or should we keep renting. I I'm actually like not like mega dogmatic about it. I I think renting is nice because there's a certain amount of capital efficiency to it. if you're buying, you know, like I mean, first of all, there's an obvious like operational complexity of it, like you have to, you know, deal with vendors and suppliers and and, you know, power and and and
16:02cables and and all kinds of stuff. But there's also, you know, financial complexity to it, like how do you fund it? Like, should you raise equity, you know, and then you can raise a billion dollars and and go buy hardware for it.
16:12That's a very diluted way to do it. um you can raise debt, but you know, if you raise debt, you know, you typically want to have like locked in long-term revenue to to to pay the interest rates. So, that's also not trivial, but you know, like we look at the economics, we look at the the the operational complexity, we look at the risks. Uh it's a complex trade-off and and we're open to to to buying. We're sort of there's other constructs too like we're it's to get the capacity we need for for 2027. It's
16:42going to take a lot of work on on you know going a lot of different avenues to to to lock it up.
16:47So do you have to hire a lot of like finance people to do all this stuff or not really? you know, I've had to spend a lot of time on this and we we've had to to to spend a lot of time on this and I I do think over time, I mean, there there is a lot of, you know, these are very expensive assets, uh, the hardware and, you know, I think I mean, it's kind of staggering. You're starting to see like AI investments starting to become like several percentage points of of of GDP of the US and, you know, starting to see like interest rates in in America
17:15shooting up in the last few weeks. And I mean like at this stage like you know and you go and look at the neoclouds like these are very very sophisticated financial engineering companies and so yeah we we look at that and you know we look at all the options and and it's possible we might have to do similar things ourselves. I wouldn't want to like I would rather focus on being you know like a very lean software company like doing the things we're good at. I don't think it's good in general for
17:42software companies to own hard assets. you know, in the same way as like should you own your office? Like I I don't think that's necessarily the right solution for for most companies. But of course, like in order to get the capacity you want, like you know, everything's on the table.
17:57Yeah. And I guess the supply is going to be constrained for a while. So you kind of have to do that. Like do do you agree with that or do you think this is just like a moment in time?
18:06I don't necessarily agree with that. I mean like Nvidia is delivering, you know, several hundred thousand black walls, you know, a month, right? And and I I I don't want to say like the market is like 100% efficient, but like you know in general like if you're willing to pay a high price, uh you can get GPU capacity, right? Like if you're willing to pay more than everyone else and and of course Enthropic and Open AAI is buying up as much as they can and but fundamentally like there's constantly supply coming on the market and and so I I don't necessarily think it's like a fully
18:36cornered resource. prices are going up and and I do think it's worth kind of reflecting on that is like the normal state of hardware is like it's like kind of like gravity like eventually hardware prices always go down so we're a little bit in like a weird sort of period of time where like hardware prices are going up and you know it's hard to say you know how long this is going to last you know could be another 6 months could be another 36 months like it's really hard to to know h how long and but but I I fundamentally don't know if like it's
19:04it's true to say like you know you need to buy capacity in to to me that's a little bit to sort of almost like the great toilet paper shortage of you know the pandemic or whatever right like you know at least this like you know run on the bank type situations and but that being said like you know capacity is going to be tight for a while and and you know prices are going up and you know luckily I think you know it's not even necessarily bad for us if prices go up cuz cuz you know scarcity also means like we can charge more from our customers and and you know and the fact
19:33that we have a lot of reservations locked in at a rate means like to some extent we're effectively like long GPU prices already. Um so um I I don't necessarily worry too much about prices going up. I do worry more about actually prices collapsing. I don't think that's going to happen anytime soon. Uh but I think that would be a quite bad thing for a lot of the players in the market.
19:54Why would that be bad?
19:55I think it would be bad for if for anyone who's locked in a lot of supply.
20:00Yeah. Because they paid way too much and they paid too much, you know. So there's a risk of buying at the top, right? And and again, like I don't worry about that that much. Like I don't I don't think it's going to be like, you know, it's going to happen anytime soon, but like you know, like I said, like hardware prices in the very long run, they tend to go down. And so it's it's worth, you know, reflecting on that, right? Like right now, I think I think your your capacity is constrained. And and that leads to sort of, you know, in economic terms, there's a scarcity rent, right?
20:28Like who captured the scarcity rent? I think that primarily goes to two places mostly Nvidia but of course but also to some extent neoclouds like neoclouds and in general like you know anyone owning capacity has better economics like that's undeniably true today with capacity being constrained long term you know in the very long run I don't think that's necessarily entirely true right like there's always going to be a little bit of a premium you can get from owning GPUs but is it worth the risk you know
20:57is it worth the risk like holding this like very expensive assets on your balance sheet. I'm not so sure, but you know, we're going to see like it's going to take a few years to get through this and and and in the long run, you know, we might own some, we might rent some, maybe a combination, maybe do some other crazy stuff, joint ventures, who knows, maybe some complex SPDs or whatever. Um, everything is on the table.
21:19How do you compete with OpenAI and Enthropic and buying GPU capacity? I'm assuming they'd want to like buy up literally everything that's available.
21:26They are. and and and I think what you're seeing is um you know a lot of the the the prices going up is a reflection of that. That being said you know Nvidia is shipping a lot of stuff.
21:36I think also like to some extent like Nvidia is like kind of prefer a little bit of a diversification like they don't want like you know small amount of large buyers in much in the same way as you see you know hyperscalers getting a little bit of a constrained allocation.
21:50Uh I I think you know also Nvidia prefers to see you know a lot of neo clouds and a lot of alternative clouds like getting more allocation and I think a little bit what we see in the market is also this is more speculation but but I think we see you know smaller clouds that are a little bit more sort of risky to work with being more of an opportunity for us like where you know maybe Nvidia sorry anthropic and openi my feeling is like they prefer to buy
22:18from like the the tried and True. Uh part of it is also because they had to do a lot of training runs. So they need to have like very large blocks, but there's plenty of capacity out there.
22:27You know, you can get 32 nodes in Armenia, you can get 128 nodes in Indonesia. Like, you know, like we love that kind of Like, you know, so so we can go out there and like get a lot of, you know, smaller capacity all over the world. Pretty good prices.
22:41Is it pretty much just like if someone offers you like a certain number of GPUs, like that's all you care about? Is there nothing like, oh, you care about like how reliable it is, you know? Is it just some like janky warehouse that someone has these in?
22:53I mean, sometimes it's kind of a janky warehouse. It feels like, you know, but but like we do have like a spec sheet basically. Like when we talk to vendors, the first thing we do is we send them a spec sheet. We say we need this number of IOPS. We need, you know, like the CPU needs to be at least at least, you know, have have this many, you know, whatever the pass mark scores or whatever it is.
23:12we need to see like you know minimum you know north north south bandwidth uh egress and and ingress so we do have a list of requirements often we try to do like an acceptance testing period where we get like one node and and run some tests over time you know like to some extent it also involves a little bit of a leap of faith you know sometimes we sign very big commitments you know tens or hundreds of millions of dollars and we start working with the vendor that we still don't fully know and you know over
23:40time like you know we'll We'll see, you know, and and I think over like in a little bit of like there's a little bit of sort of just, you know, our own sort of like, you know, how like uh semi analysis has like a cluster max. I kind of feel like we almost like have our own like modal max score internally. Like I know which vendors I like more than others, you know, like and and and we sort of prefer to do additional deals with the vendors that we've seen, you know, very good reliability. Not just that also like it's not just reliability, it's also are they very
24:09responsive when when we have you know support issues. Do they fix the issues we ask for the turnaround time? If things go down, you know, how quickly do they fix it? Like things like that.
24:18Do you agree that the demand for intelligence is basically infinite? And if so, like why wouldn't you just buy up as much as you as much capacity as possible and then like if you over buy, you can just like sell it to somebody else. I think it's very easy to say when there's like a scarce resource like let's just buy as much as you can but I think you know first of all like if we bought it today we couldn't utilize all of it like we could use utilize a lot of it right but like but you can sell it off to somebody else who wants to use it right I think the challenge is like if there
24:48was like something in the market that happens there are a lot of correlations that happen at the same time if you know prices going down and utilization going down and the market for reselling capacity goes vanishes at the kind of the same time I'm sitting here and like I'm I'm extremely aggressive like buying capacity every day and like raising money to buy you know so overall I'm I'm very bullish but like I do want to point out and like you know like there are tail risks here and and when those tail
25:16risks happen like it could be get very bad part of it is like maybe I'm you know I'm just old and I've seen this in the past like I actually had my first job in 1999 so I saw the dotcom crisis and I've seen the great financial crisis and things like that and you know like I want to build a company that that is around for like 20 to 30 50 more years, right? Like and and so, you know, I think we can certainly go out and like buy an insane amount of capacity right now and trying to just corner the market, but you know, do you know about the Hunt brothers?
25:45It's like a legendary story in trading.
25:47They try to corner the silver market.
25:49It fails spectacularly. Um you know, so so cornering the market I think in general like can be um not not a good idea. Um and but but we are in general very very aggressive but but we're ramping up you know like at a at an aggressive plan that basically matches our demand growth and matches where we want to try to get to at the end of next year in terms of run runway or run rate.
26:13Are you only buying up Nvidia stuff or also like you know maybe a capacity from other types of companies like we're we're looking at it. I I think the alternative accelerators are starting to get very interesting. It's, you know, ironically, like every every year in the last five, maybe even 10 years, I've been saying, you know, like next year it's going to be the year of like non- Nvidia accelerators and and then I'm always wrong, but but I do think 2027 might be the year of non- Nvidia accelerators finally. So, we'll see if
26:41I'm wrong this year, too. But, so I I you know, AMDs are starting to get like really good scores on certain things. I think, you know, TPUs are are obviously used very, you know, widely internally at Google seem to work really well. I I don't know much about Tranium. I don't feel like I see quite as but you know the team is very good. I know they're working on a lot of the software stuff.
27:00You know hardware is hard in a in a certain way like software is like easier and that makes me feel like you know if we just fix the sort of software layer for these accelerators like I think there's going to be a lot of demand for them and and I think fundamentally you know with Nvidia being such a bottleneck like we'll probably see you know more alternative accelerators. That being said you know there's bottlenecks further down the pipe. I mean, you know, there's like TSMC and ASL maybe like a deeper bottleneck to all of this anyway.
27:24Uh, but, you know, long-term, I think monopolies don't always last. And so, I I'm I think it'd be healthy for the market to have more alternative. As much as I love Nvidia, like to make that clear, like I mean, they're obviously an amazing company and with an amazing product. Um, I I do expect to see a little bit more divers diversity in in the market as well also things, you know, like Cerebras and Grock and these things.
27:46Yeah. What are your thoughts on like some of those uh providers?
27:49I think they're pretty exciting and you know and and you see them doing really well on on certain workloads and you know there's some interesting you know people doing these like hybrid you know disagregated inference where they run you know NVIDIA for the prefill and and cerebras for the decode and and things like that.
28:03I've never heard of that actually.
28:04Yeah. And you know Grock too like because a lot of these things you know certain things are computebound certain things are memory bound so you can sort of mix and match uh and I think that's quite exciting. it's it starts to get very complex when you're deploying these things in practice. Uh but but I I I do think there's a lot of there's a lot of you know opportunity and and and in these sort of you know more complex hardware setups.
28:25Are there any learnings that you get from uh some of these Neoclouds like I don't know like Nebus or whatever and like the way that they're doing things versus like what you're doing.
28:34I think one of the things that you know is very interesting with with Neoclouds is like how can they expand so quickly.
28:40I I think actually the most crazy example is like ncale which went from like nowhere to like you know having like you know whatever hundreds of billions of dollars in in their pipeline despite barely having any capacity. How can neoclass grow so fast? I mean there's like an you know and and and basically the answer is like you know and this might be obvious but like what they do is like they go out to like you know anthropic or or or opening eye some some big buyer or whatever and basically say like hey do you want to buy like 5 years of chips you know sign this
29:10contract we give you a good rate and then they take that contract and then they go to you know Wall Street and say look we got this like you know big buyer you know they they committed to $10 billion worth of chips uh we just need the money right now and and they're going to raise the money and then they like you know go to like super micro and dial or whatever in Nvidia and like get the hardware and then they install it and then you know now they have like you know deliver it to the customer right so so you have this like very nice sort of financials construct where you can basically sort of pre-ell things that
29:38you haven't even bought and and that means you know you can scale up extremely fast and so that's what we've seen with these neoclouds is is is they can scale very very well I think modal because we've always had a little bit more of a PLG focus and and and and not the sort of you know ability to lock in long-term contracts in the same way. It it's a little bit more challenging from a financial perspective to scale that fast. Uh even though I think you know in the long run like we'll get to the same point but I think that's been kind of a fascinating thing to see with these neo
30:06clouds and especially like core and and those big ones um been very very savvy with you know how they raise money and and how to you know think about GPU as collateral for loans and all these things. What's the relevance of like I I forget who it was but like some like financial company was like oh we're going to make oh I think it was Nvidia actually they were like we are going to turn GPUs into an asset class. Is that just cuz like they need to raise money and they were like trying to figure out where to get the money from like what's
30:36the relevance of that?
30:37I mean there's a bunch of startups I don't know if you're thinking about that. There's like Silicon Data and Compute Exchange and like or and and and a few other ones that are like basically like trying to get to like you know creating price indices and indexes and and and trading derivatives and things like that. I think it's interesting like I I think with compute like it does there are a number of challenges with compute as an asset class like it's definitely not as homogeneous as other asset classes. uh you have different network and different locations and and
31:06and different discs and and CPUs and all kinds of stuff and reliability. I mean that being said like you can say the same thing about oil and like we kind of created like you know relatively commoditized you know tradable assets out of oil. It remains to be seen like I do think you know at this scale where we are with the sort of AI like data center GPU deployment like we do need to be a little bit smarter with like risk hedging and and and and how we share risks and and you know having more like sort of insurance and and things like
31:35that. So I I I I'm I'm very happy that people are thinking a lot about this. That being said, like I I'm I'm not entirely sure if we're going to see, you know, GPUs becoming sort of, you know, whatever the sort of Henry Hub like, you know, Brent crude type like, you know, super commoditized like tradable asset.
31:54Yeah. I want to shift the conversation to modal sandboxes.
31:59I guess like before we do that, what are the different products that modal has?
32:03It's I guess like the main product obviously is like the the GPUs, then you have modal sandboxes. Is there anything else?
32:10I think of the like the two big products that we have is basically number one is modal functions and number two is modal sandboxes. So, so with modal functions we we launched that you know back in 2021. It was the idea of like serverless functions that are be very easy to deploy and very easy to scale and we manage all the capacity that turned out to be really really good for inference and it turned out to be really good for managing GPUs and and running on things like that. But we actually never had like you know dedicated like inference product or like a GPU product per se like we we we never thought of model as
32:40like a neo cloud and and I think there's like a risk of like getting sucked in that direction becoming a neocloud like to selling you know sort of what I think was like rectangular like here's like a three-year you get this many GPUs always been very good at like you deploy something you can kind of scale it up you have this like very cyclical or like bursty stuff like we manage all the capacity and the beautiful thing is like because we have many many thousands of customers we can sort of aggregate that capacity and very smooth workloads you know very smooth aggregate demand which we can manage very very efficiently soal
33:09functions was the the core we launched that in 2021 like I mentioned we started seeing product market fit in 2023 and now you know it's about twothirds of our revenue and and and empowering a lot of you know in practice it's mostly inference but but you can use it for all kinds of other stuff mobile sandboxes we launched that in 2023 and it's actually kind of you know took a while to see product market fit similar to functions why did we launch launch it. We realized a lot of people we talked to were using LMS to output code. This is like me and
33:38Asha talking over lunch one day. We're like maybe they need a place to run that code and and and we realized like actually we kind of have like a lot of the infrastructure already to just take code and run it because we already built the primitives for like managing containers and isolation. We already had you know G visor as an isolation primitive and and and things like that.
33:56So we got kind of excited about that idea and and and Nokia wrote a a prototype and launched it in basically one day or two days or something like that. Zero like we saw like very little like growth. I don't want to like trivialize it. We did see some growth like there were like some sort of you know it was 2023 so yeah it was 2023 and like you know like some people picked it up like we had some early users who kind of enjoyed it for for certain things but but like overall like it was like a pretty insignificant source of revenue into
34:26going into the summer of 2023. So two years later we even discussed like do we even need this product like should we shut it down then it just started exploding. So 2023 and I think what what really happened 2023 was like LMS got really good at generating code in 2023. 2023.
34:43I mean, it was like around that time.
34:44Like I I feel like that was like when they went from like being able to like refactor functions and like write snippets. So like generating like full apps, right? And and and so we started seeing a lot of growth coming from like vibe coding, you know, companies like lovable and and other ones uh and and then you know companies like ramp started using models for background agents and then we're seeing started seeing tremendous growth coming from a number of neolabs doing things like reinforcement learning on model sandboxes. So sandboxes is now about a
35:13third of our revenue. You know, we we considered like basically like our our second product. And and to me it's like really exciting because like even back in the early days of modal, I always knew that modal, you know, is about like creating a whole toolkit of products.
35:26Like we always wanted to be almost like a cloud. You know, you have the storage and the compute and a bunch of different ways to work with it. We always wanted to be a multi-product vendor, but that wasn't necessarily true in practice until like now where like I think now it's like kind of clear we have functions, we have sandboxes, we're working on a bunch of other stuff too.
35:42Working on training, working on batch, we're working on a bunch of other stuff. So so to me that sort of evolution into becoming like a multi-product company I think has been amazing because that was that was always so core to the vision even in the very early decks. I I had a bunch of slides on this uh but but it wasn't necessarily true to the external world. uh until recently.
36:02Do you have anything in the works that you know you think is going to be like that next product? Any predictions maybe about the market, anything like that?
36:10I mean, I'm I'm very bullish on training. I think training is like kind of an underserved market. Obviously, there's like a lot of focus on like very large, you know, training runs. Uh but we actually see a lot of demand the sort of smaller and midsize training segment. So, I'll I'll be talking a little bit about that later this week.
36:26We're organizing a conference. batch is I think like a really interesting market. I I think we see a lot of demand for sort of you know customers that need a you know we had a customer the other day that needed to transcribe you know 5,000 years of audio. They had like you know millions and millions of audio files and and we we had a lot of capacity so we let them do that you know it took a few days. I I think there's some really interesting use cases there.
36:48Very bullish on like low latency stuff.
36:50I I think there's going to be a lot of you know once we sort of get the audio like the speechtoech models working like the voice models I sort of expect that to be very interesting market then of course like all this like agentic stuff right in a way like I I have like mixed feelings about like the word agentic cuz I feel like in in certain ways like I think all software will become agentic.
37:12So it's almost like to me like what does it mean to be agentic? Like I think I think all software in a certain way like will become like AI powered over over time but but like clearly there's like a set of like building blocks of like you know doing more sort of LM powered software that seems to be sort of you know common across all the different use cases and we're we're like you know the way I think about model is like we're in the business of building building blocks you know we we want to give you like amazing generalizable battle tested you
37:39know general purpose uh composable building blocks and and I think there's some really interesting stuff also like you know with the with the agent use cases.
37:48That's interesting you bring up training cuz I think in another podcast maybe like a year ago maybe like two years ago you said like training you were not focused on like specifically like what changed in the market?
37:59I mean we we've been building a training product for multiple years. So on and off. So, so we we I I wouldn't say we're like we never been focused on it, but it's like it's like one of those things where like I mean now we're like 150 engineers but you know a year or two ago we were a much smaller number of engineers and so but just virtue of you know being smaller we had to focus a little bit more and so inference became like our sole focus and and was our sole focus for many years but we're bigger now like we we we do have opportunities to also place bets in in other markets.
38:28I'm also really interested in you know kind of continuing the the the list of things post training reinforcement learning. I think there's some really exciting stuff around that. Um I I think there's also many other like stuff around like storage and building data warehousing and feature stores and real-time streaming pipelines and and and all kinds of like I don't know. I I think the long-term view of of modal is like basically like you want to build some cool AI like you know we got you covered like you know we got all the building blocks like come to us and like
38:57we'll help you you know take all these building blocks and compose them and that's the beauty of software is like you you can you know build almost anything if you have the right building blocks that that's very much the sort of company and the product they want to build. When you say training, are you focused on more of like the bigger enterprises doing huge training runs or Okay.
39:13No, they're going to go and like, you know, do these like, you know, big hyperscaler or or rather Neo Neo cloud mega deals and get, you know, 100,000 GPUs or whatever. But but but there's like a little bit of an underserved market of like in my opinion, you know, like researchers iterating or like people training smaller models. There's you know text to speech and speech to text models. Computational biotech is actually a market that we see a lot of demand from. We have a lot of customers in that space. Training relatively you
39:43know modest models you can train on 64 GPUs or whatever recommendation system. A lot of sort of traditional machine learning. There's a lot of things like you don't need a you know 100,000 GPUs.
39:53That market I think is like we're not going to like focus on that for a while but but I think there's a lot of stuff in the you know slightly smaller markets that that that are relatively underserved. It's very hard to set up training clusters.
40:05Yeah. Have you noticed like since I don't know like GLM 5.2 or like Kimmy K3 or maybe like Kimmy 2.5 uh that you know a lot of people are using those and like fine-tuning them. If anything I've actually seen the other direction. I feel like fine-tuning the value prop a couple years ago was like, yeah, you know, open source models, they're not quite as good, but you know, if you take, you know, small model and you fine-tune it on your specific domain,
40:33you can actually reach the frontier in terms of, you know, quality and cost. Uh today I I would say like the open source models are so good that like you can basically just like swap them out and just run you know instead of you know the the Frontier ones you can run you know Kim K3 or or whatever DeepS 4.1 or GLM 5.3 and they're like basically almost as good as like the Frontier model at a much lower cost. So we still see a lot of demand of uh for for postering and I do think it's a really interesting space and we're working on
41:03some some products in that space you know both sort of SFT and and but also reinforcement learning and things like that. I would also say you know fine-tuning is no longer the thing that you kind of have to do like it's it's very optional like you can open source models like work really well just off the shelf too. back to sandboxes.
41:22Actually, I forgot to ask you. So, like what did you have to update about the sandboxes that you guys um were offering uh that like existing sandboxes did not uh did not allow for?
41:34Not a whole lot. I I think the thing we've always been very very good at, I mean, we're infra geeks, right? Like we're very good at like, you know, scaling and reliability and and doing a lot of stuff. And and I think that's why that explains to a large extent like why we have you know in my opinion I don't have numbers but like I think we have by far the largest market share in the space like we're just very good at running you know hundreds of thousands or even millions of sandboxes.
41:56I will say we'll have to we've had to rewrite our systems you know a bunch of times in order to handle that scale. So so but we were always like very good at it. There are other things like over time I mean the sandbox space in in general like you know like it's like not the most like differentiated space but you know we're working on some really cool stuff like over time like we've had to add support for VMs like virtual machines so we we we previously built our our sandbox product on on G Visor but now we offer full virtualized machines as well. Turns out a lot of for
42:24certain use cases you kind of need full Linux compatibility. There are other things we've had to build, you know, better primitives for like more interactive, you know, piping data back and forth between sandboxes, working with file system. We've had to spend a lot of time on snapshotting. I think a lot of the things that over in the long run will differentiate will be like, you know, really cool things like, you know, forking sandboxes, snapshotting, resuming snap like live migrations
42:52in general. Like I I think sandboxes, the novelty of sandboxes was always, you know, this sort of ephemererality like you can spin off sandboxes for a few seconds and and and that's something I like that's a string I I want to pull more in. I think there's a lot of really cool stuff you can do, you know, kind of decoupling the agent and and the harness decoupling it from sandboxes, having agents control many sandboxes, forking sandboxes, you know, snapshotting things like, you know, like running basically like very shortlived m maybe one per command, like things like that. So I I think there's a
43:21lot of cool stuff there. We've already spent a lot of time doing that, you know, around snapshots. Uh but but I think there's a lot of cool stuff to be to be built there as well.
43:29You brought up snapshotting a couple times. Like what do you see as like the use case for snapshotting? Like like what would that give us that we don't get right now?
43:37I think the the big thing for me is like today a lot of sandbox workloads are kind of like lift and shift. Like people think a lot about sandboxes almost like you know I have my dev box now I'm going to shove it in the cloud. I'm going to have an agent control it. Maybe I'll run the harness inside the sandbox. I don't know if like longterm that's like the right way to think about, you know, on like compute for agents. I I think a much better model would be you're going to have this like very long running
44:05agents which almost like to me are like kind of durable execution workflows.
44:11Like they're going to run, you know, 24/7. you know, every once in a while they're going to run commands and and you know, having the ability to like spin up a sandbox in, you know, 100 millisecond, run a few commands, and then like shut it down and like snapshot it. I think the faster you can get that like, you know, wake up, you know, go to sleep cycle, the more you can sort of create sandboxes that only run for like a very brief period. And and that will make compute much more efficient. And it will also enable you to to to do some really cool things you know like forking
44:40and and moving things around and and and you know having like imagine you know like clawed code instead of you know having sort of a very linear flow like sequential flow like kind of branching and you know running hundreds of agents at the same time. I think a lot about like CI is like another like illustrative example like for most of the the history of CI uh you know talk about like GitHub actions and things like that like you had a very sequential flow like you you know you slowly install a bunch of stuff then you run the test suite and then you know
45:09generate some reports or whatever and it's like you can break it up into shards but it's kind of annoying like what I would love to do is like take your entire test suite and just run every single test in its own sandbox right like and just paralyze the out of that run like a thousand sandboxes Why can't that be done? I mean the problem is like there's too much there's too much overhead around like you know starting and stopping sandboxes. So I think the more like you you create like you know frictionless like starting and stopping the more you can enter a world where like you can do this like
45:38extremely bursty like large scale like run as much as you think want in parallel. So, so, you know, I'm an infro geek, like I said, like I I I get very excited about these things. You know, once we figure out all these primitives and the right building blocks, I think it's going to unlock a lot of really cool stuff that, you know, we we previously could not do with with software.
45:57Why hasn't that already been built? I would assume even previously people would would want to parallelize as much as of their CI as possible.
46:05You can't, but the overhead it's like Andall's law or whatever, right? Like the overhead of parallelization right now is very high cuz like right now like in GitHub actions for instance like we we paralyze in in the modal test suite, right? But like what that basically means is like now if you take your test suite and you split it up in two like now each of those shards both need to do all the installs and and the test setup and that takes a long time like it it can take minutes but let's say you can do the test setup once and then you snapshot the container and now you clone
46:34it and now you can run the test suites independently. I think a lot of a lot of the the benefits cannot be uh you you can't materialize those benefits until you get the the friction and and the overhead down the transaction cost.
46:48I think you guys have a partnership with Anthropic and they're using sandboxes for something like modal sandboxes.
46:54Yeah, I mean the the managed agents with with Enthropic uh there's a sort of bring your own um sandbox integration. So you you can use modal for those.
47:04Oh, cool. How did that come out? like come maybe I think anthropic is like trying to create you know like a thriving ecosystem and community and and so I think you know they make money from inference right like so I I I think it's very much in their interest to promote you know like a for everything else you know as much of you know of an ecosystem as possible and they want to have the largest possible surface area as long as you use entropic for inference so I I think it makes like very rational sense for them to to to partner with with us as a sandbox provider also I feel like we have to bring up the
47:32hugging face incident are you okay talking about that.
47:35Yeah. Yeah. It's all there.
47:36Yeah. Yeah. So, yeah. Could you just briefly describe like what happened? And I think they were using the modal sandbox, but it wasn't your, you know, it wasn't like a fault of the sandbox.
47:45Yeah. Exactly. I I I don't I forget the all the details uh of of the exploit, but um I I mean, you know, I think what you're referring to is the fact that like modal was involved. Yeah. And so, you know, just to comment on that, like basically the the the agents running inside, you know, that that that exploited hugging face, they discovered a a vulnerability in an app that was deployed on Moto and they used that to sort of as almost like a springboard for for getting access getting full internet access is
48:16I don't remember the full details. To be clear, this is an app running a modal that was hacked like it was not modal that was hacked. So it's like AWS, a lot of people run stuff on AWS. Occasionally those app get hacked. Does that mean AWS got hacked? Like no, of course not, right? And and so there's a lot of people running apps on modal. And you know, one of those apps got hacked. And I I think but but like there are some lessons we can learn and and and I think one of the things we've been reflecting on is like could we have more sensible, you know, security restrictions by
48:45default. I think to use the AWS analogy in the just just as another analogy, you know, S3 used to be this like, you know, it used to be very easy to to accidentally make an S3 bucket public and AWS realized like, you know, it doesn't necessarily cause it, you know, pose a security problem for AWS per se, but a lot of their customers accidentally would, you know, like leak data. And so S3, it's much harder now to make S3 buckets public. And so in similar way it's like we're exploring
49:15like you know can we make security a little bit more restrictive by default without necessarily you know destroying the magic that make that makes good and I think there's also you know as kind of a separate sidetrack is like agents are getting really good at cyber security so we've actually also as a sort of you know separate sort of learning from this incident like we started using agents internally quite a lot to to to try to find vulnerabilities so far hasn't found anything but like they're very good at like probing into like very deep parts
49:42of the system and exploiting trying to exploit things and trying to escalate and get out of sandboxes. Um, so you know, it's it's it's something we're exploring very actively right now.
49:53How exactly do you get the agents to like run through the codebase? You're just like, hey, like find security v vulnerabilities.
49:58Well, no, you just like you just put them in a sandbox and and tell them like, can you try to explain?
50:02Just run through the can just try to escape from the sandbox. It's like escape the room but for agents.
50:08Cool. When did you start doing that? We started doing it a few months ago and it's pretty exciting like when Fable I guess came out.
50:15Yeah, roughly around the same time.
50:17I mean my my view on like cyber security is that you know I actually think they're very good at finding vulnerabilities but I think you can also use them defensively very very well. So I I I think you know companies like modal and many other companies right now like I think you know if you go to talk to like pretty much every season I'm sure they're all thinking about the same thing like how do we use agents defensively to to to you know fix all our vulnerabilities. So over time actually in a way like I'm not that nervous. I think if if anything like
50:45agents and and cyber security all the breakthroughs in cyber security will will make the world safer if anything.
50:51Do you think that like all of the cyber security the big cyber security companies are just cooked? I don't know if that's true because I mean you're asking something that's like very far outside of my like core expertise but when I think about like cyber security a lot of what you're selling is effectively fear and that's almost like a branding and like marketing strategy as much as you know obviously the core products but I think you know brand and and things like that are actually like
51:20arguably like more valuable in the era of AI so I I I wouldn't necessarily jump to the conclusion that they're cooked hooked. I think they have, you know, deep relationships with the enterprise customers and they have, you know, big sales teams who are very good at telling, you know, spooky stories and and things like that. And and I think they're going to continue to be very good at that.
51:39I just ask cuz like, yeah, you're doing it yourself. You're not going to one of like the Palo Alto networks or whatever. But I guess like it's only like giant enterprises that care about that. Also, they can blame somebody if something goes wrong.
51:49That is true. Yeah. Yeah. you kind of want to hire in many cases just like a established vendor uh in order to blame them. I I mean to some extent it it does reduce a little bit the need for us to do to bring in pentesters like human pentesters which most startups like do on a regular cycle. So maybe previously you know we would do it I actually don't know what the cycle was but but I think we used to do it like once every six months roughly. Maybe now we only need to do it once every year. I I don't know. But but like I think with all of
52:18these things, you know, humans can also use them to get more efficiently. So like, you know, like I I don't know. I'm kind of a believer also that like humans with AI is even more powerful now. So like maybe there's an argument that like we can bring in pen testers who are really good at using agents and, you know, find vulnerabilities even faster.
52:35I I don't know. I I'm not that you know, I'm an optimist. I started a company. So like, you know, I'm I'm generally like optimistic about everything. But like I I certainly don't think humans are cooked or companies are cooked. Like I think in the era of AI a lot of companies are going to figure out you know how to use AI to be even more efficient.
52:51I want to transition a little bit to like developer experience because like early on I mean even now like you guys focus a lot on that and you've talked a lot about that. Does developer experience even matter anymore? I think it still matters and then I think you're you're sort of you know like the context I think you're fish or like the thing you're talking about is like now with agents right like does developer like the docs are already there the agents are going to look at the docs like all that stuff to that point though like almost all the things that make developer experience
53:21good also makes the agent experience good is kind of what we've seen in practice which is like things like having good docs having descriptive error messages having fast feedback effect loops, right? Like I do think there's certain things have changed. The other day I was like setting something up with Cloudflare and it's like I find Cloudflare like notoriously annoying to work with sometimes and and I threw an agent at it and it was able to like just configure a bunch of like as you know things in in Palumi for me and you know
53:50and and so I will say like agents does they do make it a little bit easier to work with things that are like harder to work with in the past you know like the sort of human toll of dealing with YAML has arguably gone down a little bit because you can just like you know tell your agent to deal with it but but For the most part, I I think you know the things that make developer experience good also make agent experience good. I think to some extent also like infrastructure over time anyway becomes and needs to be more programmatic. I
54:19think that's always been the thesis about sandboxes is that you know like things that you previously had to configure using YAML that should just be an SDK that should be more like imperative like launch a sandbox like do stuff with it tear the sandbox down right like rather than sort of declarative using using you know some sort of DSL it also pushes like a little bit of a different interface that maybe previously didn't exist. I think in either way like SDKs are as relevant as they as they always were if not more. Is there anything you
54:49changed in your product just because like you know most people are just running agents on on it?
54:53We do uh I mean we also we actually do run a lot of internal tests where we have agents we tell agents to like build a mobile app and run it. And I mean a very concrete thing that is kind of funny is that we've actually seen that like because of agent training data it's very hard to deprecate things now. uh we see that like agents often like love using you know old versions of modal syntax and and it's kind of hard to nudge them away from that. So so we've actually had to be a little bit more conservative going
55:23forward like you know over the last year or so with with how we deprecate and remove features. It takes we have to be very careful to update every single example that's out there online. Yeah.
55:33And and and you know skills, right?
55:35Yeah. There's like stuff like that. But but like there's also an insane amount of like modal code that's now like online too that goes into the training data. I've seen like that to be like kind of an you know a little bit of like an interesting example where like another thing by the way too is like there's obviously like the sort of a similar to like search engine optimization there's like agent engine you know whatever people call it agent optimization. We often like run internal benchmarks and we're like ask an agent like hey like can you deploy uh stable
56:04diffusion for us? use whatever tool you want and we see if it recommends modal or not and those things are very useful for us to sort of track over time or like hey spin up some sandboxes or like hey like rank all the sandbox vendors.
56:16So so we we spend a lot of time like doing doing those things to to make sure I mean what do you do about that? Yeah, you say hey like you know build this stable diffusion thing does it recommend modal or not? Like yeah what do you do about that?
56:25Yeah I think that's harder. I I think you know it's like search engine optimization. I think it's primarily through content is like you just have to put out a lot of content.
56:35You can also to some extent like sort of see the perception I think a little bit more like you can ask an agent like you know how does you know rank all the sandbox vendors by reliability or rank all the sandbox vendors by price and and sort of see and it does feel like agents are a little bit more prone to like misunderstanding pricing and and you know things like that. So we also spend a lot of time like just making content more clear and more like you know understandable for
57:04agents cuz we know that all the indexing like everything that's out there and that's talking about modal.
57:09Have you thought about just like making a bunch of GitHub repos and having the agents like write a bunch of modal code in there? Have you tried that?
57:17Slop cannon. Uh I think the truth is like that is probably something you can do. My my guess is like LM training data today is probably quite insensitive to like what's like AI output in the first place and you can probably like create a lot of slop like we thought about it like I don't love that idea but like in theory like much like you know SEO in the early days and still to some extent like autogenerated content is like a very effective strategy I think it's possible like you know you could you can do those things
57:47too today I want to switch to FTEEs apparently you guys hire like forward deployance engineers like what do they do? We're in a new era of like building applications and and and you know things that we couldn't build you know just a few years ago and we see a lot of companies that you know they need a little bit of help right like you know and so we started investing in FDES a couple years ago but it but it's really been this year we've kind of scaled it up a lot I think we started this year having like
58:16four fds now I think we have like 24 or something like that so we've grown a lot and they do a you know it's more it's not exactly the counter model but they do a lot of like heavy you know sort of customer engagement like they work with a lot of customers on performance tuning especially with LMS we see that there's so much benefit from like you know sitting down with customers and training draft models and tweaking uh you know parameters and DM and SG lang um they often you know fix bugs or
58:45or not fix bugs but like you know when customers run into issues like they often work with customers trying to figure out like how do we work around that or uh another thing we we've used FD these a lot for is like you just kind of proactively you know if if we see certain customers you know overprovisioning with modal like customers requesting too much CPU or too much memory for instance like that's kind of expensive and like we we try to catch those things and we're trying to sit down with those customers like yo like you're spending a lot of money on modal right now we can actually help you save money and then of course like with
59:13with sort of active deals you know we we help them migrate code from other platforms uh things like that um I think you know we we never have the intent ention of becoming like a service heavy business. I I think that the goal for FDS is like you know A is to be a part of you know the the sort of sales process and like closing deals. B is to serve customers and and do some like you know customer support but like I actually think like the most important thing is like C is like bridging gaps in
59:43the product that we later want to turn into like real features. So I think to a large extent, you know, anything an FDE does, you know, my question is always like long-term, how do we turn this into product? So we don't need an FD for this.
59:58Do you guys hire PMs, product managers?
1:00:02Would you say these FDAs are kind of like pseudoish PMS?
1:00:06It's a little bit of a different function. Like I I think certainly on the more like you know customerf facing side they are but there's like you know on the more like inward you know internal side like there's a sort of separate part of like product management which is more like what are the next building blocks Moto should invest in and like you know and then there's like something in in in between like you know looking at like maybe 10 20 different customers. They're all asking for like slightly different things but maybe what are the patterns like how do we turn this into product? We do have people doing product with us and and it sort of evolved you know we've had people
1:00:36basically like become product managers after previously being either deval or or engineers just showing strong aptitude on on doing thinking very strategically about what we have to build. So I'm actually not like anti- product management. I do think that a lot of companies in the past hired product management product managers too eagerly and and I think there's a risk that you you kind of I can sort of see it in a lot of products like certain
1:01:04products like you can tell like there's just like eager product managers that just thought of like kind of like widgety things and just ship them. I think doing product management at an infra company really well is very very hard because I think it takes like a very deep you know level of strategic thinking thinking about like what are the foundational building blocks where do we want to be in one to two years because the truth is building infrastructure is very hard and how do we make sure we don't end up with like 35 features that are like kind of
1:01:33similar looking but all do like a little bit slightly different things. That sort of level of thinking is something that you know is very rare to find and and for that reason we haven't aggressively tried to build up a product management team yet.
1:01:45Yeah, that makes sense. How would you say your teams have been evolving maybe like since GI? Cuz I would assume Yeah, I mean you obviously worked at like Spotify. You had like a certain team structure? Um have your team structures like evolved a lot now that you know we have coding agents? in a funny way like yes I I do think team structures have changed a lot but I'm actually not convinced it was like geni driving that.
1:02:07I I think in particular like one thing I've seen a lot is that there's been sort of a focus like a swing back towards being an IC that I actually think is very very positive for the space as a whole. I think you know if you look back at the space like 10 years ago 20 years ago everyone wanted to be in management because that was a way you you would get more money and more you know more recognition and more prestige.
1:02:30I think that shift actually started even before Gen AI but I do think it's like heavily accelerated with AI like you know we we we make it very clear to everyone at modal that like manager RC it's like there's no difference in prestige like everyone is just a member of technical staff you know it's about ultimately your contributions and we see a lot of ex-managers you know becoming IC's and and because they want to go back and build code and so and you know and there's been there's been sort of shift away from like you
1:02:59non-technical managers. Everyone pretty including me. I I still write a lot of code at Model and in Oxhat 2 and and pretty much everyone in in the tech team and a lot of people who are not in the tech team still write code. So that to me is like you know a recent phenomenon but but I actually don't know if it's like necessarily cost by Gen AI but I think Gen AI has like greatly accelerated it because it it made it even lower the cost of like writing code is even easier. So if you have like non-technical people like committing
1:03:28code like how do you how do you review that like how do you guys do code reviews?
1:03:32We do code reviews but it's we we think of it as optional. I mean for most my my view is like code reviews in in large part is like people in many cases trying to just escape the the accountability of like you know merging things into production and and if you have that like you know if you have that culture like then to me that's a sign that you need to invest way more in you know CD and and CI and and and things like that CI in particular but also CD uh continuous
1:04:01integration because it's deployment we do code reviews like I mean like when I write code like occasionally like I'm I'm asking you know authors like is this in line with how you think about things?
1:04:10I hope I'm not changing the architecture fundamentally. But we also make it clear internally that like code reviews are are quite optional. And you know if you're just changing docs, if you're just doing you know some basic stuff like don't wait for a human to look at it especially in the era of AI. Why do you need a human to look at every piece of code? You don't have to but that does require you to have you know very good controls you know internally around you know testing for a company. Yeah. And exactly like you know infra companies like we we need one more nine than the average startup
1:04:39right in terms of uptime but like you know these are investments you you just have to make like you have to build systems that are very resilient to like you know you have to have rate limits everywhere. You have to decouple systems. You have to make sure that like nothing is a single point of failure. They have to have extremely good test coverage.
1:04:57Agents are actually really good at writing tests these days, which was like, you know, a few years ago I thought like one of the major things missing in agents. Now they're actually quite good at like because I think it takes a little bit of like sort of kind of like need to step out of what you're trying to do for a second and be almost a little bit like devil's advocate to write good tests, but now they're actually quite good at writing tests too. So I think I think all these things help.
1:05:18Do you guys use like frontier models for coding or do you guys have you guys experimented with uh like the open source ones? We experiment with everything but yeah I mean you know initially was a lot of cursor then you know of course claude and codeex and and now a lot of people are using you know night shift and and all these open source stuff or or other stacks so I don't necessarily have like a strong point of view like I want people to experiment and try different things like we you know use frontier models use open source models we're trying to do a little bit more dog fooding internally now that we're actually running inference for a lot of the open source
1:05:46models like at a very large scale I think it's also helpful to like use our internal endpoints for a lot of agentic development Uh, so we try to promote that a little bit.
1:05:56Something I read maybe like I don't know a year ago, maybe like two years ago, you were saying that you didn't think that there was going to be like a big onrem business for you guys. Um, have you maybe like changed your opinion on that?
1:06:09No, I have not changed my opinion. Um I think modal to me I I'm a cloud maximalist and maybe to take a step back like when when the cloud came in 2008 2009 or whatever my first reaction was like that's crazy like why would I want to take my code and run it on someone else's computer that's like insane and then like 6 months later I was doing it and I think you know like it took me 6 months to change my mind it took probably enterprises like another 15 to
1:06:3720 years but like you look at you know vendors like snowflake or I mean AWS and like you know I I think there's many examples of data dog etc like the cloud is clearly winning right like and and the multi-tenant systems are clearly winning and you know to me onrem looks increasingly like sort of a legacy business model and and I actually think you know BYOC you know sort of VPC peering like all of these things you
1:07:06know like to me they reflect sort of an outdated security model of like we need to protect everything inside our own network, right? Like using the network as a moat and that I don't think is true anything anymore, right? Like we've sort of moved on to more like actually robust security paradigms like you know zero trust and things like that. And so to me the things like they they look increasingly as like outdated at least from a security perspective from our
1:07:35perspective also like the fact that we have full control over our entire infrastructure stack. We rent all the hardware. We deploy we manage all the infrastructure. We deploy and like manage health and observe you know all the metrics on all the hosts. It means we can iterate very fast and build a greater product and and I think that's very very valuable versus you know customers in the past who had enterprise additions that they had to deploy on customers environment. they have to be very you know they have to think very carefully about release management and and having depend like like we don't
1:08:05care like if we want to introduce a new database like we can introduce a new database because we have control over it like we don't need to think about what customers have access to. So to me multi-tenency is is it has so many different benefits. I mean that being said like you know if we were to work with like you know some very large Fortune 500 customer in the future and you know they ask for BYOC you know we've done this in the past we'll give them a number like you know that number is very high and no one said yes to it
1:08:33so far but like who knows like you know at the right price I I would still build it and and I think there's like maybe separately like another thing which I think is interesting which is like in theory BC might also be a way to work with and and get capacity in certain ways like we we sometimes work with Neolabs and those Neolabs already have very large capacity reservations and you know the question is like you know how how do we how do we get them to run modal when they already have capacity and BYC might be a solution we can still keep the control plane in in our
1:09:03environment and the other flip side of that is like we could maybe even run like other people's stuff on that capacity so so there's some like kind of fun stuff you could do in theory where like you can sort of you know like kind of aggregate all that capacity into one big pool We're excited about some of those ideas, but in general, like I feel like on prem I don't know if we're ever going to do, but BC like maybe bring your own cloud.
1:09:23I keep saying BC. I should I should explain acronyms.
1:09:26Um let's say hypothetically you are going to raise I should send more money like soon. What would you use that money for?
1:09:35I mean like I mentioned like in in this capacity situation the a lot of the money a lot of the cash flow I should say that's required is is capacity prepayments, right? and and so that to me represents the best use of of of cash today. Uh we're ramping up capacity capacity very aggressively. Maybe we move into, you know, other types of deployment models in the future, ownership or whatever. But I think there's always going to be a cash component of that, like some sort of down payment or prepayment or some sort
1:10:03of that. So that to me is is the the greatest use of cash right now. There's of course other things like you know salaries and office spaces and stuff like that but that actually in the grand scheme of like our our uh financials like it's it's basically a rounding error right now.
1:10:17Yeah. Interesting. Do you think that's going to change?
1:10:20No. I I think you know the the sort of operating leverage that we have in this business is kind of insane. Like you know we have 200 employees you know they cost a lot of money but you know the the the GPUs cost a lot more money.
1:10:32Yeah. If you had to say like what are the top like I don't know like three things that are your biggest focus or the business's biggest focus right now like what would you say those are capacity and the sort of financial complexity and you know in general like how do we how do we think about that space how do we think about pricing and packaging how do we think about the sort of tension between ownership and renting should we be go in a more neo cloud direction should we go in the opposite direction the sort of counter precision like that's probably the biggest thing that I think a lot about uh the second
1:11:02biggest thing I think a lot about is um probably around like more like product strategy like I think a lot about like what is the next building blocks like if if we you know do more like agentic stuff like what are the things we need you know are there certain storage primitives or compute primitives or in general like what is the thing that the world is going to wake up and realize they need over the next few years and and ideally we have that at that point we've been pretty good in general at like you know anticipating demand and building products in advance of it with the serless functions and the sandboxes
1:11:32so what's the thing. Then I would say like actually the third thing I think a lot about is like more around like culture and company building and stuff like that. I mean we recently crossed 200 people. We're we're growing extremely fast. We'll probably hit 250 by the end of the year. Like when you're growing very fast, you have 250 people that are all trying to figure out how they can add the most value to Modal and you know making sure that they have the right context that they have the right motivation that they're excited to come into work every day cuz they get to
1:11:59build something incredibly cool and they know you know what makes modal successful what they can do. aligning the incentives and and making sure that you know we we people have like a very positive culture where they have the sort of implicit trust to do a lot of stuff but also the accountability to like you know deliver the best things and you know and and be held accountable to like if they're not doing it. To me those are like you know super critical things around this scale. Like when when
1:12:27you're like 10 20 30 people you can all just like kind of sit in a room and like it's all kind of implicit and you kind of just like feel it. people can sort of read read me like they can tell me when I get a little bit annoyed. They can tell when I'm happy like now we're 250 people and like people don't necessarily see it and you know of course there's like a lot of new people they're like maybe a little bit nervous to like talk to CEO and like all this stuff. So like you know how do I remain approachable and how do I like it's it's there's like the Dunbar number which I think is like
1:12:55more sort of you know faked than than anything like at least like the story behind it but like it is actually kind of real still like there's something around like 150 200. We're like laying the foundation of the culture is very very important.
1:13:07How many people are you guys trying to hire over the next year?
1:13:09We've grown about 4x on an annualized basis in the last 12 months. I think that's like an unsustainable growth rate on a relative basis. I've seen that, you know, you can double the team every year and that's not super hard to do. But Forex, what you start to see is, you know, cultural drift. You start to see islands of different working styles. You start to see people doing like very speculative things like that. That's something I always watch out for in the engineering team. You go and talk to the an engineer and I'm like, "What are you
1:13:39working on?" And and it turns out they're like building something like kind of like fringe and like is this really what you should spend time on? So growing very fast has that cost and you know it has the cost of ramping people up. So I I don't know I think by the end of the year maybe 250 by the end of next year you know but but like you know we're going to keep growing like maybe 2 and a half 3x so it's very likely we'll end up like next year like 700 people or something like that.
1:14:00I'm going to assume most of those are engineers.
1:14:02Most mostly engineers have sales a lot too.
1:14:05Okay. What kinds of engineers?
1:14:08It's primarily systems engineers. I mean, we've always been like very deep into sort of, you know, Linux and file systems and scalable systems and networking and and cloud and and stuff like that. So, so that that's that's the core of what we're looking for like people are really good at those things.
1:14:23We're also looking a lot at like you know more like cuz we moved up in the stack a little bit like we're doing a lot of work in the LM inference space.
1:14:30So now we're dealing with a separate set of concerns around, you know, CUDA and like GPUs and and and and MFU, you know, like maxing out the tokens that you can get per GPU and and and a lot of that things. And you know, even though I actually like will say a lot of the work in that space in my opinion turns out is like more systems driven still because it turns out like KV caching and routing is actually like is still like very like systems like like these are very like
1:14:59systems heavy problems but yeah a little bit more like sort of GPU performance that kind of stuff. We're starting to build up a research team. We're very excited about, you know, moving into stuff around like, you know, post- training and and distillation and and better methods for for quantization and for your customers or for internally both. I I think it's like a little bit like with the FTEES like I I I think we initially will do a lot of stuff with our customers to optimize their models, but over time figure out how do we turn
1:15:29those into products. I think we'll sort of arguably fail in a certain way if every time a customer needs to optimize a model we have to go and work with them and you know longterm it has to be turned into products and and so with the FDS and with the research team initially you're going to have to do a lot of work with the customers and we are that but over time we're also building a lot of internal tooling and a lot of overtime internal products that are fully self-s serve uh so we're working on a lot of stuff right now in in the post- training
1:15:58space but also in in the GPU performance space. So, a lot of cool stuff. A lot of cool stuff.
1:16:03Back to the product strategy stuff that you were talking about. Um, yeah. Like what are your thoughts? Like what do we need over the next year, 2 years?
1:16:11I think it's incredibly hard to say and and if I said like I know exactly what we need to build like I think I would be like delusional. I think what what I've what we've seen so far, you know, and what I kind of believe in on a more general level is like we need better building blocks. like this is a new era of of software and we unlocking a whole new type of applications that could not be built a few years ago you know doing you know all the geni stuff code that
1:16:39you know LMS that output code that you know it needs to run like in an isolated way like there's so many like new primitives that are needed to to work with GPUs and work with LMS and you know we built some building blocks so far and and you know that's unlocked you know a bunch of applications that that the world you know has seen over the last few ers, not just us, of course, but I think over the next few years, like there's going to be so many more cool things. And so I I I mentioned some stuff like I'm very bullish on training.
1:17:04I'm very bullish on bass jobs. I'm very bullish on, you know, certain things around like agents and and low latency and and things like that. But but like in the end, we consider ourselves like an infrastructure shop, right? Like we're not trying to build, you know, very narrow things. Like we try to build like very general purpose building blocks and put them in front of customers and then those customers will combine those building blocks and build some incredible applications that we could have never anticipated.
1:17:30I want to go through some of your tweets. This will last part of it and we can just like maybe you want to double click on anything.
1:17:37A lot of long running persistent sandbox stuff right now. Uh today feels lift and shift. I suspect sandboxes will evolve from devox in the cloud to something more ephemeral. uh one environment for every command and snapshotting in between so you can pause resume clone. Yeah, I mean we talked about this a lot.
1:17:52Yeah, we talked a bit about it, right?
1:17:53Like I I I'm fully I I think I think the median soundbox in modal today is like 8 seconds long or something like that like which is kind of incredible when you think about it and and over time I think it might get even shorter. We want to support longunning sandboxes too. Like I think there are some like very valid use cases around longunning sandboxes. But like my vision is like you know the world needs also we need to figure out this like how how can we just massively parallelize things and reducing the overhead of starting and stopping sandboxes.
1:18:22I don't know if I ran an IPO ready startup engaged in capex arms race with a competitor it would be nice to coordinate and pace the arms race a bit.
1:18:31The biggest downside of GPU scarcity is not that it's hard to get GPUs, but that almost all capacity is sold as three years as a three-year rectangle starting 3 to 6 months in the future. That's not what demand looks like for most users. I think we talked about that a little bit as well.
1:18:44Yeah. And you know, maybe just go a little bit deeper. Like one thing I think model has always been very very good at is fully on demand capacity, right? Like if a customer comes to us and need 100 GPUs, like we can typically give them that like immediately. You don't even have to talk to salesperson.
1:18:58And a lot of workloads in in in real life like they don't look like these like flat, you know, rectangular workloads. Like they scale up and down throughout the 24 cycle. They maybe grow over time. They maybe have certain burst periods. And to me that is a challenge with a lot of customers today. Like the and I do think it's a legit problem for for the space is like the only the prime way to get capacity is to to buy these rectangular blocks. It's not great. I think the world needs to evolve and and that's fundamentally I mean I'm very biased but like that's why I think modal
1:19:27can play a very large role in in that sort of you know whatever you want to call it sort of duration transformation of offering more flexible capacity and we can do that because we can aggregate many many thousands of customers and the demand gets very smooth in aggregate so that's a lot of the value prop of modal is exactly that would you say that's a differentiator like the oh really okay yeah massive differentiator actually in fact like I think modal's like if you want to like very sort of reductive like you can break down modal's value prop into two things. One is the developer
1:19:56experience the ability to like iterate and like build things and not have to think about infrastructure. The other thing is the capacity stuff. The more and more I think about modal and like where we're going in in the long run is like we're building like we're aggregating the world's compute capacity and we're very good at slicing it out like very granularly cuz we built all this work around you know scaling up and down extremely fast you know all the stuff around GPU snapshotting. We're working on some some ways to make it much much faster and that model works at very large scale like really really well because you can aggregate lots of customers.
1:20:26Every month I keep saying our sandbox revenue might have plateaued but it keeps going up 30% every month.
1:20:31It's kind of nuts like I don't you know like every month I'm like well you know sandboxes grew a lot but like clearly they kind of plateaued and then they keep growing even more. So so I I I don't know and a lot of VCs like they're like how did you know sandboxes was going to be big? like and and the truth is like I didn't like like I I thought it was going to be bigger than it was in 2023 for sure, but like the fact that Sandboxes is a third of our revenue today, it's it's insane. And you know, we're obviously very happy about it. We we have like a very large team investing
1:21:01in sandboxes. You know, I don't want to like trivialize it like we obviously I do want to give ourselves like a lot of credit for for seeing sandboxes for what it is today a little bit back in even 2023. But I I I think you know if like I probably would have never you know taken a bet that revenue would be this large from sandboxes and the market demand would be so much. There's an interesting structural reason why in inference providers should exist separately from model builders. The best model keep changing a lot. Inference providers keep can easily reallocate resources between
1:21:30models since overall demand is quite smooth.
1:21:32So I mean if you think about all the Chinese labs and you know in the future hopefully more US labs too. every time a new lab releases you know the latest and greatest open source models like you see big shifts in demand you know for for these models and and so I think a challenge with you know if all these labs independently like held capacity they would see when they launch a new model demand would go up a lot and then another lab launches a better model and then the demand goes down you know but but of course like in the background they're working on the next
1:22:01version so like I think you know there's like a structural reason like I said like why you're going to you're seeing demand kind of shifting to like the latest and greatest Chinese lab. I mean, not like it's not like you know this is like everything just moves overnight cuz cuz a lot of models and prompts and stuff like like a lot of things like get sort of there's like a certain stickiness to it. But I I do think you know there's an argument that it's very easy for modal to like you know right today we're running you know x number of thousand GPUs for one model tomorrow we
1:22:31need to shift like you know 50% of that to another model. Sure. That's just like that's what the system was built to handle. No one even needs to tell us to do that. It just magically does that.
1:22:39So you agree that basically the LLMs are commodities.
1:22:44And if you agree, then where do you think the business for the big labs is going to be?
1:22:50I'm not sure if that's necessarily true, right? Like is Taylor Swift a commodity?
1:22:54Like she releases a big album and goes on a tour and makes a lot of money and then you know then there's like a down year when she goes and like records. I don't know if this is the best analogy and then she releases a new album, you know. So I I I don't know like she still has like a lot of brand and there's a lot of you know power and like makes a lot of margin. So like as long as the the labs like repeatedly can launch new models and as long as the margin that they have on those models are such that they they can sort of pay off amvertise like the the cost of training those models it could still be like a decent
1:23:24So you don't think that the intelligence is like plateauing?
1:23:28No, I don't I don't think so. I mean I I think it remains to be seen, right? Like models are today are like very good at like doing things that can be trained very efficiently, you know, using reinforcement learning. So So we've seen that initially with, you know, coding and writing simple snippets. Over time, it's gotten really good at also writing production level code. It's gotten really good at math. It's gotten really good at cyber security. Like these are all like very quite verifiable things. I
1:23:56I still think it remains to be seen. you know, our models going to be really good at whatever cooking food or I I I don't know like things that have like more like diffused targets and maybe longer feedback loops or whatever. I think humans probably like my feeling is like we'll be quite good and quite competitive with with models for a long time in those domains. But you know certainly you know AI models they're they're getting very very good as as I'm sure everyone can agree with.
1:24:23Yeah. So obviously like the models are getting really good at coding. They're already really good at coding. What other applications do you think maybe that aren't as big right now are going to become huge over the next few years?
1:24:34I I I don't know, but I think anything that has like kind of a verifiable the computational bio, I think you mentioned.
1:24:39Yeah. Yeah. Computational bio. I I think those are like not exactly verifiable, but there's like certain ways you can sort of run sim like you can use an LM to output predictions. Then you can do like computational like like methods to like simulate the molecular dynamics and get like kind of feedback loops. I mean like I think like material science is like maybe another area where like you have a little bit of that loop where you can sort of you know synthetically manufacture certain materials and measure the properties and you just kind of put an agent in the loop and like you
1:25:08know do things. So you know I have a physics background I'm pretty excited about that. So drugs materials what else drawing a blank but I'm sure there's many other things.
1:25:16Cool. I mean those are all the questions that I had. Is there anything you wanted to talk about that I haven't I'm good. This is great. This is a lot of fun.
1:25:23Awesome. Thank you.
