I recently joined David Goldman, a partner at Celesta Capital, on the TechSurge podcast to talk about the race to build the next trillion-dollar AI chip company.
We debated things like:
What if you could get 10x more tokens, at a fixed interactivity, for a particular model, out of your megawatts than you could buying something off the shelf?
It was a great conversation; hereâs the video and transcript.
Highly recommend reading / listening / AIâing.
This transcript is lightly edited for clarity.
Why AI sells as systems
David Goldman: Austin, welcome to the show.
Austin Lyons: Thank you for having me.
David Goldman: AI infrastructure is in the middle of one of the biggest capital buildouts in history, but under the surface, even though the numbers are going up, thereâs been a pretty big change as the dollar spend on inference has gotten bigger than the spending on training. This is a trend that youâve covered a lot in your writing and something you have a very unique perspective on. So I want to spend a little bit of time unpacking the implications of this shift.
Letâs start at the top with how these systems get sold and what system selling is. Historically, when cloud buyers went out to the semiconductor industry, they were trying to commoditize server hardware as much as possible. They would work with industry groups, they would try to get standard designs done so that they could drive down pricing and negotiate with each part of that design. But increasingly over the last few years, theyâve been buying whole systems. So maybe letâs start at the top and just say what goes in a system, and why is this trend happening?
Austin Lyons: Yes. I like the idea of starting with systems when weâre talking about AI, because thatâs really what itâs all about these days. Itâs rack scale, all the way out to full data center clusters. The compute at the heart of these systems is ultimately GPUs, and thatâs what gets the airtime â or AI accelerators â but itâs actually a full system.
So stepping back, the question is how did we get here, and why are people buying a full system from one vendor and not commoditizing it? The workload that we all care about today is LLM inference. If youâre running at the frontier and youâre wanting to use the latest and greatest OpenAI model or Anthropic model, these are models that are two trillion or more parameters. What that means is, even as you quantize these and try to run them without using as much memory as possible, you might still need something like a terabyte, or two terabytes, or several terabytes of memory just for the model. Because with these models, as we saw with the scaling laws over late 2022, 2023, 2024 â the bigger the models and the more compute you have, the better answers you get.
Way back in the day, we might have taken an AI model and put it on just one GPU. But if you think about todayâs frontier models, you need maybe terabytes of memory, and yet a single GPU can only have, for example, 288 gigabytes of HBM on it to store these model weights. So quickly you say, wait a minute, one model canât fit in a single GPUâs memory. So what do you have to do? You need several GPUs. You might say, okay, take that model and split it across four GPUs or eight GPUs. And thatâs how we started to work our way into âthis isnât just a single chip anymore, this is a whole system.â Thatâs how we got to, for example, the Grace Blackwell NVL72 rack â a rack with 72 GPUs in it.
Weâve gotten to a place where weâre talking about AI systems even just for inference. Even if you were only to have one rack, youâve got GPUs, CPUs, networking, power, and cooling. So back to the question of why the industry isnât commoditizing it: a big piece of where weâre at today is that Nvidia did a really good job of getting to this rack scale first. They said, hey, we can help design the whole system and make sure it works across 72 GPUs and 36 CPUs, and that they can all talk to each other. And by the way, it also takes software and compilers so that you, the model developer, can write your model and actually run it across all this. So even though the industry always wants commoditization, competition, and multiple suppliers, Nvidia did a good job of frankly getting there first and making this complicated system turnkey, so that for the software developer itâs as easy as writing their model, deploying it, and it â just worksâ.
David Goldman: So you need a very big, complex system to make these LLMs work in a cloud. But if Iâm a cloud buyer â Nvidia has famously high margins, and they charge that on all of the different parts of the system, not just on the GPU. If you go out in the Valley, there are all sorts of companies offering one piece of this puzzle. If Iâm a cloud buyer, why would I be willing to give so much margin to Nvidia instead of trying to piece each piece together â buy my GPU from Nvidia maybe, my networking from someone else, my software from someone else, my cooling from someone else? What kind of value do you get from getting it all at once? Is it around speed? Is it around simplicity? What are the trade-offs that people think about?
Austin Lyons: Yeah. We are not yet in the era where people are trying to squeeze down their costs as much. Weâve still been in the era of speed to market. We saw that with Elon Musk and xAI, where heâs been able to stand up data centers very quickly using Nvidia systems. I donât think Elonâs goal was to say âhow can we do this as cheaply as possible,â but more âhow can we stand up this compute and get productive tokens out of it as quickly as possible?â So even though a company can potentially pull the components from many different vendors, thereâs just a desire to buy it and stand it up quickly.
And to your other point, thereâs definitely simplicity there. We see even AMD, with their Helios rack that theyâre bringing to market, seeing that same pull from customers: I want to be able to buy the full rack, get it, stand it up as quickly as possible, and get software running as quickly as possible.
David Goldman: Each of Nvidia and AMD have had decades to put together this puzzle, and tons of financial firepower to do M&A. Nvidia acquired Mellanox; AMD has done a bunch of acquisitions to build out these system-selling strategies. Is it possible for a startup to compete now that you have to sell systems?
Austin Lyons: That is a tough and very interesting question. To your point, if youâre a startup now, you canât necessarily just come in and say âIâm going to build a better GPU, a better AI ASIC.â People are going to say, okay, great â now what do I do with that? Are you going to make your end customers piece together the whole system? We just said they want to stand it up and deploy it as quickly as possible. Or are you, as the startup, now going to have to take on building the rest of the system?
I think there are probably different approaches here. And by the way, this is why we see that it takes hundreds of millions of dollars now for a chip startup, when maybe back in the day you used to do several rounds of just a couple million dollars to prove out your little proof of concept. So can a startup compete here? Yes, but they have to be very smart. It could also be death by a thousand paper cuts if youâre trying to invent a new AI accelerator and youâre also trying to invent your own proprietary networking and maybe your own proprietary cooling.
Actually, for example, Cerebras took a very interesting approach. Theyâre an AI accelerator startup â theyâve gone public â and they make what they call wafer-scale engines. For those listeners who arenât as familiar: instead of taking a whole wafer of chips and dicing them up into individual chips, like two GPUs and a CPU that you package, they said, why donât we just leave it at the wafer scale and make the whole wafer our compute engine? But then they had to invent how you make them all communicate, how you cool all of that, how you deliver power to all of that. They basically had to reinvent all of it, and that takes a lot of time. So I think any other AI ASIC company that followed could look at an example like Cerebras and say: wow, they did very innovative technical things here, but they had to invent everything, and thatâs time-consuming and expensive. How should we do this differently?
David Goldman: Yeah. I think you see some parallels with the buildout of internet infrastructure in the â90s and early 2000s. You start with proprietary, then as open standards proliferate, thereâs more capability for more people to offer things, and you can get a little bit more mix and match. I know youâve seen SambaNova, for example, has partnered with Intel to offer x86 CPUs alongside their system. So there seem to be some trends, even though incumbents clearly have an advantage in system selling.
Prefill, decode, and the split inference workload
David Goldman: One of those trends, which I think youâve touched on a lot in your writing: as we move into this inference era, youâre starting to see the inference workload itself get split, into whatâs called prefill and decode. Can you tell us what those two things are, and why you might want to split that workload onto different types of silicon?
Austin Lyons: Sure. At the end of the day, if you just think of simply talking to an AI chatbot â you write a paragraph of explanation, âhey, I want you to go research this and give me an answerâ â the first thing the AI system, the model under the hood, needs to do is read through the whole prompt that you gave it. Thatâs the prefill stage.
Whatâs happening is, in parallel â one of the key innovations of the transformer model is this idea of attention â you work through everything in the long English paragraph, and you ask which of these words are connected to other words in the sentence. That way you can piece together the context of the whole paragraph and what each word is referring to. And you can do all of this in parallel. If there are a hundred words, you can look at all hundred words in parallel and compare them to the other hundred words and do these calculations to figure out whether theyâre related or not. Thatâs prefill. Youâre ultimately doing a bunch of linear algebra, a bunch of matrix multiplication.
Now, when itâs time to give you an answer â especially if you remember back to when ChatGPT first launched and was a lot slower â youâd see the answer come out word by word. Thatâs because every word that Iâm about to say depends on the words that I just said. And when I say that word, the next word also depends on those other words. This is decode. This is where youâre predicting the next tokens, or the next words if weâre talking about English. And that is sequential. You canât do it in parallel.
People running this inference at scale started to realize: hey, wait a minute. In prefill, Iâve got all these GPUs doing all of this parallel computation, and it actually doesnât require a huge amount of memory. Iâve paid for this high-bandwidth memory â which is very expensive, and getting more expensive by the day â and in prefill, that HBM is just sitting there being underutilized. Interesting. Okay, now Iâve got these other GPUs that are running decode, and theyâre actually not fully utilizing all of their FLOPS, all their units of compute, because decode is whatâs called memory-bound. Itâs a lot of âI made a prediction, and I need to get the weights and whatever I need from the KV cache, then do a little calculation, then go back and forth and back and forth,â one at a time. Youâre not doing things in parallel; youâre really just waiting on memory.
So at the highest level â even Nvidia led the way with this â they said: weâve got a bunch of GPUs in the prefill phase being heavily utilized for compute with their memory underutilized, and a bunch of GPUs in the decode phase basically underutilizing their compute and totally utilizing their memory. Naturally, any engineer is going to look at that and say: huh, maybe we should disaggregate these. Maybe prefill should run on systems that have lots of compute but donât necessarily need all that memory. And maybe on the decode side, we should really emphasize memory bandwidth â how quickly information can get shuttled around â and maybe it doesnât even need quite as much compute; we should essentially overindex on the memory part.
So naturally, we started to move into a world where Nvidia shipped a software layer for their systems called Dynamo, which helps orchestrate this across Nvidia GPUs. And that actually gave rise to the Groqs and the Cerebrases â these AI ASIC startups that were actually started even before transformer-based LLMs were the defining workload of our era. They had made architectural choices where they used a lot of this really fast on-chip memory called SRAM â where you use transistors to store the memory, instead of DRAM, which is capacitors and transistors, which is what HBM is made of. We wonât go way down into those memory details, but basically these early startups had made a bet on having really high memory bandwidth. And once the workload got separated into prefill and decode, they could raise their hand and go: oh wait, weâre actually really good at decode. In fact, we can go even faster than GPUs.
This took us from the inference era where everything was on GPUs to saying: what if you could slot in one of these AI ASICs, heavily built on SRAM, that can maybe unlock a thousand tokens a second? Whereas a GPU running decode â just the way GPUs are more general-purpose in their design and their memory hierarchy decisions â maybe could only run at a fraction of that.
David Goldman: Is that something you should always do? Do we always need to split up prefill and decode? Or are there just certain applications where itâs really good to have speed â Iâm willing to pay a premium for speed, therefore Iâm willing to go through the hassle of splitting these things up, having orchestration software, having different types of silicon and all the things associated with that â versus just running it on the GPUs or the system Iâve already bought?
Austin Lyons: Yeah, there are so many different nuances here. Take the hyperscalers deploying tens of billions of dollars of GPUs. Early on, GPUs are very general-purpose, very flexible â they give you the freedom to change your workloads. But letâs say youâre OpenAI and you say: no, these are our specific models that we know weâre iterating on â the frontier one, the medium-sized one, the small one. You start to say, hey, we should really cater to this workloadâs needs, and therefore it would make sense to deal with the complexities you pointed out â splitting up prefill and decode and orchestrating that. It might be worth it. Maybe that unlocks, for example, being able to sell an ultra-premium tier where you get really, really fast inference, and maybe thereâs a small subset of users who would pay 10x more to get tokens that are 5x faster. So I think there are certain model labs and hyperscalers at scale that are saying the complexity is totally worth it â weâre willing to deal with different SKUs, different chips.
On the other hand, I think there are going to be tons of enterprises, and arguably the long tail of consumers, who of course want inference, and theyâre thinking about cost and speed, but they arenât going to want to manage all of that complexity. For them, where weâre at today, theyâre not going to need it. Of course, everyone always wants faster inference â so if thereâs a way for them to get that outcome without dealing with all the complexity, theyâre going to want it.
When Iâm trying to think about this space and ask whether itâs one-size-fits-all GPU or many different chips â where is this going to go in the end â I look to CPUs. When you look at any CPU vendor, or any cloud like Google Cloud, they donât just deploy one CPU, even for customers who want to rent them from their cloud. They have a portfolio: hey, this one has a lot of memory in case youâre running a database; oh, this oneâs actually pretty vanilla and itâs nice and cheap if youâre just running an API server. There are still shapes â this family of chip, that family of chip. I do think thereâs a world where we get to a couple of different shapes of AI accelerators.
Where we are now: we had this training era, then this inference era â those were all on GPUs. Then weâve got this next era, which is a GPU plus a specific decode chip thatâs made a good memory-hierarchy trade-off so you can have high interactivity, as they call it â really fast tokens. But where we are today, those are multi-vendor: Nvidia plus Groq â of course, Nvidia acquired Groq, so theyâre trying to bring that all in-house â or someone else plus Cerebras â AMD plus Cerebras, or Trainium plus Cerebras, or whatever. My thought is that we will ultimately go to those SKUs living inside the same silicon vendor, because it starts to get complicated when your route to market depends on another company â I sell a GPU and they sell a decode-specific thing. But it is working right now. Thereâs definitely demand for it.
David Goldman: It feels like thereâs this inherent tension in the market. You want to buy systems â thatâs what customers are saying, they want someone to do the work of putting all of these things together. But at the same time, they also want the right silicon for the right job. They want specialized things for decode, and they want the right proportion of CPUs for the workload. So itâs not a one-size-fits-all system. Ultimately, do you think you get more and more fragmentation here? Or are customers going to say: âNvidia, please solve this problem for me â buy Groq, buy the next company, buy the next company, keep selling me systems. AMD â buy Talis [sp?], work with Cerebras, figure it out for me.â
Because, to give you the counter side: there are separate companies for CPUs and GPUs. Weâve decided those are separate enough markets that they can have standalone companies. We can figure out the system with CPUs and GPUs from different vendors â thatâs been a solved problem for a long time. So maybe that could be an end state, where you have decode silicon that is a completely different market, and we give it a catchy name like DPU or something. Well, not that one, because itâs already been used.
Austin Lyons: Right, totally. Zooming way out, when you look at the semiconductor industry in the long run, there always seem to be three or four vendors in a certain thing. Whether you look at wafer fab equipment, foundries â itâs complicated, but weâre getting back to maybe having two or three â CPU vendors, GPU vendors. We are of course in an era, as happens whenever thereâs drastic innovation, where a ton of competitors have popped up. So I think it would not be crazy to zoom out and say: in the grand arc, maybe only three or four people will shake out, and therefore thereâll be some sort of consolidation.
But I personally think this isnât as simple as âNvidia buys [that startup], AMD buys them,â even though weâre seeing some of that â and of course there are regulatory things to talk about there. If you look up and down the stack, thereâs actually a lot of competition.
For example, at the neocloud layer: a neocloud might be incentivized to say, hey, I can tell the customer wants the right silicon for the right job, but they also donât want to deal with the complexity. I could buy a bunch of different silicon, deal with all the complexity myself, sell them tokens as a service, and try to give them the speed or the cost theyâre looking for â and differentiate from other neoclouds that way.
So I do think there are still routes to market where startups can come in today and say: hey, I am the best decode solution, you should try me. Or maybe Iâm a prefill solution. And figure out a way to get to customers while letting the end developer not deal with all that complexity. Then you keep pulling on that thread and play it forward: could an AI ASIC startup ever merge with a neocloud? I donât know. Maybe it wonât just be Nvidia or AMD buying all of these companies.
The rise of the neoclouds
David Goldman: You touch on an interesting thing here, because I think neoclouds are really under-discussed when people talk about AI infrastructure. We tend to focus a lot on semiconductor companies and systems companies, and not as much on the cloud layer. And I sort of have a pet theory that itâs because most of the traditional VCs missed out on those as investments, and so we donât like to spend too much time giving credit where we donât get to claim any.
Austin Lyons: Yeah.
David Goldman: But the reality is that if you look at the handful of neoclouds that came up in this first wave, theyâve created a lot more equity value. The top three are worth something like $125 billion in public markets, which is a lot more than what weâve been talking about with some of these chip startups. So why do you think investors had a lot of trouble understanding the first wave of neoclouds and missed out on those investments?
Austin Lyons: Yeah. For listeners, by âneocloudâ we mean a cloud company that started by just renting GPUs. Iâll talk through my hesitation when I first saw the idea of neoclouds, and maybe Iâm a fair proxy for some investors â maybe they thought this way too.
Letâs back up. Whoâs renting from these neoclouds? Well, it turns out a lot of it is hyperscalers, which is ultimately driven by demand from the biggest model labs. So the question is: wait a minute, youâre saying that OpenAI is running on GPUs that Microsoft is renting from some neocloud? Doesnât Microsoft just have their own data centers? And the answer is: of course they do, and theyâre trying to build more, but at the end of the day we are limited by access to power. There are financial reasons to rent versus buy â maybe even go off balance sheet as the capex has increased year over year over year. So there are very legitimate reasons why even someone as sophisticated as Microsoft Azure might say: actually, I want to rent GPUs from someone, and Iâm also building other data centers, but maybe itâs a stopgap.
Looking at that, I said: okay, so youâre going to have a neocloud company â maybe they have access to power, and maybe they have a good way to raise capital against assets like this. Bitcoin miners, for example, historically have had this experience, and maybe theyâre shifting into GPUs. And so I thought: huh, okay. They were doing Bitcoin mining, now GPUs are hot, so theyâre going to do GPUs, but theyâre just going to rent them to Azure. But Azure is also standing up their own data centers. So how is this sustainable? Theyâve got huge customer concentration â literally maybe one customer. But to their credit, thatâs how you get the financing: you say, Iâve got Microsoft whoâs going to rent these from me, and so people will lend against that and believe in that.
Thatâs part of why I missed it as these companies were popping up. Thereâs a legitimate need in the marketplace for people who have access to power, who can get financing, who can manage this, and who can move quickly â to stand up GPUs, run models on them, and rent it out as bare metal or maybe at a higher level of abstraction. And even though they have serious customer concentration risk, so does everyone else in the semiconductor industry right now. Who are Nvidiaâs end customers? Even Nvidia, the biggest and best, is selling a lot of GPUs to a small set of customers.
David Goldman: Yeah, itâs not uncommon to see someone go public with 90% customer concentration these days.
Austin Lyons: Totally. That is the name of the game. Look at any component supplier â even in the interconnect space or the switching space. Look at Credo, who made these active electrical cables, a really awesome invention. Same thing: even when they went public, they were selling to a handful of customers. The name of the game for this era is watching how this unfolds and seeing how some of the people who win one big customer ultimately win a couple more, build out more of a portfolio, and reduce a little bit of that risk. But thatâs just the way the industry works right now.
David Goldman: If you wind the clock back five years, before these neoclouds got big â AWS is an incredible business; Amazon and Microsoft have fortress balance sheets, great relationships with Nvidia and all the semiconductor companies, expertise in setting up data centers, software, customer relationships. It seems to me they could have done this, and certainly most people thought they would, which is why so many missed out on these investments. Was it a strategic decision, where they thought âmaybe this isnât going to be a big enough market, Iâm not sure I want to spend all the money and take the riskâ? Or was there some special sauce in what these neoclouds were able to do â setting up quicker, being more creative in financing, converting old Bitcoin data centers? Were they doing something different from what Microsoft might have done, or was Microsoft ceding market share to them, or some other factor? Not to pick on Microsoft â insert anyone here.
Austin Lyons: Maybe a little bit of everything. At the end of the day, this is risky business, because the investments might be $40 billion this year, $80 billion next year, a hundred billion the year after that. I donât think itâs that the existing players didnât believe the future weâre in now would manifest â I think itâs all about timing. And theyâre probably not incentivized to just sprint out and stand up all these data centers. Itâs a huge investment.
The neoclouds were able to move faster and take on more risk. They can kind of go for broke: all right, cool, letâs convert, letâs buy a bunch of GPUs, letâs get moving. Itâs a bit like the innovatorâs dilemma. There are already three big clouds, they already have all the customers, and I think they were probably looking at each other â AWS and Google and Azure â saying: are we all believing this future is coming and investing in GPUs at the same rate as each other? But that could still not be enough supply to meet demand.
Now, I do think demand went higher faster than everyone expected, especially once we got reasoning models, and now of course the agentic age. So even if they all looked around and said, âwe think if we grow supply like this â itâs a little risky and these numbers feel really big, but we think the demand will be thereâ â demand skyrocketed past that, and it gave neoclouds an opportunity to come in and say: yeah, weâll fill that gap.
David Goldman: I think there was also maybe a difference in interest level. You heard some of the hyperscalers make comments about how these bare-metal GPU instances are low-margin and sort of commodity, and therefore they didnât want to support them as much. Whereas the neoclouds read them as revenue, and therefore good.
Austin Lyons: Yeah, you make a fair point. Anyone whose business was renting CPUs, or selling services on top of CPUs â the cost structure there is much better than GPUs. So you could see CFOs saying: wait a minute, weâre going to spend a ton of money and our margins are going to go down, even if our margin dollars go up. There are still conversations to be had that might make you slow down or hesitate a little bit.
David Goldman: So is this a durable state of affairs? Obviously this is a very fast-moving market, but âfirst mover,â âmore willing to take risk,â âwilling to take on lower marginâ â those are not necessarily durable advantages that will last a decade or more. What do you see playing out with these neoclouds? Do they get acquired by hyperscalers? Do they consolidate into a neo-hyperscaler or something?
Austin Lyons: Yeah, itâs very interesting. Iâm not sure there will always be a need for a hundred neoclouds, but I do think itâs real demand, and it will remain real. I donât think the big three clouds will always meet everyoneâs needs indefinitely. Not only that â thereâs always going to be innovation. Take World Labs, Fei-Fei Liâs company. Theyâre coming out with world models. Whoâs going to make the bet there? What if those donât run well on the exact shape of hardware all these big clouds have invested in? I think there will always be new workloads or new demands popping into existence where a neocloud can pop up quickly and say: I can meet your need, and I can innovate there. That will always exist. There may not be enough cutting-edge frontier demand for a hundred neoclouds to hop on, but I definitely think there will always be a need for these more nimble, smaller GPU and AI ASIC rental companies that can innovate a lot closer to where the model labs and AI-enabled software companies are innovating.
Circular financing
David Goldman: I feel like we canât talk about neoclouds without at least discussing circular financing risk. This is probably the thing I hear most from people who are skeptical about the durability of AI infrastructure. The argument goes: the neoclouds get equity investment from Nvidia in many cases. They use that equity to buy GPUs. They then use those GPUs as collateral to take on debt. A lot of that debt is sometimes backstopped by either their hyperscaler or by Nvidia itself. It all sort of perpetuates, and the revenue from the neocloud buying GPUs goes back to Nvidia, who invests in more neoclouds. People see a circular financing issue, like the vendor financing that happened in the dot-com bubble and was revealed to be an inflator of that bubble. On their last earnings call, Nvidia addressed this directly and said they see it differently. Do you agree with them, or do you think there are some concerns here?
Austin Lyons: You know, I was definitely the type of person where, right away, I said: okay, this is different, this feels funny, I need to dig in and understand it and try to understand both sides. I can definitely see why it looks like circular financing â call it what you will.
But going back to thinking through demand, supply, and the cost of capital: it is a fact that thereâs just insatiable demand, especially now with agentic AI, where the barrier to entry for software development has gone as close to zero as possible. Iâve got a son who made a 70,000-line video game this summer, and he didnât write those lines himself â he used Codex to do it. Thatâs amazing, itâs unreal. And just wait until heâs in high school and college and beyond. Heâs going to use AI so much more intelligently than me. Heâs going to use way more tokens than me. I canât even believe what itâs going to be like in the future. You can look at every industry and see lots of people writing software, doing interesting things they couldnât do before. So the demand is totally real.
The supply is very fixed. At the end of the day, you might look at someone like TSMC â there are only so many wafers that come out, only so much CoWoS capacity. But even as GPUs get built, the question is who has the capital to buy them. Today we might be talking $5â10 million a rack, or more. Who has that kind of money lying around? So there is this cost-of-capital, financing thing that comes into play. If customers are saying âI just want inference, as fast as possible, as soon as possible, please make it happenâ â who in the supply chain has the money to invest in standing up all these data centers and running them? It could be neoclouds. Okay â do they have access to the capital they need to make tens of billions of dollars of investment, or even a few billion? A lot of these might be early companies, or former Bitcoin miners. They might have some access to capital, but not enough.
If youâre Nvidia, sitting there with all this money as the worldâs biggest company, and youâre saying demand is incredible and supply is what it is â but itâs not just building the GPUs, itâs getting them powered up, financed, stood up â of course it makes sense from their perspective to help get this stood up. The banks arenât so sure they want to lend to the neoclouds, but if Microsoft or Google or AWS says âIâll be the offtaker â donât think about the neocloud when youâre lending, think about me,â and if Nvidia can also come in and say âwe want to help make this happen, can we put our brand behind it, backstop it, whateverâ â I can see why Nvidia would want to do that. Now, does that mean they also benefit from it? Of course, totally. Itâs customers, and maybe even revenue sharing â a new source of revenue for Nvidia.
David Goldman: Well, thatâs capitalism. Theyâre not going to do it if they donât benefit.
Austin Lyons: Exactly. But as sort of a techno-optimist: if itâs going to feel funny, but there are ways to get more compute stood up faster so that more people around the world can do the awesome things theyâre trying to do, then Iâd say, all right, I can get behind that.
David Goldman: I think it ultimately boils down to differences of opinion on the durability of the cash flow that comes from these assets. If you went out and said âIâm going to build a toll road,â you can get a lot of financing for that. You donât have to put in a lot of equity, and you can get a lot of debt, because people know this road is going to have X number of cars, we know the traffic patterns, youâll collect this amount of money. Itâs very safe. You can raise lots of debt at very low rates for projects like that, even if youâre a new company. Obviously thereâs a big difference between a toll road and a GPU-based AI factory, as some people are calling them. But from Nvidiaâs perspective, and from some of the hyperscalersâ perspectives, these are fairly safe assets that will be completely paid back in two or three years, and then thereâs a stream of cash flows coming out of them. Even if demand goes down a little or doesnât grow at the same rate, youâll still get value out of them, and theyâre not going to depreciate super quickly. Nvidia has that view and is willing to put their balance sheet behind it, the hyperscalers feel the same way â and not everyone else has the same view. Thatâs where the rubber is going to meet the road.
The flip side, though, is that you have the existing neocloud set who are now very embroiled with Nvidia â theyâre in lockstep, they rely on them for financing. To the extent that customers are going to demand more and different silicon, that creates an opportunity for new neoclouds who can figure out how to make it all work together and choose the right silicon for what customers want. Do you think thatâs effective counter-positioning? Will we see another wave of neoclouds like the first time, or will the existing set â the CoreWeaves, the Nebiuses, companies like that â figure this out and just start using Nvidia plus Groq, or AMD plus whoever?
Austin Lyons: Right. I think both will exist, but you make a good point about the trade-off. Itâs kind of like golden handcuffs. The trade-off for a neocloud is: hey, theyâre backstopped by Nvidia, maybe they got financing, and theyâre obviously getting allocation for GPUs. So those particular neoclouds might feel like, if thereâs different silicon out there thatâs very competitive, even for a subset of workloads, we may not feel like we can go buy it and offer it â because what if we donât get as much allocation in the future? Donât bite the hand that feeds you. So I do think there will be opportunities for other neoclouds to come in and say: weâve got a bunch of different silicon, and maybe we can abstract it and run your workloads across it so you donât need to worry about it. There will be opportunities for someone to come in and counter-position.
Four conditions for the next trillion-dollar chip company
David Goldman: In the last couple of years there have really been two big outcomes â weâve touched on both â in the semiconductor space: Groq, which sold to Nvidia, and Cerebras, which went public. But neither one really won a hyperscaler before they were able to do this, and in the case of Groq, they sold to Nvidia. You wrote an article about what you think the conditions are for the next trillion-dollar chip company, and you had four conditions: capable of running trillion-plus-parameter models; rack-scale chips, which weâve touched on; beating an incumbent on a KPI; and landing a frontier anchor. Points one and four â running trillion-parameter models and landing a frontier anchor â are correlated around the idea that the frontier matters most. Why do you think thatâs the case, and a prerequisite to being the next big breakout company in silicon?
Austin Lyons: Very interesting question. Ultimately, today, itâs the best models â especially if you can run them at fast enough speeds â where my belief is the outsized value will accrue. Yes, there are lots of use cases where you can use older models, smaller models, and you donât have to run them as fast. I think that pie will continue to expand. I just donât think people will pay a premium for it.
So if youâre an AI accelerator company and youâre trying to put as much muscle behind a few arrows as possible, youâd want to compete at the frontier. Today itâs software developers saying: yes, I will pay â not $200 a month; weâll pay for tokens, thousands of dollars a month, tens of thousands a month â if we can get it fast, if we can get Claude Fable, for example, or the latest OpenAI model. And of course the frontier will always keep getting better.
It also feels like the frontier is where there will be the least competition, because GPUs, for example, canât get there today on speed, and we know that. If youâre aiming at a 70-billion-parameter Llama 3 and going after all those workloads that are valuable but donât need the highest intelligence, thereâs going to be a lot of competition there too â and it could literally be old Hoppers or old Amperes from Nvidia.
When you look at that Pareto frontier curve we always see â Iâll describe it for people who are just listening: the x-axis is interactivity, which is how fast the tokens come, tokens per second per user; the y-axis is throughput. The slower you go, the more tokens you can generate concurrently; but way out there on the far right, going fast, even if you canât serve as many users â thatâs where the value is accruing today. And itâs hard to see a world where that changes.
David Goldman: That graph gets shown a lot, particularly when Jensen or people from Nvidia talk about what theyâre going to be able to do with Groq plus Nvidia. But is tokens per second per user â that interactivity KPI â still the right one for startups to think about? Or are there changing needs because of power constraints, cost constraints, new workloads like agentic coding? Is it still all about speed?
Austin Lyons: Itâs not all about speed. In my opinion, the way to compare people is at a fixed interactivity, for a given unit of power. We are power-constrained. Ultimately, if a neocloud gets access to 100 megawatts, theyâre going to have to ask themselves: how can I get as much revenue as possible out of this 100 megawatts? They might say, I want to bet on allocating some of my megawatts to really fast tokens, because I think we can charge more for them. So if youâre going way to the right on the interactivity curve â letâs say theyâre aiming for 800 tokens per second or higher, because they feel they can charge a premium for that, and theyâve only got so many megawatts â then theyâre going to ask: how many concurrent users can I serve? What is my token throughput? So I think itâs token throughput at a fixed interactivity, normalized by power. But again, not every workload needs that.
David Goldman: I didnât hear you say the word âcost.â One of the ways you get better interactivity is by using more expensive memory, using SRAM. So how much does cost play into that equation?
Austin Lyons: Cost plays into it. If youâre in that use case â youâre a neocloud, youâve got 100 megawatts, youâre trying to generate as many tokens at a fixed interactivity as you can â you also have a fixed budget to spend on compute. If you buy an Nvidia Vera Rubin rack, the latest and greatest, plus nine accompanying Groq LPU racks, you might get really high on that interactivity, and it might be pretty good power-normalized â but you might have spent half your budget, or all of your budget, right there. So cost absolutely comes into play. When Iâm thinking about the user experience, Iâm thinking about interactivity and how many people can be served. But if youâre a neocloud or any buyer of compute, youâre definitely thinking about cost. And cost could mean: I can get the same performance out of two racks from this vendor versus eleven racks from that one. Or, for the same fixed cost, what if I could get eleven racks from this new competitor â and therefore five or ten times more tokens at that interactivity?
Enterprise and on-prem AI
David Goldman: This kind of buyer thinking is very emblematic of a cloud or hyperscaler with a huge instance theyâre trying to spread over lots of users. I personally always struggle with holding two ideas in my head at the same time. On the one hand, all of the initial demand and value has been going to the frontier labs, served by a combination of hyperscalers and neoclouds. On the other hand, we and many other people believe AI is going to be something as big as the internet. Itâs going to diffuse into businesses all over the world. Every company is going to have some AI element, in the same way every company has a website now â there are no more dot-com versions of companies; everyoneâs got a website, everyoneâs got an app. Soon everyone will have some AI element in their business. And in todayâs internet world, more workloads exist on-prem than in the cloud. If Iâm a coffee shop, I might have a server in my coffee shop; Iâm probably not going to have an AWS account. So ultimately, does enterprise actually become the big market here instead of cloud?
Austin Lyons: That is a very good question. I think theyâre both going to be massive. Letâs talk enterprise, because I donât think people appreciate that enough. Even Nvidia is trying to get ahead of it â they changed their reporting and their business units so that for data center itâs essentially hyperscaler and non-hyperscaler. It gets a little fuzzy, but theyâre saying that, by the way, the non-hyperscaler segment is growing faster than the hyperscaler one, and right now the revenue is pretty close on both â a little fuzzy because they put neoclouds in there.
So the question is: what workloads are going to go to the enterprise, and why? Youâre totally right that there is going to be diffusion of generative AI across every industry, and I definitely donât think weâre there yet. Marc Andreessen said fifteen years ago that software is eating the world â every company is going to be a software company. And to some extent that was right: logistics, manufacturing, healthcare â theyâre all using software, even if itâs just internal tools. I think weâre going to a world where agentic AI eats the world, and every company is going to be an agentic AI company. Like I said, pointing at my son: fast-forward fifteen years, and yes, theyâre all going to be agentic.
Okay â so if agentic AI is core to how businesses run, are they all going to have hundreds of millions of dollars to spend on tokens every year? Totally not. There are going to be all sorts of reasons for companies to want to deploy workloads on premises. One: cost. Two: owning your own data, and figuring out how you even differentiate in a world like that. It wonât be all-or-nothing â go to the cloud for the frontier workloads, and do as much as you can of the older, smaller models on premises.
By the way, you could also read Nvidiaâs hyperscaler/non-hyperscaler split as a proxy for frontier closed models versus open-source models, because if youâre running enterprise AI locally today â on premises, on a server, on your desktop â itâs got to be an open-source model. Thatâs a little bit why weâre in the world weâre in today: if you want the best model, you have to go to the cloud. And for everything else â do you want to pay for tokens, or for token generators? I think a lot of people, if they can afford it, would rather own token generators.
But fast-forward a little: what happens if open-source frontier models can keep up, or be good enough? I think there will continue to be a rise in the amount of workloads run on premises. There are all sorts of reasons. Of course you can point to regulated industries that will have to run on premises. But even if Iâm not regulated: maybe I think the labeled data my humans generate is valuable. Say Iâm an insurance company. We have all this agentic stuff doing claims processing, and my humans are going in and correcting it â that was good, that was good, that was wrong. Letâs keep that data internally and fine-tune our own model so it gets it right in the future. Maybe I want to run that locally because Iâm in charge of the model, Iâm in charge of the data, I keep it, itâs all my intellectual property. And maybe I see that as how I differentiate in the future â Iâve got better agents than my insurance competitor.
Can you do all this stuff in the cloud and feel like itâs secure? You totally can. But at the end of the day, when weâre talking about diffusion, we want every engineer at every company to be able to tinker with it, touch it, play with it, use it themselves. Sometimes when stuffâs in the cloud, you get the convenience, but you lose the ability to get under the hood â whether itâs a closed model or even an open model in the cloud.
David Goldman: Itâs funny you say that â I almost feel the opposite way about enterprise. To me it seems gated a little bit by software. So many of these companies havenât moved workloads even to the cloud, for data sovereignty, regulatory, privacy, IP protection reasons. Theyâre very concerned about stuff leaving their IT premises, and they would gladly do more in AI, but they donât have engineers in-house who know how to post-train a model or tinker with this stuff. Those people are expensive, and they donât want to hire them. So they want to do it on-prem in the enterprise, but no oneâs quite figured out how to help them do that yet. That, to me, feels like the missing piece that would unlock a lot of enterprise hardware sales.
Austin Lyons: I definitely agree with you â thatâs where we are today. Back up eight years: everyone wanted to be a software company, but they didnât have software engineers. I live in Iowa, so if you were a software engineer there and willing to work in insurance or ag or retail, you were a rock star. You could walk in and theyâd say: yes, thank you, we need you, we didnât have this capability before. Fast-forward to now, and anyone can vibe-code, which is actually pretty awesome, because now the domain experts â the person in insurance who knows insurance really well â can actually build the solution they want.
I think weâre now where we were eight years ago, but for fine-tuning: Iâm an insurance expert and I can vibe-code a thing, and Iâve got some software people here, but none of us know how to fine-tune yet. Thatâs the education piece. If my children had to go to college today and pick a major, Iâd say: pick anything, plus machine learning, and learn how to fine-tune, because then you can understand a domain and also understand how to actually apply AI. So I think there will probably be a rise of AI engineers, if you will. Maybe agents will do this for you and bring that cost down to zero faster than agents brought software down to zero â which took, you know, forty years. But it is a pain point today that companies donât have generative AI familiarity yet. I donât think that pain point will be there forever. Maybe itâs five years, maybe more, I donât know. But I donât think it will always be a blocker. If every company became a software company with software-literate people on staff, I think eventually everyone will have LLM-fine-tuning-literate people on staff too.
David Goldman: So when that day comes, is this just a huge unlock for Nvidia, and they get that much more revenue? Or do you think anyone else has a chance at that market?
Austin Lyons: Thatâs a great question. What happened to IBM, you know? There are giants, and theyâre first, and they ride a huge wave. And then, to all the points weâve talked about â zooming out, there are three or four winners, and people want competition. The more people can tinker, the more this diffuses, the more opportunity there is. One company cannot meet everyoneâs needs at the right price point, at the right speed. They can meet lots of peopleâs needs, but there will always be people trying to do some interesting bespoke thing who say the off-the-shelf stuff uses too much power â I know Iâve got this crazy setup, but I canât do 130 kilowatts, I can only do 50. There will always be emerging workloads where the off-the-shelf stuff just doesnât fit.
And the tough part is, when youâre Nvidia and youâve got these huge hyperscalers, youâre not necessarily incentivized to go find those little people. Youâre not interested in picking up pennies; youâre interested in picking up billion-dollar bills. So I think Nvidia will be totally fine â they have great solutions, and theyâre always going to have customers coming to them for the latest and greatest, deployed quickly. But I do think there will continue to be new opportunities popping up, especially as this diffuses, where people can compete.
Clean-sheet silicon for LLMs
David Goldman: Circling back to this idea of the next trillion-dollar company. Thereâs an explosion of opportunities in different workloads, and also, as weâre hearing, different markets. Does that mean thereâs an opportunity for a trillion-dollar company? Or are we going to get $500 billion companies? AMD is still not even a trillion-dollar company, and theyâve been around for a really long time and have a lot of pieces of this puzzle.
Austin Lyons: I just think about the size of the market. Hereâs the deal, and part of why I came to that conclusion. GPUs obviously have a history in graphics and doing things in parallel, and that has been shifting toward AI-centric. Nvidia is saying: yes, these data center GPUs â youâre not going to run Doom on them. They used to support FP64, and they still do, but weâre going to spend all of our transistors, as we do a node shrink, on FP4 and FP8, the lower precision that AI models really want. But at the same time, it was still a general-purpose GPU.
Thatâs why I said we went from training with GPUs, to inference with GPUs, to this next era where LLMs are the workload â and we still havenât had silicon designed specifically for LLMs. We had GPUs that morphed from their early roots to fit the shape of what weâre doing. Letâs take a GPU and slap on this SRAM thing. But no one yet has really brought a chip to market that was designed specifically for LLMs. So the question is: what if you can be the first one that can stand up a gigawattâs worth? Which is not simple.
David Goldman: Well, to some degree, the Nvidia GPUs of the last couple of cycles are chips designed specifically for LLMs. Itâs not like you can take Grace Blackwell and play a video game on it easily. Itâs highly specialized for this type of workload. And in particular, when you get into these combination GPU-LPU systems, or the AMD Helios, these are really designed specifically for LLM workloads.
Austin Lyons: They are morphed specifically for LLM workloads. They have legacy ways of doing the networking, legacy ways of thinking about the memory hierarchy. I definitely agree that they are iterating toward what is best for the inference workloads they serve. But the question is: what if you started with a blank sheet?
We actually saw this from OpenAI with their JalapeĂąo chip, which they presented at Hot Chips recently. They said: we started with a blank design, and weâre thinking very differently about it, making very different architectural decisions. They said, instead of shipping all this KV cache around and having all this shared memory and all this contention â the data is never in the right place when we want it, and our compute is always sitting around â what if every accelerator had its own little HBM slice? What if we mapped the workload and rethought it so we donât have all this contention and shipping data around?
Tensordyne is another example. They said: should it be matrix multiplication, or could we do log math? That would turn multiplies into adds, which are really fast in silicon. So I actually do think there are architectural knobs.
Nvidiaâs next GPU still has to have backward support for the software â not exactly, but to some extent; they want to support all the workloads, so if it ran on Hopper they want it to mostly run on Blackwell. But what if you could start with a clean sheet and make different architectural decisions â about the way you do the compute, the memory hierarchy, the way you network it up?
Etched said: letâs do low-voltage inference. What would happen if we ran this at lower voltage, instead of putting in a lot of oomph so it can go really fast? If power is fixed, maybe there are benefits, even with trade-offs â it doesnât run as fast, but itâs significantly lower power.
So I do think thereâs opportunity to make clean-sheet designs, make different architectural decisions, and therefore unlock that KPI I talked about. What if you could get 10x more tokens, at that fixed interactivity, for that particular model, out of your 100 megawatts, than you could buying something off the shelf from Nvidia or AMD?
David Goldman: You hit on another interesting tension. On the one hand, you have someone like OpenAI, who is a large customer of Nvidia but is also now designing their own chip. When you hear them talk about how they did that design, they claim it was very AI-optimized â that they were able to do it much quicker because of AI acceleration. And then you also have companies like Tensordyne, who have an entirely new paradigm, a whole new way of approaching this problem, that a customer probably wouldnât have thought of on their own, because theyâre more focused on their specific workload than on new ideas from first principles. How do you think about the trade-off between custom silicon companies using AI to create something specific for what they need â maybe working with a Broadcom or Marvell to help them finish and get it into production â versus these companies with entirely new approaches?
Austin Lyons: Itâs so interesting to think about OpenAI buying compute from the big vendors, mostly GPUs, while also building their own silicon â with the advantage of their chip designers working hand-in-hand with their software team, co-designing for specific workloads. Off-the-shelf silicon vendors are not inventing in a vacuum; theyâre talking with their biggest customers, asking where their roadmaps are going and how to make sure their silicon meets those needs. But thatâs different from OpenAI internally having their ML team and their chip team working closely together and co-designing.
That said, theyâre going to land on a particular set of trade-offs. Everything in engineering is about trade-offs. Do you want more HBM or more SRAM? Itâs going to cost you something either way. Do you want more die size, or do you stack it higher and take the thermal trade-offs? The merchant vendors made a particular set of trade-offs â and they need to sell their chips to as many customers as possible, even if itâs only a handful these days. The internal teams will make their own set of trade-offs, given what they know about the workloads theyâre running.
But to think those two sets of trade-offs are all youâll ever need? I think thereâs opportunity for someone like a Tensordyne to say: what if there are these totally crazy trade-offs, and we take the risk of doing the R&D on them? The log math stuff â surely itâs crossed the minds of people at the merchant silicon vendors or the internal XPU teams; theyâve seen a paper. But they may not be incentivized to take the risk. If youâre OpenAIâs XPU team making your first chip, are you going to play around with log math? Or are you going to say: no, no, letâs make these other interesting but proven decisions, like HBM slices that have been used elsewhere in industry?
David Goldman: Yeah. People forget there are people involved in these decisions who have career risk.
Austin Lyons: Exactly.
David Goldman: If you can make a decision thatâs high-probability and still works, but maybe isnât the 10x â thatâs probably better for you if youâre at a big company, particularly delivering your first version of something. You donât want to screw up.
Austin Lyons: And itâs expensive, too. Youâre going to tape it out, and you might be all-in a billion dollars or something. You donât want to get that wrong, or have it get cancelled before you can stand it up. So yes, thereâs this whole human side, these incentives. And again, that continues to create opportunities for startups to say: weâre going to take that risk. Or, in Tensordyneâs case: weâve been taking that risk â we were trying the log math in a different market, and now weâre ready to bring it to this one. Could an OpenAI say âthatâs super interesting, letâs try a couple racksâ? Absolutely â these are very sophisticated buyers, and theyâre always watching what else is out there, because they completely understand that designs made at a particular point in time may not fit exactly what they need a couple of years from now.
AI-designed chips and the falling barrier to custom silicon
David Goldman: Are more companies looking at custom chips? OpenAI has a lot of money and a lot of really talented engineers â even without AI, they probably could have designed their own chip. But as AI makes designing a chip easier â itâs not at the level of your son creating a video game with Codex, but you talk to teams and theyâre seeing lots of improvements â thereâs some floor to how cheap it can get, because you ultimately need to tape out and do things in the physical world. Do you think weâll see a big proliferation of companies that never would have tried custom silicon giving it a shot?
Austin Lyons: I definitely think so, and you can already look at examples where itâs happening. Why would you design your own silicon? If you know your workload really well, what you bought off the shelf â where design decisions were already made â might not map to your workload perfectly. That might be okay at first. But take Rivian, for example. They went through a couple of different merchant silicon vendors, mapping their workload to the parts, and it was fine enough. But if youâre running an electric vehicle and trying to do autonomous driving, you have very specific needs. You want to use as little power as possible, because otherwise youâre taking battery away from the customer being able to drive another couple of miles. On the other hand, you need to run as fast as possible, because if it takes too long to make a decision, thatâs another 20 meters of braking distance while you were thinking. Now all of a sudden theyâve got lidar and cameras and all this data flowing around that they need to do inference on as fast as possible. And by the way, it used to be convolutional neural networks; now theyâre doing end-to-end VLA â vision-language-action â models. The workload has been changing, and in this world of real-time inference on big models with power under control, plus a particular interconnect bandwidth and memory capacity and memory bandwidth â theyâre saying: man, this stuff off the shelf just doesnât fit our needs and our cost profile.
So the answer could be: design your own chip. Well, thereâs a cost to that. You need engineers familiar with front-end design, back-end â maybe you can partner for testing â and at the end of the day, that might take you three years of development. Youâre paying those engineers for years, thereâs a roadmap, itâs a very big investment, and then thereâs the cost to tape out and actually get it built. But what if, with the help of AI, it still takes a hundred people but instead of three years it takes one? Maybe your cost is cut by two-thirds. Where before, even though the performance and the headroom would let you do interesting things, your CFO was saying âwe just canât add another $2,000 to the bill of materialsâ â maybe now you come back and say itâs only going to be $700 on the BOM, and they say: okay, thatâs really interesting, we think we can hack it.
So I think adding AI to chip design â speeding up time to market, doing more with the same number of people â will ultimately be net good, and will reduce that cost barrier to entry. Or it could be a talent thing: how do I go find a hundred people? Maybe you only need fifty. I just think it will reduce barriers to entry, and weâll see all sorts of cases where people who never thought about making their own chip will say: thatâs actually something we could do.
David Goldman: Itâs an interesting dynamic, because the same speed-up available to these companies is also available to the merchant silicon teams â who are probably even better positioned to use these tools, because an experienced engineer who understands the trade-offs is going to use the tool better than someone approaching their first chip design. So maybe they can start proliferating their number of SKUs and serve more customers with more semi-custom things within what is somewhat merchant. I just think itâs really interesting. I donât know how itâs going to work.
Austin Lyons: Yeah. I would hope that merchant silicon companies â the thing is, when youâre a merchant silicon company making a particular product, there has to be a big enough market to capture enough customers for it to be worth your time and investment. Thereâs probably a point where the ROI didnât make sense: the marketâs only this big, the SKU costs this much to develop â not worth it. But if the cost can come down by half, maybe it clears the internal rate of return you needed. So yes, Iâd expect big companies to be able to create more innovations as well.
David Goldman: Awesome. Well, I donât think anyone really knows how itâs going to play out. This is such an exciting time. Thank you so much â this has been an incredible conversation. Iâve really enjoyed it.
Austin Lyons: Yes, thank you. This was fun. Letâs do it again.
David Goldman: Absolutely.