In the pandemic-era supercycle, TSMC spent 53% of revenue on CapEx. In the AI supercycle, with roughly $100 billion a year of operating cash flow, they’re spending only 34-36%:

Matching the 2021 playbook today would mean another $30 billion a year. What to make of that?

Maybe TSMC got burned by 2021-2023 and ~35% is the preferred comfort level for management. Or it could be a lack of faith in the long-term demand?

TSMC has arguably the best ability to ascertain true demand given the breadth of customers, and talking to customers’ customers... So why aren’t they investing more? What did Wei see?! ... jk

Let’s pull out a few interesting questions from TSMC’s great quarter.

What’s in this piece

• N2 has arrived

• Gross margin peaks, gross profit keeps growing

• 🔐 Digging into the CapEx gap

• 🔐 What happens if they underinvest

• 🔐 Pricing… no talk, but it’s in the margins

• 🔐 What to watch going forward

N2 has arrived!

N2 debuted at 3% of wafer revenue this quarter:

Only 3%? Look at that baby little red bar in 2Q26... But interestingly every node debuts at about a billion dollars of revenue:

Mix share shrinks because TSMC’s revenue keeps growing.

Given that wafer prices rise every generation, flat debut dollars implies shrinking debut volume (wafer prices are rough, just for illustration):

Volume shrinks because each new node has more layers and longer cycle times, so the same capacity ships fewer wafers in its first ninety days.

So N2’s debut is the fewest, most expensive wafers TSMC has ever sold. Customers pay the highest price per wafer ever, and these wafers cost TSMC the most to make, with yields still climbing and a new fab’s depreciation spread over small volume.

Interestingly, N5 is still the biggest node in the portfolio (33%), even though it’s six years old:

Why is such an old node still so dominant? AI. The answer is always AI. 😂

N5 is the AI node. Chips ship on whatever node was mature when they were designed two or three years earlier, and today’s AI fleet (Blackwell on N4P, Google’s TPUs, Trainium, most custom XPUs) was designed in 2022-24, when N5 was the obvious choice for big dies needing proven yield.

It’s interesting how long each node family hangs on:

Of course, TSMC’s “5nm” node isn’t one process, it’s a node family. N5, then N5P, N4, N4P, N4X. So what’s six years old and 33% is the family, not the original process.

On this earnings call, C.C. said A14 will be “an even larger and long-lasting node for TSMC than N2, just like a 2-nanometer technology is a larger and longer-lasting node than 3-nanometer”. Each node is a decade-long product line, each bigger than the last.

Rubin-class accelerators move to N3 through 2027, hence N3 getting three new fabs. TSMC even approved three greenfield fabs for N3, a four-year-old node. One in Taiwan, one in Arizona, one in Japan. New fabs for kind of old nodes.

The longer a node family hangs on, the better the margins. N3 is “very tight,” its gross margin set to “cross over to the corporate average in second half 2026”, and its 2022 tools depreciate off in 2027 (equipment runs on a five-year schedule). Sold out, above average, depreciation-free.

Given that fabs are a ten-year+ asset, today’s CapEx is TSMC’s opinion on demand for the next decade...

Gross margin peaks, gross profit keeps growing

Gross margin printed 67.7%, a fifth straight record. But Q3 is guided down to 66.0%.

Reminds me of an MBA professor that I had, who told our class you bank profit dollars, not margin percentages. i.e. don’t sweat the margins too much

And the dollars keep climbing. Gross profit was roughly $27.2 billion this quarter; the Q3 guide implies about $29.8 billion. Annualized, that’s about $120 billion a year of gross profit.

Why the margin decline? N2 ramp. TSMC guided Q3 margin down “primarily as we expect the steep ramp-up of our 2-nanometer technology to dilute our gross margin by about 3 to 4 percentage points”. That’s worse than the 2 to 3 points guided in April, because the ramp got steeper. But steep ramp and more dilution sooner means more premium wafers sooner! So it’s fine.

The 67.7% gross margin might be the high-water mark until roughly 2028. Through 2027 there’s more margin drag as N2 dilutes for about eight quarters, A16 ramps behind it, and Arizona Fab 2 arrives 2H27. But if Q3 prints margins above 66.5%, well then, it would mean N2 pricing is covering its own ramp. One way to get pricing hints...

The last margin peak, 62.2% in 4Q22, was largely an FX bump at a cyclical top, and it stood for eleven quarters because demand then collapsed and margins fell. This peak is different! It comes from loading a new node as fast as physically possible while demand is spiking. So we’ll see another little mountaintop on the margins chart, but for an opposite and much better cause.

What paid subscribers get in the rest of this piece

First we’ll digging into the CapEx gap to figure out how much they could invest but aren’t. Then we’ll talk through the implications of TSMC underinvesting.

Also, pricing… TSMC doesn’t credit it to margins verbally, but it’s there. We dig in.

And what to watch going forward.

Thoughts, numbers and charts. Carry on:

Read more

Things we cover:

• Inventing the active optical cable with Mellanox

• Why hyperscale AI’s error-free requirement changed the game

• GaAs (unconstrained) vs. InP (substrate-constrained)

• The three 1.6T flavors: 8×200, 16×100 LPO, 32×50 NRZ

• The roadmap to 3.2T and 12.8T via bi-di and 2-D arrays

• The fabless model and WIN Semiconductors capacity

This podcast is lightly edited for clarity.

Subscribe now

From HP Labs to the First 10 Gigabit Ethernet

Austin: Hello everyone. Today we have a special guest, Al Yuen, CEO of PicoJool. PicoJool is an optical connectivity company and we’ll get into all the interesting details. But first I wanted to introduce you guys to Al. So Al, tell us about you and your background — I know you’ve been in the industry for a long time.

Al: Yes. So after grad school at UC Santa Barbara — where a lot of the photonics folks have originated — I went into HP, HP Labs, where we focused on photonics research. And then around ‘99 I left HP and started my first company, called Alvesta, and we created the world’s first 10 gigabit Ethernet. At the time, 10 gigabit — which is 10 billion bits per second — we were actually trying to figure out applications, how people would use this in ‘99. We would make up things like, people will want to stream video someday in all the rooms in their house. But today, of course, we’re doing 1600 gigabit, or 1.6 terabits. So since then I’ve gone to various companies. I ran a division for Coherent, then started some other solar and clean-tech companies, and finally ended up at Lumentum in my last gig. And about two years ago, through Playground Global — which is our funder — we started PicoJool. So far so good on what we’re doing, which is basically in the interconnect space, and we’ll talk more about that today.

Austin: Awesome. Wow, what a great background. I love that you guys invented early 10 gigabit Ethernet and then had to create ideas to sell people — to convince people that yes, this is useful, people will want to use it. That’s awesome. Now, remind me — you also helped invent the active optical cable. Is that right?

Al: Sure. Very interesting story. Back in Alvesta, my first startup — another small startup at the time called Mellanox, which of course now is inside Nvidia and created this whole hyperscale and the whole InfiniBand ecosystem — they approached us. They had these very bulky copper cables, even back then, 25 years ago. They said, the copper cables are very bulky, they could only reach tens of meters — now it’s even shorter. We’d like an optical option, but we don’t really want to commit to a full optical solution. So could you put the optics inside the connector? And we said, why not? So we took that same transceiver that’s typically on a board and literally embedded it right into the connector. Then we thought, well, that’s not very efficient and clean — so we embedded the whole thing inside; the optics went inside the connector. And this is the world’s first demo of an active optical cable, meaning electrical to electrical: electrical comes in, electrical goes out, but inside there’s the electrical-to-optical transition. Electrons come in, photons carry the information, and electrons go back out at point B. So that’s how the whole active optical cable concept came through Mellanox. And since then, for the last 25 years, the AOC has been the standard workhorse in many, many data centers.

The Engineer’s Mindset and Why Optics Must Exit the Rack

Austin: Amazing. So having invented that and then watched it become widely adopted — produced in the millions of cables — how has that impacted the way you think about what’s possible as an entrepreneur in this space?

Al: I like to think of ourselves more as engineers. There’s a difference between scientists or researchers — what people call R&D — and engineering, or product development. I always tell people I’m more of an engineer. Engineers solve problems and want to create products that meet specifications. If you need a Prius, you certainly don’t design a Ferrari — that’s overkill. The product fits the need: spec, cost, reliability, and in our case reach, or amount of power. So what we do is really look at solving the specific problem. Today, copper has shrunk to about 3 to 4 meters of reach at 200 gigabits per second per lane. And therefore they can’t get the information off each rack. Racks and racks of GPUs, CPUs — lots of compute power — and these racks are getting very hot because they’re packing more and more GPUs per rack, because they can’t exit the rack. To exit the rack, you need an optical solution. There are many technologies, and VCSEL-based active optical cables are one of them — that’s what we’re focused on. Low cost, highly reliable — going way back to our roots, replacing a copper cable with an active optical cable. So the problem really hasn’t changed; it’s just that the speed and aggregate bandwidth is now 1600 times more than the old 1 gigabit Ethernet. It’s exciting. It’s been a long journey, but we’re still at it, and we see a long future for VCSEL-based technology going forward.

Austin: Nice. So you have this history of thinking like an engineer — what is the problem right ahead of us in the industry? Not “we need to invent new physics,” but how can we be thoughtful about it. Same cable form factor in the AOC case, electrons in, electrons out, but we could use optics here — could we put the transceiver on the end of the cable? Thinking very pragmatically. And now, fast forward 25 years, in 2024 you started PicoJool. It sounds like you’re tackling the problem again of communicating lots of information — this time GPUs talking to each other, rack to rack, where it’s such high bandwidth that copper is shrinking to 3 meters or so. So you’re again thinking about how to use active optical cables, but this time with VCSELs. So tell us — why VCSELs, and what about the problem made you want to start PicoJool and actually get in the game?

Why VCSELs — 25 Years of Shipping in the Millions

Al: Great question. VCSELs, for one thing, have been around since ‘96 — a technology that’s been in product, in the field, in data centers since 1996, starting with the first gigabit Ethernet. I’m more of a historian today: here’s a 1 gigabit Ethernet that HP created, and at the time Honeywell — very early companies that did 1 gigabit. And from 1 gigabit all the way to today — 1600 gigabit, what they call 1.6 terabits — it’s all been VCSEL-based. The practicality of these solutions has to meet capacity, demand, cost — all of it. If you’re trying to get into a hyperscaler today, you need to meet the whole checklist. You can’t say, I meet everything but it’s three times the cost of your target; or everything’s great but I can’t get the reach. They want all of it. And right now the only thing that meets it is copper — mainly driven by cost. Cost for copper is very minimal. Even active copper, where you have some signal integrity and signal processing built in, like an AEC — active electrical cable — the cost is still quite minimal compared to more elegant, longer-reach technologies.

Now, what’s important in a data center: when you say “fiber optic communication,” everybody hears it and goes, oh, isn’t that the over-the-ocean stuff, the transatlantic subsea cables? Absolutely — there are many flavors of optical communication; it’s a very big umbrella. What we’re talking about today for data centers is a very short reach — today they call it scale-up. It’s basically a row, typically 25 to 30 meters, or in English terms 75 feet. Very short — within your house, from one end to the other, or potentially even shorter. And in that short reach, the shorter the reach, typically the higher the volume. In the data center you’re connecting all these GPUs, CPUs, and ASICs together — millions of interconnections, miles of fiber inside a data center that may be a football-field length. But when you leave that data center and go between cities, buildings, countries, you have fewer fibers but need longer distance. So back to the data center: you have to have millions-per-month capacity to meet the volume demand.

That’s one of the boxes — you’ve got to check off all the boxes. If you demonstrate one or two racks, or go to a show and demonstrate just the technology — technology, science, R&D show the capability — that’s a demonstration. But to ship in volume, millions per month, you need this whole ecosystem: the connector, the transceiver, the sockets — everything has to be millions per month. And any one of those bill-of-material parts, if it’s missing, then you can’t ship in millions per month. That’s what’s happening with a lot of these new technologies that are fantastic from a demonstration and capability standpoint going forward. But VCSEL’s domination is that for the last 25 years we’ve been shipping in the millions per month. So there’s no invention of a technology, capacity, or foundries needed — we already have that. So we design the latest VCSEL — today it’s 200 gigabit, which we just announced — put it into the whole ecosystem, they package it together, and voila, we can build millions per month without waiting for machines, or even buildings, to be built up, then new machines, then new processes. All of that has existed for the last 25 years to service this short-reach data center application.

Austin: Okay, let me reflect it back to you. The problem you’re solving is scale-up with optical interconnects. But the key insight — with your pragmatic hat on — is: how can we use technology and a supply chain that already exists and can already ship millions of cables, components, and parts per month? That’s one reason VCSELs are so attractive — they’re not a new technology, the supply chain is not new. So even though we hear about all these other interesting things — the Broadcoms, Coherents, Lumentums; silicon photonics, EMLs, micro-LEDs — you’re saying, hey, don’t rule out VCSELs. Even though they’re not new, there are advantages. But if VCSELs have been around so long, why aren’t other folks trying to take them to 200 gig per lane, times eight lanes, 1.6T and beyond?

The Hyperscale Shift to Error-Free

Al: Yeah, like you said, there are multiple technologies — silicon photonics, EML, even micro-LEDs — all vying for this 200 gigabit electrical signal coming in. From the ASIC, GPU, or CPU you have 200G per lane of electrical signal. When you go to the electrical-to-photonic (E-to-O) transition, it can go to different lanes through a different IC that may be a gearbox. You don’t necessarily have to run at 200G straight through — which we can — and that’s the most elegant: straight through without an intermediary gearbox saves power, cost, and latency. So 200 gigabit would be ideal.

But what’s changed in hyperscale is different from Ethernet. Ethernet information is sent in packets. Whatever information — “where should I go in Italy, I’m going on vacation” — gets divided up by your search engine, sent in packets, and comes back to you. We’re not very cognizant if there’s an error drop or some delay, because those are in the hundreds of nanoseconds or milliseconds — to us it’s a thousandth of a second, we don’t notice. But for hyperscale systems, which are literally thousands of GPUs acting as one brain — one supercomputer, one high-performance cluster — that latency is super critical, because the GPU notices anything in the tens or hundreds of nanoseconds. So the typical Ethernet bit error rate needs to drop from 10⁻⁶ to below 10⁻¹⁰, what we call error-free, because any errors slow down the whole training and inference — all the AI infrastructure. That’s changed, and it’s allowed these higher-cost single-mode solutions — single-mode being longer distance, longer reach, very high performance — to come into the data center and hyperscale systems, because of this requirement for very low bit error rate. And now VCSELs have to raise the bar from 10⁻⁶ (typical Ethernet the last 25 years) to 10⁻¹⁰, 10⁻¹². And we’ve done that — we’ve pushed our VCSELs to 200 gigabit, and even used at 100 or 50 gigabit NRZ you can leverage that to very low bit error rate. So the answer is: things changed for hyperscalers to require error-free, and that’s allowed these high-end single-mode solutions to compete directly with VCSELs, because of that additional spec that’s new to hyperscale AI systems.

Austin: Okay. So because the capacity for sustaining errors is much lower — we don’t want all these GPUs just waiting — that changes the game from the cloud/SaaS days to now, when everything’s acting as one big computer. So the bit error rate has to be much lower. And VCSELs, which are short reach, have to come down to meet that; or, what you’re saying is, these other technologies that are longer reach and higher power were already closer to the necessary bit error rate, so people say, why don’t we bring those into short reach? But surely that has power and cost tradeoffs — taking something that talks at low error rate over a long distance and trying to bring it in. So you guys must be taking a different tack: no, no, let’s just make VCSELs error-free. And presumably that’s a cost or power tradeoff you’d rather make — or is it back to the manufacturing supply chain capacity?

One Pizza Oven vs. 5,000 Pizzas — The Capacity Argument

Al: Yeah, it’s again a complicated matrix of items you have to check off. Those shipping silicon photonics and EML for long reach — or DFBs — single-mode, high-performance devices, have been around a similar time, 25 years. They’ve been used for long reach because of that super-high performance, very few fibers, and what we call WDM — wavelength division multiplexing. Fibers are very expensive when you’re going hundreds of kilometers, so you want to use very few fibers but pass more information through more wavelengths, more colors, in that same fiber. For short reach — tens of meters — you’re not as locked into the cost of the fiber; it comes down, because you’re only 10 meters versus 10 kilometers. So the volume supply chain of the high-performance, low-bit-rate device would seem a natural fit: hyperscalers want low bit rate, let’s go with the Ferrari — the super-high-speed, high-performance single mode. But those players have been used to building in maybe 100K — 100,000 — volumes, because you don’t need as many between cities and countries. And all of a sudden they come into the data center. Even though their performance is excellent, the cost is a little higher because of the single-mode packaging. And the infrastructure to build millions per month is 10, 20, 50× what exists. That’s brick and mortar. It’s like: I only have one pizza oven, and I’ve been used to a small clientele — maybe 20 an hour. Someone comes in and says, I’d like to order 5,000 pizzas, and I need them in an hour. You’re going, I need 100 pizza ovens. That’s the exact same problem the high-performance single-mode long-reach players are coming into. Silicon photonics, EML — excellent technology, very good bit rate — but the infrastructure needs to be built up. That’s what you’re seeing: a lot of announcements with people holding shovels, saying we’re investing in the next supply chain, the buildings. That’s great — bringing more manufacturing to the world, and back to the US, all great for the industry. But VCSELs have been around for 25 years and shipping in the millions. So we’re not building, we’re not putting shovel to ground — we’re just changing the actual VCSEL.

We’re just changing the actual VCSEL performance — the chip — and leveraging the existing infrastructure. And so for VCSEL capacity, we call it unconstrained. Constrained means they’re sold out: we’re sold out through next year; if you’re going to place an order, it’s going to be an eight-month, 18-month lead time — a year and a half from now we can get it to you. For us, unconstrained just means we have a certain lead time that’s limited only by our cycle time of building through the factory. A VCSEL run and then a packaging run may be four, eight, twelve weeks — but that’s limited just by the fact that we have to build it and ship it, not by the constraint of the supply and ecosystem of machines, or pizza ovens. We have plenty of pizza ovens: place the order, and we’ll get you your order in the cycle time we commit to.

Austin: Gotcha. Okay, that’s very interesting, and a great competitive advantage for you. So on the constrained side — is that what listeners hear about when they hear about indium phosphide being a bottleneck? Where in the supply chain is it constrained? And for you, what’s different about VCSELs that makes them unconstrained?

InP vs. GaAs and the Fabless Split

Al: Yeah. So a lot of silicon photonics and EMLs are based on this material, indium phosphide. For VCSELs, our technology has always been gallium arsenide. For the listeners it may be, okay, one III-V compound versus another — what’s the difference? Indium phosphide is material-constrained from the very beginning. You can’t even get a base substrate — even before you process it, just the substrates for indium phosphide are limited, before you get it made into a product, whether that’s EML, silicon photonics, or VCSELs. The indium phosphide bare material is already limited. Gallium arsenide: unconstrained. So we start with that. And then you go through the fabs — fabrication depends on foundries, usually very large companies. Companies like PicoJool and others don’t own large clean-room factories. We design the individual VCSEL chip device, and then we use foundries to manufacture it. Those foundries are available — but they can’t get enough indium phosphide starting material. Now, after that, once you get to the chip level — you dice it up, okay, great, I’ve got the laser, I’m ready to go — to get from the chip to a pluggable device, an actual optical engine, there’s a ton of stuff that happens. You’ve got laser drivers, you’ve got boards, and for single mode you have to have all the machines that align that silicon photonics or EML to a very, very small-core single-mode fiber. Those machines have to be readily available. VCSELs, again, have that millions-per-month volume infrastructure — it doesn’t need to be built up. So from the very beginning: indium phosphide material constraint, then you have to build it into lasers, then finally package it into transceivers — and all along that supply chain, it’s not used to building millions per month. All of that has to be built up: hardware, alignment machines, testers. VCSELs have all of that infrastructure existing already.

Austin: I see — yeah, that makes a ton of sense. And I love your props, by the way — I liked the little VCSEL you held up. So tell us more: what are we looking at, and walk us through exactly what you design, and where it gets handed off, built, and packaged — where your responsibilities end.

The VCSEL Tree of Knowledge and the Handoff to WIN

Al: Yeah — so background is super important here. There’s what we call a tree of knowledge of VCSELs. You have this line from Honeywell through Finisar, and Finisar goes into II-VI, which goes into Coherent — so there’s the Coherent line. Then for us, obviously, HP went into Avago, went into Broadcom — the HP–Broadcom line. And finally there’s Picolight and E2O — I’m throwing in some old names from 25, 30 years ago — which go into JDSU, another big name from the dot-com time, and JDSU spins off Lumentum and Viavi. We’re from the Lumentum arm. All three of these major arms have a lot of VCSEL knowledge — and the strength of PicoJool is that we’ve tapped into designers from all three branches. Imagine all that know-how — again, not patented; know-how, recipes. I use this example: you can hand three different chefs a recipe for a soufflé and you’ll most likely get three different soufflés, because it’s really difficult to get a perfect one. It’s not just crack the eggs, beat the eggs. Same thing in the VCSEL.

So to answer your question — what do we do? We take all that know-how and design the epi layers. All these little lines you see — these are epi layers designing this vertical cavity. It’s not edge-emitting, where the light comes out of the edge of a flat chip; VCSELs are vertical-cavity, surface-emitting — the light comes out of the surface. We design the internal cavity of the laser, all the dopings, all the process. After we design, we hand it to an epi foundry that grows the material and gives us back an unprocessed epi wafer. We’ve been working with WIN Semiconductors in Taiwan — that’s our foundry. Once the epi wafer is ready, we hand it to WIN, they run it through their clean-room process, and they make the final VCSEL device. And here’s another beauty of VCSELs versus an edge emitter: at the wafer level you can start testing and probing each one — 100% tested, what they call known good die — before you have to singulate and dice it into arrays. That advantage is huge, because the more work you add before you yield the device, the more value you lose downstream. You always want to yield upstream. Wafer-level testing for VCSELs is a real advantage versus edge-emitting technologies.

And WIN is only the wafer processing. The wafer comes out and is diced into individual VCSELs, and we work with our partners to build those into either active optical cables or transceivers. That’s another foundry, if you will, but a packaging company. That said — companies like TSMC are now going into co-packaging: after they make their silicon wafer, they’ll package the optics directly on top of it. There’s a whole emerging field called CPO, where the traditional wafer-processing foundries are stacking technologies together — 3-D wafer-level packaging. For us: we only do the wafer at WIN, and then that WIN VCSEL goes to our module-integrator partners, who build it up into the active optical cables or transceivers.

Austin: Gotcha, that’s helpful. So take us back to your roadmap. I know you mentioned a 50G version, 100G, 200G — and a recent launch. Tell us more about your roadmap, what you launched, what you announced.

200G/Lane and the Three Flavors of 1.6T

Al: As I mentioned early on, 200 gigabit per lane is the benchmark — the bar you have to clear. EMLs and silicon photonics have all done that, and now VCSELs have reached it. PicoJool just announced our 200 gigabit; we’ll start sampling next quarter. That’s for a very simple transceiver where you have eight channels of 200 gigabit coming in — the aggregate bandwidth of 8×200 is 1600 gigabit, or 1.6 terabits.

So that’s the standard ramping today. There are 800 gigabit transceivers as well, that’s also shipping — that’s 8×100. And the next generation, today’s generation, is 8×200 — the 200 gigabit VCSEL we announced. But there are many flavors of that because of the specification. Aggregate bandwidth 1.6T — check. But there are different ways to get there if you want very low power or very low bit error rate.

Running 200G, I liken it to a Ferrari — it can give you 200 miles per hour, but it’s very high-end, relatively expensive, because you need certain signal integrity and signal processing. So now I say, I want very low cost, low power, but I still want low bit error rate. And what people have done is: let’s slow it down. Let’s use the 200G-performance VCSEL, but run it at 100 gigabit.

So now you have an excellent VCSEL that gives you a lot more performance, and if you use it at half the speed, you get much better bit error rates — the signal-to-noise, or relative intensity noise, drops as well. That’s 100G. But you need more lanes — to get to 1600 you need 16 lanes of 100. And recently something came out called micro-VCSELs.

And that goes even slower, down to 50G — they call it NRZ. So instead of PAM4, which has four levels (0, 1, 2, 3), we go back to the original NRZ, which is 0 and 1. Now you use the entire 0-to-1 signal-to-noise, which again reduces your bit error rate. But you need more channels — 32 channels at 50G to get to 1.6T.

But we’re shipping all three, and all three are in demand depending on whether customers want what they call fast and narrow (8×200); or an LPO — linear drive, no DSP, low power — which is 16×100G; or, if they want really low bit error rate and very low power, the 32×50G NRZ, which they call slow and wide.

We tend to call it “fast and wide” and “faster and narrow.” When you’re in high-speed interconnect, we try not to use “slow” in any of our marketing.

Austin: That’s good, that’s good. Okay, interesting — this is really cool. So you’re saying there are many ways to get to 1.6T. You could have 8×200, which uses PAM4 and requires a lot of DSP and power, but it’s definitely possible. Or 16×100 or 32×50, and you need less DSP for each of those — the 32×50 has much less because it’s NRZ. Yeah, this is all fascinating. So will that approach still hold once you move to 3.2T? Is it going to be that same combination of possibilities?

The Roadmap to 3.2T and 12.8T

Al: Great question — people always ask, what’s the roadmap ahead, what’s the future, is this the end of the road? Okay, 1.6T, VCSELs can do it — but is there a 3.2T? A 6.4T? A 12.8T? I mean, Andy Bechtolsheim — notorious — he’s created an XPO that’s literally going to give you 12.8T in a big pluggable today.

So they’re planning way ahead, because no one has ever told us in the last 30 years, “whoa, we have way too much bandwidth.” We have to have that bandwidth. And exactly like you said — what’s the future? Number one, we can use something called bi-di, bidirectional. Meaning we just add another wavelength — not the complexity of WDM where you have eight or 16 wavelengths like single mode.

We basically just add another wavelength to our existing one. So two wavelengths, passing them in both directions — bidirectional. That doubles the bandwidth without changing anything else, except you add another laser at a different wavelength, and you leverage the entire ecosystem. So from 1.6 to 3.2, we could add another wavelength. The other way, of course, is to double the speed. Can we do 100G NRZ?

That’s in the works. We’re developing 100G NRZ — today is 50G NRZ, but we’re developing 100G NRZ, leveraging our 200 gigabit VCSEL running at 100G NRZ. And in the future, we can go to more channels. The beauty of VCSELs again — surface emitting. Edge emitters can only have a one-dimensional array — a 1×4, 1×8, 1×12 — it just makes a long bar.

But for surface emitting, we can have a two-dimensional array — 2×4, 2×12, 2×16 — and couple all the light very elegantly with an optical fiber bundle. In that case I’m kind of unlimited. I can go up to 64 channels today in a 4×16 connector that’s the size of — let me show you. A 4×16 fiber is this size — here’s my finger. And that has 64 channels in it. If I run them at 200, that gets me to 12.8T. So in essence, the technology of today — without having to go to 400G per lane, which we’re also looking at — but at 200G, with more channels, with more colors (another color for bi-di), you double, you triple, by size. So that roadmap to 12.8T, we believe, is very solid and very clear — without even having to invent any new technology. And then with new technology, it just gets easier, if you can do 400G per lane.

Austin: Sure. Fascinating. So it’s just the same 200G VCSEL over and over — put it in an array and you get more of those. Or are you having to invent a new VCSEL to get the 100G version to run at NRZ?

Al: We’re just starting tests. We believe the 200G VCSEL has the capability to go to 100G NRZ. So it’s not a new VCSEL — it’s just a different coding on the signal coming in, running at NRZ instead of PAM4, to take advantage of the 0-to-1, using the whole signal-to-noise ratio as one bit as opposed to four levels. So that’s the difference.

Lead Times, Yields, and WIN’s Capacity

Austin: Gotcha. Yeah, this goes back to your engineering, pragmatic mindset — taking a Lego block and figuring out different ways to place it, different ways to use it, to unlock this whole roadmap. That’s pretty cool. So you talked about unconstrained gallium arsenide and working with WIN — they’re used to making this stuff, they’ve got all the pizza ovens they need. So if a big hyperscaler comes to PicoJool and says, we want a million of your pizzas — what does that look like? How does that actually happen?

Al: Right. If they want a million VCSELs, we’d typically give them an eight- to ten-week lead time. That’s the basic. We can accelerate it — push and have engineering carry certain wafers — but typically eight to ten weeks on the VCSEL side. You’ll get a wafer, or individually diced VCSELs. If you want transceivers, then that eight weeks tags on a certain number of weeks to package it all into the transceiver.

Those are constrained strictly by the packaging process — not by ordering equipment or building capacity. That ecosystem works, and it leverages the typical cycle time of building out. Typical cycle time: eight weeks for the VCSEL device, then four to six weeks for the module after that. So you’re looking at anywhere from 12 to 16 weeks to get to the full module, starting from an epi reactor growing epi and going all the way through.

And we’ll continue to drive that lower — lead time or cycle time. But also yields, which I didn’t mention much. Yields are: if I make a VCSEL, how many known-good die can I get out of a wafer? The higher the yield — up to 100% — the fewer wafers I have to run, and the higher the capacity of the factory. If every wafer goes through and I get 100%, I need fewer wafers, and therefore less capacity for the demand. Obviously getting to 100% is very hard, but VCSELs have been perfecting that process for many years, for many decades — and now we’re leveraging all of that. It’s not new, it’s not something that has to be established, it’s not based on new technology. That’s very important. WIN has been doing it for about 10 years, since we transferred that for a consumer electronics application back in 2016. They have a capacity of up to a thousand of these wafers per week. And there are about 240,000 VCSELs on each wafer, because they’re really tiny. So that adds up to a million yielded from maybe 10 wafers. The capacity is huge for datacom. So I think we have no worries — once we get to the 200G, the product specs are met with the customer, and reliability qualification is done, then we just ramp readily with WIN, and they’re ready to go.

The Fabless Model — Stay Small, Leverage the Supply Chain

Austin: Nice. Amazing. It’s quite compelling. Normally when people hear “there’s a startup trying to compete in a space with these huge incumbents,” the question is: how’s this startup going to compete? How do they get to market, find customers, build up supply chain? But what I hear you saying is that you’re taking industry veterans with process know-how — similar ways of thinking as competitors, you’ve been in the game a long time — and tapping into an existing supply chain. And at the end of the day, it’s not like you have to win 50 customers; there’s a handful of big customers that would really make a difference for PicoJool if they said yes. But the most important point is the unconstrained gallium arsenide — being able to make a million, 10 wafers with 240,000 on each, whatever you yield, we’re talking millions of VCSELs very quickly. Because across all of semiconductors — memory, CPUs, AI accelerators — there’s so much demand and such constrained supply that, yes, you want to compete on cost and engineering performance, but there’s also a bit of: if it’s good enough and it’s in production and you can install it into my data center, game on. So it feels like you have a strategy that lets you deliver shipped VCSELs as soon as possible.

Al: Yeah, the model has been around for decades in silicon. I’m in Palo Alto, in Silicon Valley. Most chip companies designing integrated circuits, CPUs, GPUs, don’t have their own foundry. A lot of people use Intel; even AMD uses TSMC. AMD is a huge chip company and they don’t have their own foundries today. TSMC, Intel, Global Foundries — many foundries are the factory floor, the clean rooms, of all these startups. So just because of our small startup size doesn’t mean we can’t ship in the millions per month and compete directly with the very large presence of other optical suppliers and competitors. That’s the beauty — we can stay very lean. Our motto is “stay small,” and that has to do with many things — our name is PicoJool, right? Very, very low power, and that’s one of the things driving us. We’re a well-experienced small team, but we get huge benefit by leveraging WIN Semiconductors and working with our supply chain. They’ve got the factories and the clean rooms, and we can ramp very quickly by providing our designs and our unique specialty, then partnering with these large companies that are already shipping — and they just drop-ship. So we don’t need a big company to ship in the millions per month.

Austin: That’s amazing. Definitely punching above your weight — that is awesome. So when is your high-volume ramp targeted for? If I recall, the press release said something —

Ramp Timing — Sampling Now, HVM in Early 2027

Al: Yes, the press release said we’re starting to sample next quarter. And that’s in different flavors — we’ve got customers for the 50G NRZ, the 100G LPO, and the 200G. So all of those begin to sample. And the process from our device to actual product shipment — revenue, in our case — is a period they call qualification, or reliability testing. Everything has to not only meet spec at zero hour, but be predicted to last 10 years, or a number of years, through what they call accelerated aging — they test it at higher temperature and higher bias conditions, then estimate back to normal operating conditions: can it last 10 years in the field? For very small, tier-two customers, that can be as short as three months. But for tier one — because they have much more to lose if there’s an issue with their connection — it takes more than six months to get up and running. So ramping at WIN is available today; but we have to go through this qualification cycle with our customers before they give the orders and everything is approved. So we’ve also said we’ll most likely start ramping in early 2027, next year.

Austin: Okay, got it. Thank you for the education here. So — sampling, then the qualification process, then ramping. And probably, as you’ve been saying throughout, you’re not concerned about the ramping. You had to build the product and let customers kick the tires, and once they say let’s go, it’s off to the races. Okay, awesome. Well, we’ve covered so much — this has been amazing. I’ve learned a lot, and I know the listeners will have too. Is there anything else, any last things about PicoJool, or anything we didn’t talk about that you were hoping to cover?

Mentoring the Next Generation of Photonics Engineers

Al: One thing that’s very interesting to me — as you can see from the image, I’m a very experienced, elderly startup person. One thing to point out: very few people went into hardware and photonics in our space, because young people over the last 25 years — literally since dot-com — came into the workforce more on the application side, the software side. People wanted to go into computer science. So the aging, experienced hardware folks in photonics need to transfer all this knowledge. That’s one of my passions — to bring on the next generation, and the generation after that, for photonics. Because just like VCSELs and other technology, we see many decades ahead, and as I said, no one’s saying “way too much bandwidth.” We see bandwidth demand increasing with robotics and autonomous vehicles and what have you — everything is going to be bit- and connectivity-constrained. So we want to spend time to educate and train. We’re trying to hire hardware engineers, and train young folks — maybe without the experience — to be the VCSEL designers and transceiver designers of the future. That’s really exciting for us, because we’ve got all this knowledge, 10, 20, 30, 40 years, and it feels wonderful to have the hardware excitement again — not only in the markets, but the investment community. Silicon Valley is booming with photonics and hardware. So we don’t take it for granted. It’s a great opportunity, and we definitely want to take young entrepreneurs, young engineers, folks interested in this space, along for the ride — and then they take it from there.

Austin: I love it. Very inspiring, very cool. It’s never been a better time for interconnects, photonics, and optics folks. I love that you industry veterans want to bring up the next generation and give them the opportunity to learn from folks like you — to revitalize, rebuild, make sure we have a reinvigorated workforce, so that for my generation and the generation of my children, they can keep having more and more data moved around, faster and faster.

Al: That’s right. Yeah, really enjoyed our conversation, Austin. Thank you.

Austin: Awesome. Thank you, Al. Appreciate it.

But Groq and Cerebras run architectures you can poke holes in. Both were designed before LLMs went mainstream, and their early design choices don’t suit today’s models. I call them “preGPT” accelerators.

The famous example (well-covered ground already) is that they’re SRAM-only. It takes many Groq racks to serve even one smallish model, and even Cerebras’ wafer-scale marvel (900,000 cores on one piece of silicon) can’t hold a frontier model’s weights on a single wafer, so you have to expand to many wafer-level systems. Tough economics, a KV cache that doesn’t scale gracefully, and so on.

Admittedly, I looked at these engineering details and figured these companies were dead in the water when it came to LLMs. Well, to their credit, who could have predicted such thicc models 10 years ago? But no HBM to support long-context... how will they survive? More context is better…. failure to thrive?

What’s important is that they both had production silicon available, and, regardless of the shortcomings of SRAM-only architecture, they outperformed GPUs on a certain KPI (interactivity). As Jensen nicely drew for us, SRAM chips for decode unlocked new possibilities on the Pareto frontier for 2026:

And that’s why they had eleven figure outcomes. $XX Billion, kind of nuts right!

I was so wrong about architectural shortcomings preventing success.

Looking back, what mattered most? Timing.

Groq was in production right when Nvidia came calling. Cerebras, right when OpenAI did. Forget the architectural shortcomings. If you can ship and unlock a new Pareto frontier, good things happen. In fact, I thought the fundamental problem for Groq and Cerebras was being too early, making design decisions before transformers took off. Being too early is indistinguishable from being wrong...

And yes, starting that early nearly killed them; both came close to running out of cash. But starting early was also the whole advantage, because when the unforeseen inference wave hit, they had silicon in production. Well, they were both still default dead until Meta released Llama, the first useful open weights LLM, which allowed Groq to go show the world how awesome a super fast chatbot experience was.

Again, the combination of production silicon at the right moment and a Pareto frontier the GPU can’t reach is all that mattered. And then Nvidia’s Dynamo was icing on the cake, disaggregating the workload so the memory-bound decode could run on SRAM chips.

The postGPT era

If the first era of inference was GPUs, Groq and Nvidia ushered in the next era with the GPU+LPU inference. But another era is incoming. The postGPT accelerators. Rack-scale systems specifically designed for LLM inference from day one. From matmuls to interconnects to memory hierarchy.

There are many startups in this space, not to mention custom silicon XPUs (Maia, MTIA, Trainium, etc). So which of these could be the next Groq/Cerebras with a whale of a customer and an acquisition/IPO?

What it takes to be the next trillion-dollar chip company

If we’ve learned anything so far, it’s that we shouldn’t overindex on engineering architectures. Sure they matter in the long run, but in this demand environment, contenders simply need to hit these four criteria:

• Runs frontier 1T+ models

• Ships rack-scale

• Beats the incumbent on one KPI

• Lands a frontier anchor

Runs frontier 1T+ models. Frontier models are where the value gets captured and where scale happens, so frontier customers are the ones most willing and incentivized to deploy a postGPT accelerator at volume. A system that can’t run a 1T-param model is playing a sub-frontier game. I’m bullish on the usefulness of sub-frontier, especially in the enterprise, but the SAM is small. They won’t mint the next Groq/Cerebras.

Ships rack-scale. The race to gigawatt-scale inference runs on racks. To win, one must design and deploy racks of accelerators, quickly and reliably.

Beats the incumbent, clearly, on one KPI. The challenger doesn’t need to be best at everything. But they must be an order of magnitude better at something.

Lands a frontier anchor. The challenger needs someone serious about gigawatt-scale and buying into a roadmap and building a long-term relationship. A model lab or hyperscaler is best. Could be a big neocloud. Probably not a sovereign.

Of course, software support is table stakes.

But otherwise, I think it’s that simple.

Yes, the technical details matter; quantization, memory hierarchy, scale-up domain size, etc. But truly, right now they matter only insofar as they lead to differentiated performance (”beats the incumbent on one KPI”). The architecture behind each contender is in the teardowns; up here, judge them by the KPI it buys.

This is a unique time. Token demand FAR exceeds supply, and the shortage is worst for low-latency frontier tokens, where GPUs can’t keep up or can only do so at uneconomical throughput. This high-interactivity token scarcity turned “good enough architecture in production” into a winning hand for Groq and Cerebras.

But it’s time-bound; there was a window, and Groq/Cerebras grabbed it. And being first matters.

Which raises the next question: what’s the next window, and who will get there soonest?

Picking the next winner could be as simple as asking who can deploy a gigawatt of postGPT inference accelerators soonest?

To answer that, here’s the rest of the piece.

• The frontier race, all eleven ranked by when their frontier rack ships, as a scoreboard and a timeline

• A full teardown of Tenstorrent, the only one serving a frontier-class rack today, a free sample of the diligence

• [Paid] The other ten teardowns, each scored and sourced, the frontier contenders, the decode add-in, and the sub-frontier plays

• [Paid] The receipts, only two of the eight frontier names have a clean third-party number

• [Paid] Who actually looks like the next trillion-dollar chip company, and the argument that reframes the whole list

The frontier race: who reaches rack-scale, soonest

Well then: “Which startups clear those four hurdles, sorted by ship date soonest?”

Let’s take a look. I’ve sorted them by when they claim their frontier-class generation hits rack scale. Take it with a grain of salt, and remember this isn’t yet anything about performance, but just when they say they’ll have silicon deployed.

Also note that many of these companies have shipped products already, but the earlier products were sub-frontier as they pivoted their way toward rack scale for modern frontier LLMs. So I’m only focusing on when they are shipping frontier-capable products according to their roadmap

If we scour all the public sources (press releases, blogs, podcasts), we get the following:

Visualized:

Caveat though. Every entry above is a first-rack or first-sample milestone, not a gigawatt figure. IMO first rack matters, but time to first gigawatt matters most.

First rack proves the whole shebang works. But gigawatt scale proves the company works.

Who is shipping in 2026, and who isn’t?

Tenstorrent, Etched, and SambaNova ship production racks in 2026.

A year back sit Positron, Tensordyne, and Rebellions. Rebellions claims 2026, but its frontier REBEL is still only sampling, so its real rack is more like 2027.

Fractile and MatX are two years back. Both have 2027 milestones on their roadmaps, but those are samples and footholds, not scaled production shipments. Production racks at customers are realistically a 2028 story.

In this supply-starved environment, a year is expensive. That’s not to say the 2027+ companies aren’t technically competitive, nor does it imply they won’t win serious customer share in the grand arc of time. After all, it only takes one of the few large customers to get to scale. But you’re not in the game until you’re on the field! No matter how technically sound your architecture is, if you’re not on the field, you’re not in the game.

Who will be on the field first?

Tenstorrent claims to be “shipping production silicon now”. Promising! But what about customers? Tenstorrent has neoclouds and sovereigns; no frontier lab or hyperscaler has been announced yet. Could those early customers accelerate Tenstorrent to gigawatt scale? We dig into that in its teardown below.

An acquisition could help Tenstorrent get to market, as big silicon incumbents already have relationships with hyperscalers. Qualcomm was reported in mid-June to be circling a roughly $8–10B deal. But Keller supposedly denied any Qualcomm talks recently, so that path looks cold for now.

Etched is a bigger contender than it looked just months ago. It claims rack-scale shipments this summer, well ahead of the other popular names like MatX and Fractile. I think Etched was left for dead, but it turns out they are alive and claim to be right at the front of the line.

Notably, Etched is talking gigawatts. It claims “a path to gigawatt-scale in 2027,” and it’s talking about building the company infrastructure to pull it off with a Taiwan factory, plus a data center, test house, and NPI prototyping lab at its San Jose HQ. From the press release, co-founder Rob Wachen said “Our approach from the beginning has been to build for gigawatt-scale... Production is the product.”

Etched hasn’t disclosed specific customers but mentioned $1B+ in contracts booked. Etched’s talk about a “path to gigawatt-scale” and “building for gigawatt-scale” seems to imply they are targeting the type of companies that can support that scale.

Whether Etched actually hits gigawatt scale in 2027 remains to be seen, but it’s at least aimed at the right target if they want to be in the trillion-dollar race.

SambaNova also claims a 2026 deployment on its frontier SN50 landing in the second half. This is going to make for an exciting 2H26 and early 2027 if we’ve got SambaNova, Etched, and Tenstorrent all shipping competitive racks!

So far, SambaNova’s named customers are enterprises JPMorgan Chase and SoftBank, and these are explicitly NOT gigawatt installations. From Sally Ward-Foxton

SambaNova has a multi-year agreement in place with JPMorganChase.

“This is a breakthrough because most suppliers haven’t figured out how to get into the enterprise,” Liang said. “The enterprise market is going to be significant; it won’t be 100%, but it’s going to be a meaningful player in the overall market, and we’re starting to see that.”

Enterprise customers are looking to host private models and private data in their own environment, which is unlikely to be a gigawatt-scale data center, Liang said.

Of course, if you have tens of 50-100MW deployments, well, that counts!

Maybe SambaNova’s plan is to demonstrate feasibility with reasonably sized enterprises and use that success to try to win hyperscalers? Or just win with enterprises, which should turn into a very large market on its own.

Behind the 2026 crew are five more competitors including Rebellions, Positron, and Tensordyne in 2027, with Fractile and MatX not reaching production racks until 2028.

Rebellions reaches a rack soonest of this 2027+ tier, with REBEL sampling now. But its named customers, mostly Korean enterprises plus Saudi Aramco, sit on the sub-frontier ATOM, not the frontier REBEL, and none is a gigawatt buyer.

Positron has a promising customer, Oracle.

Some consider OCI (Oracle Cloud Infrastructure) a hyperscaler and others call it a neocloud (ClusterMax rates it as a gold tier neocloud). Regardless, OCI can support GW scale installations.

So is a promising path to GW scale for Positron, and they are the first on this list who can publicly say they’re already deployed in a hyperscaler. The catch is that Oracle is currently running Atlas, Positron’s sub-frontier first-gen server. It’s not rackscale. The frontier system, Asimov, doesn’t tape out until late 2026.

So Positron already has a customer who could scale to a gigawatt, just not yet on a frontier product.

Tensordyne recently announced its Napier system has been taped out on TSMC 3nm and production is underway. And Tensordyne’s logarithmic math approach is super interesting.

But its customer book looks a lot like Tenstorrent’s. Cirrascale and BlueSky Compute are named, plus a dozen-plus letters of intent to evaluate beta systems, and roughly $200M of forecast, not booked, demand. Real interest, but neoclouds and LOIs, not a frontier lab ordering a gigawatt.

Fractile is further out. Its roadmap targets a late-2026 commercial tape-out, customer samples in Q2 2027, and production silicon only at the very end of 2027, so racks in customer data centers are realistically a 2028 story. But it’s courting exactly the right customer. In May 2026, The Information reported Anthropic was in early talks to buy Fractile chips once they ship, which would make a frontier lab its anchor, the kind that scales fastest. The talks were preliminary and the deal size wasn’t disclosed though.

MatX is pre-silicon, on roughly the same clock as Fractile. It targets tape-out within a year of its $500M Feb 2026 raise, first chips a few months after that, and a 2027 foothold inside a frontier lab, routing ~1% of production traffic to its silicon as an A/B test.

That foothold is a validation experiment, not production racks; by Pope’s own tape-out-to-production math of one to two years, volume racks are a 2028-at-earliest story.

It hasn’t dated real rack scale, only that CEO Reiner Pope “would like to be... shipping multiple gigawatts a year” (Cheeky Pint). So it’s got the right goal, just behind on the timeline.

Yet it may have the best customer list of anyone here. MatX says it sells only to frontier labs, “maybe five different customers in the world” and its 1% trial would be a foothold with such a customer. If that little trial is a success, that’s the type of customer who would scale quickly to a gigawatt faster than any neocloud or enterprise mentioned.

Deeper Teardown

Of course, details matter. Here’s the due diligence on eleven companies with one question for each.

Can it put a frontier-class rack in front of a whale of a customer before the window closes?

Below is the summarized diligence on all eleven, the eight frontier contenders first, then the add-in and the sub-frontier plays.

For each we’ll hit on

• KPI is the one outcome the chip beats a GPU on.

• Anchor is who is actually deploying it.

• Architecture

• Verified?

Then a bunch of extra sourced details.

Tenstorrent (Q2 2026)

Jim Keller is arguing a contrarian narrative. One chip runs everything, no disaggregation. Keller believes that, in the long run, general-purpose can perform as well as specialized + disaggregation.

• KPI: low cost at high interactivity. Runs ~any model (~90% of Hugging Face), prefill and decode on the same silicon.

• Anchor: five-plus neocloud colos serving live, plus sovereign and IP licensing (Japan, LG). No frontier lab, no hyperscaler.

• Architecture: GDDR6, no HBM; RISC-V Tensix cores; Ethernet scale-out on-chip; 100% open-source software stack. One chip runs both prefill and decode, the deliberate counter to disaggregation.

• Verified? ⚠️ DeepSeek ~308 tok/s/user is a served-model number, but relayed by Keller, not an independently-published AA figure. The Prodia video claim is similar: AA independently ranks Prodia’s model at the top of its leaderboard, but that predates the Tenstorrent partnership, and the 10× speed-on-Tenstorrent-hardware number is Tenstorrent/Prodia’s own benchmark, not AA-reproduced. No MLPerf; the 350→500 tok/s headline is partly forward-looking.

DETAILS

The anti-thesis, on record.

• At TT-Deploy (May 2026), Jim Keller bet against specialization and disaggregation: “the big fad is disaggregation, special purpose hardware, SRAM. Do you know how many people are going to be talking about that in 2 years? None.”

• And: “you don’t just run part of an LLM… that’s not a good business plan long run. You could be one-shotted by one model.”

• Stan Sokorac’s (Sr. Fellow, Software) proof point: a pipeline that downloads random Hugging Face models achieves a ~90% pass rate, so “we can run ~2.5M of the 2.8M models on Hugging Face.” (TT-Deploy keynote; Sokorac talk)

Architecture.

• Tensix core = matrix engine + vector unit + 5 “baby” RISC-V cores per tile, each with local L1 SRAM, over a 2D NoC mesh (no shared global memory); Blackhole adds 16 “big” 64-bit RISC-V CPU cores and can run host-less (The Register).

• Anti-disaggregation by design: each chip has SRAM + DRAM + networking, so it runs prefill + decode + video on the same silicon (Vasiljevic talk).

• Scale-out: standard Ethernet, 10×400 Gbps = 1 TB/s chip-to-chip, the anti-NVLink bet. Blackhole = TSMC 6nm.

• Memory: large on-chip SRAM + GDDR6, explicitly no HBM. Official Blackhole spec: 180 MB on-chip SRAM, 28–32 GB GDDR6 at 448–512 GB/s (tenstorrent.com/hardware/blackhole).

Shipping status.

• Three taped-out generations (Grayskull → Wormhole → Blackhole).

• Shipping Blackhole cards: 120 Tensix / 664 TFLOPS BLOCKFP8 / 300 W; p100a $999, p150a/b $1,399.

• Blackhole Galaxy production servers reached GA Apr 28 2026 (6U, 32 chips, 23 PFLOPS dense FP8, ~$110K) (The Register GA).

• Yield caveat, now in the official spec: launched at 140 Tensix, a Jan 2026 firmware cut to 120 (≈1–2% perf hit on typical workloads; new cards ship Bin-3 parts with two disabled Tensix columns) (Tom’s Hardware).

• “Installed in ≥5 neocloud colos outside Tenstorrent” goal hit (TT-Deploy).

Deployments / anchors.

• Infra/colo/neocloud + finance, not hyperscalers: Equinix, Cirrascale, ai&, Prodia Labs, Virtu Financial, Turiyam, OrionVM (Tenstorrent newsroom).

• The other half is RISC-V/chiplet IP licensing + sovereign design: Japan/LSTC (Ascalon IP, Rapidus), a ~$50M Japan program to train 200 designers, LG co-developing SoCs.

• The gigawatt question. The July 2026 slate widened the book (Galaxy superclusters now anchor Equinix’s Distributed AI Hub, launched with OrionVM and BetterBrain, plus new ai&, Virtu Financial, and Cirrascale deployments), but none of it is gigawatt-class. The headline neocloud, Cirrascale, is ClusterMAX Silver, running in the thousands of GPUs, and already lost OpenAI to Azure (ClusterMAX). Every name here buys racks, yes, but not gigawatts. Aggregating many mid-size buyers may get there, but slower.

Performance / verification.

• Absent from MLPerf v6.0.

• ~308 tok/s/user on DeepSeek live is a served-model measurement, but per Keller’s own telling, not an independently published Artificial Analysis number; the roadmap to 500 tok/s/user at $6/M TCO is forward-looking.

• The Prodia video record (Wan 2.2, 2.4s, 10× a GPU) needs the same unpacking: Prodia’s model already led the Artificial Analysis leaderboard before the Tenstorrent partnership, so that ranking is genuinely AA-verified — but the 10× speedup on Tenstorrent hardware (33.8 fps vs. 5.5 fps on Nvidia) is Tenstorrent and Prodia’s own collaboration benchmark, not a number AA independently re-ran (Tenstorrent newsroom; Vasiljevic talk).

• Stack is 100% open source (TT-Forge, TT-Lang), the anti-CUDA-moat argument.

Team / funding.

• Founded 2016 (Toronto) by Ljubisa Bajic, Ivan Hamer, and Milos Trajkovic; Jim Keller CEO since ~2023.

• Series D $693M+ closed Dec 2024 at ~$2.6–2.7B post (Samsung Securities + AFW lead; Bezos, Hyundai, LG) (Series D).

• A reported ~$800M / ~$3.2B Fidelity round (Nov 2025) is not company-confirmed.

That’s the free read. Behind the paywall:

• The other ten teardowns, each scored on the same four questions as Tenstorrent, its KPI, its anchor, its architecture, and what’s actually verified, then the full sourced details behind every line. The seven remaining frontier contenders (Etched, Rebellions, SambaNova, Positron, Tensordyne, Fractile, MatX), the decode add-in (d-Matrix), and the two sub-frontier plays (Furiosa, Taalas).

• The verification scorecard, which two of the eight frontier contenders have a clean independent number, which one has only a partner-run result, and which remain vendor-stated.

• Which “frontier” chips can’t yet run a frontier model, the sub-1T ceiling sitting under every verified number, and whose only MLPerf result is for the wrong chip.

• The memory split, company by company, who leans on HBM for raw bandwidth and who drops it for commodity LPDDR or GDDR to dodge the supply crunch, and what each choice costs them.

• Where the field lands on disaggregation, one chip running prefill and decode versus a split tier, and which way it’s converging.

• Who actually looks like the next trillion-dollar chip company

If you’re deciding who to watch, partner with, invest in, or compete against, this is the half that matters.

Etched (H2 2026)

A frontier inference system, sold as a full rack, has its real edge in low-voltage inference and cluster-scale memory. And more flexible than it’s remembered for: the system that launched in June runs DeepSeek, Qwen, Llama, and Mamba. Mamba is an SSM, not a transformer… so clearly it’s more flexible than previous talk of “transformer-only ASIC” made it out to be…

Read more

If you’re not up to speed, the basic idea is that HBF is a stack of NAND dies built the way an HBM stack is built; dies stacked vertically, wired together with through-silicon vias (TSVs), and sitting right next to the GPU on the package interposer:

What’s interesting is that it has the same read bandwidth as an HBM4 stack, but with roughly 10x the capacity. And it’s made of NAND, not DRAM like HBM. (NAND is the cheap stuff.)

The first samples of the memory itself are expected very soon from Sandisk, sometime in the second half of 2026. Samples of the first AI inference devices built with HBF follow in early 2027.

Everything has tradeoffs, flash too. We’ll look at why those tradeoffs aren’t so bad for inference decode workloads, and the impact HBF will have on the memory market.

Table of contents

• Flash is storage. But could it work for memory?

• Weight memory is what drives GPU count

• DRAM is no longer scaling well. But NAND is

• CMOS directly Bonded to Array: how the HBF stack is built (for the device physics layer, see Vik’s High Bandwidth Flash: NAND’s Bid for AI Memory)

• [Paid] Inference decode fits this profile; training does not

• [Paid] Four ways HBF attaches to a GPU, each with a different supply chain winner

• [Paid] The disaggregated inference cost model

• [Paid] What HBF does to the memory markets

• [Paid] Nvidia is the certification gate; custom silicon moves first

• [Paid] Competitive landscape: Sandisk, SK Hynix, Samsung, YMTC

• [Paid] The patent: a processor bonded to NAND

• [Paid] HBM is infeasible for edge devices; HBF is not

• [Paid] Timeline and risks

Flash is storage. But could it work for memory?

NAND flash has traditionally been used for information storage, i.e. where data lives when it’s not being used. Cheap, dense, and non-volatile. But slooooooowwwww....

But memory is where the accelerator keeps information handy during computation, and it has to be fast enough that compute never waits. SRAM is used for memory and can be read very quickly, on the order of about a nanosecond. Next in the memory hierarchy is DRAM, which is slower to read at ~100 nanoseconds.

But storage like NAND takes about a hundred microseconds, roughly 1,000x slower than DRAM. By definition that’s storage and not memory, right? It takes way too long to read, but good for long term storage. That won’t work for computation... way too slow! Compute would just be sitting idle, waiting forever for data.

So how could NAND flash possibly be used as memory?

Well, could there be a way to get the right data to the accelerator in time for computation, even with NAND’s slow reads? If you start the read it WAY before the accelerator needs the data, it could work?

In addition to intelligently scheduling when data is requested, one can also try to achieve high bandwidth from NAND. Bandwidth is the amount of data that arrives per unit of time. So if you know the read is going to take a long time, well, might as well have a bunch of reads running in parallel, if possible, right?

HBF unlocks high bandwidth by stacking NAND dies to increase the “width” or the parallelism. Each die is divided into many sub-arrays, which are small blocks of NAND that can each be read at the same time, independently of one another. An HBF stack places 16 of these dies behind a single interface, thousands of bits wide, so a huge number of sub-arrays are available to read at once. Any single read still takes thousands of nanoseconds, but thousands of reads can run in parallel, so a lot of data can be moved simultaneously.

The 16 flash dies are stacked and connected with through-silicon vias (TSVs). A controller logic die is then bonded directly onto the NAND array. That logic die schedules the parallel sub-array reads and drives the results out over the wide interface to the GPU. The bonded die is called “CMOS directly Bonded to Array” or CBA:

The logic + HBF stack is on a high-bandwidth interposer. The resulting HBF delivers 1.6 TB/s of read bandwidth, which is the same as an HBM4 stack at the JEDEC spec’s 6.4 Gb/s operating point:

To underscore the importance of what the packaging is doing here, compare against the NAND you can buy today. Conventional flash ships about 14 GB/s behind a PCIe 5.0 NVMe controller. Packaged as HBF, the same material delivers 1.6 TB/s. Roughly 100x the bandwidth, from packaging alone. Bullish advanced packaging!

Again, HBF’s latency is still 10-100x slower than HBM. But if the accelerator knows what data it needs in advance, it can prefetch it and avoid waiting on any single slow read. The argument for HBF is that inference decode is the perfect workload. I’ll explain in more detail later for paid subscribers.

Weight memory is what drives GPU count

What are the economic implications of HBF?

Inference cost scales with GPU count, and for today’s massive frontier models, GPU count is often driven by memory capacity per GPU.

Why?

Well, you need enough HBM to hold all the weights, but HBM is co-packaged with the accelerator; a fixed amount of HBM is bonded into each GPU package. So you can’t add memory without adding GPUs. Hence, bigger models mean more GPUs.

Of course, weights aren’t the only thing the accelerator needs to store in memory and acccess quickly and often. The KV cache and activations sit in memory too, and both stay in HBM or DRAM.

But the KV cache takes new writes every token; NAND’s endurance can’t handle that. NAND has much lower write endurace.

And activations need low-latency random access that NAND is too slow to give.

So HBF is for storing model weights.

And frontier weights are huge and always wanting to be even bigger. Wouldn’t it be nice to hold the model in significantly cheaper memory than HBM?

Recall that a 70B parameter model at fp16 (2 bytes per parameter) requires 70 × 10⁹ × 2 = 140 GB just for weights. A 1T parameter model thus needs 1,000 × 10⁹ × 2 = 2 TB. But that’s a lot of HBM; today’s shipping HBM4 stacks hold 36 GB (12-Hi); 16-Hi parts push 48 GB, and the JEDEC spec tops out at 64 GB. So a large model needs many stacks, which means many GPUs. Expensive!

That also means more interconnect (to move data around all those GPUs), which requires more power and, ultimately, a higher cost per output token (watts and $).

But HBF can provide 512 GB of capacity per stack!

So 512 GB per stack versus 48 GB for HBM4 is roughly 10x the capacity at the same bandwidth. Sandisk states up to 8-16x across the HBM family, depending on which generation you compare against.

Interesting! Super promising. Consider the implications of storing frontier models in a few HBF stacks rather than spread across many HBM stacks across many GPUs in a server or rack.

New opportunities arise too! Could on-premises, air-cooled deployments at enterprises have frontier-model weights capacity without needing so many expensive GPUs?

CMOS directly Bonded to Array (CBA)

Let’s dig into the manufacturing of HBF a bit more. Dang, I seriously didn’t mean to be punny with that bit.

Sandisk stacks 16 BiCS NAND dies (BiCS, or Bit Cost Scalable, is just its brand of 3D NAND) with TSVs (the same way HBM is built) and bonds the controller onto the array:

The result matches HBM4’s footprint, height, and power envelope, so it can drop into the same interposer slot an HBM stack sits in today. It speaks nearly the same electrical interface, too, though the host memory controller has to change, so it isn’t quite drop-in. And being NAND, it’s non-volatile (it holds data with the power off), so unlike DRAM it needs no constant refresh power.

One open question is the NAND cell type. NAND can pack more than one bit into each memory cell:

Highly recommend this explainer from Vik:

Vik's Newsletter

High Bandwidth Flash: NAND’s Bid for AI Memory

Welcome to a 🔒 subscriber-only deep-dive edition 🔒 of my weekly newsletter. Each week, I help investors, professionals and students stay up-to-date on complex topics, and navigate the semiconductor industry…

Read more

10 months ago · 26 likes · 4 comments · Vikram Sekar

Note that the fewer bits each NAND cell packs, the faster and more durable the cell is, but at the tradeoff of density.

QLC (quad-level cell) holds four bits, cheap and dense, but is the quickest to wear out. SLC (single-level cell) holds one, yet is far more durable.

Sandisk hasn’t said which HBF uses, but Irrational Analysis reported recently that industry sources tell him it will be SLC, which would improve HBF’s write endurance by an order of magnitude, but at the cost of density.

Sandisk makes the NAND, but turning 16 dies into a stack that matches HBM’s footprint, thermals, and interface is advanced packaging work... the same thing the big 3 HBM folks have already perfected. So SK Hynix is the stacking partner.

Sandisk brings the flash, SK Hynix brings the stack.

The two began an OCP standardization effort in February 2026.

Standardization?

You might expect Sandisk to want a monopoly on HBF, right? Actually, it wants other suppliers, so long as Sandisk is the biggest. After all, hyperscalers won’t design a sole-source part into a billion-dollar platform, so being the only supplier is a risk to HBF adoption.

A ratified standard with multiple suppliers removes that risk and gets HBF designed in. Again, Sandisk would rather own a big share of a market that exists than all of one that never does. It reminds me of with Credo and active electrical cables. Grow the category into a real market, and win the largest slice of it.

So how does HBF compare to HBM?

HBF has the edge when it comes to capacity vs HBM4, but trails in most other respects:

Note the 512 GB capacity is 14x the 36 GB 12-Hi HBM4 shipping today, ~10.7x the 48 GB 16-Hi flagship, and 8x the 64 GB JEDEC max.

For the same bandwidth, HBF has up to 8-16x the capacity at roughly 2x the power.

Yes, HBF has worse bandwidth per watt, but for model weight storage one can argue the metrics that matter most are capacity and capacity per watt, where HBF wins.

SanDisk also claims a similar cost to an HBM stack despite 8-16x the capacity, which works out to roughly 10x lower cost per GB.

I thought NAND was way cheaper?!

The raw NAND is cheap per bit, but an HBF stack isn’t cheap. Most of its cost is the same advanced packaging that makes HBM expensive like the 16-high TSV stacking, the CBA logic die, and the interposer. So you get HBM-stack pricing with ~10x the bits, not SSD pricing.

Of course, HBF has weaknesses compared to HBM. It’s all tradeoffs.

Microsecond latency against HBM’s ~100 ns, a minimum read 128x larger, and write endurance that rules out anything write-heavy.

Objections to HBF are primarily those three issues. But the painfulness of those weaknesses depends on the workload. And for decode inference, HBF isn’t so bad! We’ll cover it more below.

Interestingly, John Carmack has been thinking about this too; yesterday’s X post has over 1M views:

Click on it and read the whole thing! Here’s an important snippet:

“model inference can have a deterministic memory access pattern. You don’t need ‘random access memory’ at all for model weights, and you could tolerate cold-start latencies in the multiple milliseconds, as long as continuous reads were delivered at the necessary bandwidth.

Clearly, there’s merit to flash for weights for inference.

There are lots of nuances too.

And many big questions. How does HBF impact HBM demand? How does HBF impact the already hot NAND market? I’ll address those and more.

What paid subscribers get in the rest of this piece:

• Why decode doesn’t care about NAND’s weaknesses

• The four ways HBF attaches to a GPU: direct HBM replacement, mixed HBM+HBF slots, HBM-as-cache, and disaggregated prefill/decode. Each implies a different GPU attachment rate and a different set of supply chain winners, and the earliest route to production isn’t the obvious one.

• The disaggregated cost model, worked: a table pricing the model weight tier for Llama 405B, DeepSeek-V3, and a 1T-class model. HBM vs HBF

• What HBF does to the DRAM and NAND markets

• Route to market: Nvidia, hyperscaler custom silicon, what about ASIC startups and other merchant GPU vendors?

• A Sandisk patent that hints at the long term roadmap for HBF

• Timeline and risks

And more!

Read more

Just look at the past four quarters:

Revenue went from $11.3 billion to $41.5 billion. Gross margin went from 45.7% to 84.9% (!!!)

Each quarter beat the guidance Micron gave the quarter before, and the magnitude of the beat grew each time.

And here’s a mind-bender. Micron’s fiscal Q3 revenue of $41.5 billion was larger than its revenue in any previous full year in company history. Bigger than the $37.4 billion it booked in all of FY2025. Bigger than the prior record of $30.8 billion in FY2022.

Of course, what does that margin chart scream for folks who’ve been around the block? Peak. Memory is the most cyclical business in semiconductors. It’s like a sine wave:

That nice little rolling wave there is the semiconductor cycle.

I love how Doug O’Laughlin visualized the cycle here with various markets moving around on a merry-go-round:

Think of memory’s revenue wave as the weighted average of all the memory-consuming semis markets. Each one sits somewhere on the merry-go-round and carries weight equal to how much memory it buys, so the cycle you see in Micron’s revenues is really just the combined center of mass.

Which raises the question… which quadrant is that center of mass in now, and what would tip it into the next one (top right)?

Well, the merry-go-round still exists, it’s just that the mass of datacenter AI is sooo big that all the other kids on the merry-go-round don’t meaningfully contribute to the center of gravity as much anymore. AI is the dad who jumped on the merry-go-round with a couple of three-year-olds. Dad’s weight wins.

And yeah, I cropped that earlier Micron revenue chart for dramatic effect. Here’s what it looks like now:

YO! The amplitude of that sine wave is much bigger!

So let’s dig into that demand more and figure out if and when more supply is coming online. Can supply counterbalance, or will this memory supercycle persist?

Behind the paywall:

• Demand has two drivers… and don’t forget NAND: HBM is stacked DRAM, so it pulls many wafers off the DRAM market, right as server CPUs pull from the same pool. And AI is eating NAND too.

• The shortage goes out through 2027: CapEx up by half while the shortage horizon slid out a full year, and the construction-clock, node-shrink, and cleanroom limits behind it.

• What the contracts change: sixteen take-or-pay deals lock ~20% of DRAM and a third of NAND to 2030

• SK Hynix and Samsung say it on their own calls: the supply-demand gap widening into 2027, demand for the next three years above capacity, DRAM ASPs up ~60% in a quarter, HBM “sold out”.

• What still cuts the other way: ~80% of DRAM still reprices with the market, and Micron’s own guide already flags a “moderation in the rate of price increases”.

• The scorecard: the handful of variables that decide persist versus revert, each with a read and a date.

Read more

The coherent host is the most valuable. The traditional cloud CPU is the least.

Many readers asked if it matters whether the CPU is x86 or Arm.

Honestly, not as much as made out to be. But let’s go socket by socket.

Quick context

The ISA is the language a CPU speaks. Software gets compiled into that language, and a chip can only run code written for its dialect.

x86 has been the server default for decades. Yet Arm has been gaining in servers, first slowly, then quickly as Graviton, Axion, and Cobalt took hold in cloud, and now inside AI infrastructure as hyperscalers build Arm into their GPU server stacks.

Naturally, everyone asks which ISA is “better” for agentic AI; they’re both just fine.

The more interesting question at each socket is whether the software running there cares which ISA it runs on? Specifically, is the ISA a “moat” at any of the agentic sockets? Let’s see:

1. Coherent host: ISA is irrelevant

The coherent host’s moat is the coherent link to the GPU, not its ISA.

NVLink-C2C connects Nvidia’s Grace CPU to the Blackwell GPU at 900 GB/s, providing a shared address space in which the GPU reads CPU DRAM as if it were local. Vera doubles that to 1.8 TB/s with Rubin. Infinity Fabric ties AMD’s EPYC to the Instinct MI455X at comparable bandwidth. The coherent link is what makes this socket valuable. It’s what no other CPU can replicate without a bilateral design agreement with the GPU vendor... like NVLink Fusion...

Before Grace, Nvidia GPU servers shipped with standard x86 hosts (Intel Xeon or AMD EPYC) connected over PCIe. Grace Hopper (2023) was Nvidia’s first coherent superchip: Grace CPU (Arm, Neoverse V2) connected to the Hopper GPU via NVLink-C2C at 900 GB/s — and Nvidia’s first deployment of the full datacenter CUDA stack on an Arm server CPU. CUDA already ran on Arm through the Jetson embedded line, but this was the server-grade debut.

Grace Blackwell carried that forward; Vera Rubin extends it with a custom Arm CPU (88 Nvidia-designed cores) at 1.8 TB/s to Rubin.

So clearly, ISA isn’t a differentiator for the 800-lb gorilla. Host software runs on either.

What about AMD? ROCm is effectively x86-native. AMD’s coherent platform is built around EPYC, so an Arm port has naturally never been a priority.

The main takeaway is that the ISA is baked into the accelerator platform choice.

NVLink Fusion is Nvidia’s move to open the coherent-host socket to third-party CPUs. Previously, the only CPU that could claim a coherent seat on Nvidia’s backend was the one Nvidia built (Grace/Vera). NVLink Fusion allows other vendors to couple their processors to Blackwell GPUs over the same high-bandwidth coherent link Grace uses. Note that no NVLink Fusion product has actually shipped yet, these are simply announced partnerships. But the partner list includes Qualcomm (Arm), Fujitsu, Intel (x86), and SiFive (RISC-V).

If and when these ship, the coherent-host socket will be accessible to any ISA, so the moat is most definitely not the ISA. RISC-V even... although lots of software porting required.

2. Standard host: ISA nearly irrelevant, and eroding

The standard host’s job is to keep the GPU fed: tokenize inputs, batch requests, stage data over PCIe, manage memory. The CPU needs to work as fast as possible and also move a lot of data. PCIe can become a bottleneck here… hence the coherent host.

The hyperscalers started with x86 standard hosts paired with their XPUs, but that has moved toward Arm. AWS pairs Graviton with Trainium. Google pairs Axion with its gen 8 TPUs.

The feed-the-XPU stack runs on x86 or Arm interchangeably; ISA is not the moat.

Note that there is still an x86 standard-host business in smaller deployments, specifically enterprises and small neoclouds running DGX, Instinct MI355X, RTX Pro 6000 servers, and so on.

In these setups, the host often runs double duty with GPU feeding and application-tier workloads on the same box. That brings legacy x86 software dependencies back into the picture, and ISA does matter. Lower volume, but will grow.

Takeaway: if the host is doing double duty as application processor, then ISA matters. Otherwise, nope.

The two orbits closest to the GPU give the same answer: ISA does not matter there. The three that remain do not all agree. One has a real x86 lock-in story. One has a wrinkle. One… not so much.

3. Thinker: ISA irrelevant, and the category is still forming

The thinker is the GPU-coupled agent: reasoning-heavy work that runs close to the GPU on the scale-out backend, passes large contexts to the model, and needs per-core performance and low round-trip latency above all else. Real-time world model controllers, agents driving tight perception-action loops, anything where a single network hop eats the frame budget.

The thinker commands a higher ASP than the doer. That premium is earned on per-core performance and bandwidth, not protected by a proprietary link. The scale-out backend runs over open InfiniBand or Ethernet, reachable by any NIC-equipped CPU. AMD Venice Classic (Zen 6, high-frequency cores), Intel Diamond Rapids (P-cores, 2027), Nvidia Vera, Qualcomm Oryon, and Arm AGI in its thinker-flex configuration can all reach it. Every major ISA in the same fight.

The software stack for thinker agents consists of Python, orchestration frameworks, and custom agent runtimes built over the last two years, running in containers and targeting both architectures from day one. There is no installed base of thinker software to be locked into, because the thinker socket barely exists at volume yet. Most agentic work today is coding agents, and that is a doer workload. So when the thinker software stack gets written at scale, it will be ISA-agnostic by default.

Hence, the ISA question is already settled before the volume arrives.

4. Doer: ISA matters, with a specific wrinkle

The doer is the most valuable orbit where ISA matters.

The doer is the action agent: coding, tool use, web search, API calls, compiling, running tests, filing pull requests. The metric is threads per watt — run as many agents as possible per rack. The agent’s own orchestration code is ISA-agnostic. Python runs anywhere. The frameworks that drive agents are all multi-architecture.

Yet the ISA matters in the execution environment the agent spins up.

If the target codebase compiles x86 binaries, e.g. a C++ service, a Go application, a Rust binary targeting x86 Linux and runs x86 test suites, you need x86. Yes technically you could use an x86 CPU emulator on Arm but that would kill performance.

So the doer’s ISA dependency is a dependency on what the target codebase compiles to, not on the agent runtime itself. A coding agent deployed to assist a Python web shop has no meaningful ISA dependency, but a coding agent deployed against a legacy enterprise C++ codebase does!

This matters for companies running agents against their own internal codebases. Their internal build pipelines, CI systems, and test suites were likely written assuming x86. Years of accumulated toolchain dependencies. The long tail of enterprise repos still carries enough x86-specific build logic that moving to Arm would require real migration work. That’s where ISA creates genuine friction today. LLMs make it easier than ever, but it’s still work. Maybe Fable can one shot it though...

So doer agentic CPUs, especially in bigger enterprise deployments, have an ISA dependence.

5. Traditional cloud: the strongest moat

Agentic AI is also driving traditional cloud workloads. Agents ultimately query ERPs, hit databases, and call APIs, all of which run on general-purpose cloud CPUs.

This is the orbit with real x86 legacy lock-in. Proprietary enterprise software from the 2000s and 2010s shipped as x86 binaries, sometimes with no source code and no porting path.

Definitely an ISA moat here. But the least value capture in the agentic AI path. Moreover, this ISA moat is shrinking over time.

But it’s real, especially for enterprises.

Where it lands

x86’s moat lives at the outer orbits: the traditional cloud socket, where old software licenses and ISV certifications built a real moat, and the doer socket, where it depends on what the agent actually executes.

The playing field is level at the inner orbits where the software is new, the frameworks were built for containers, and competing on power efficiency matters more than installed base.

The most valuable socket, the coherent host, does not care about the ISA. It cares about which accelerator agreed to share a coherent address space with you.

The closer you get to the GPU, the newer the software and the less the ISA matters.

So, who wins?

Every one of them will tell you it’s them. I’ve spent a lot of time listening to executives at all of these companies, and each frames the comparison around the socket where its own chip happens to win. The market, meanwhile, sees the crowd of launches and weighs them all the same. Up and to the right! But these are really several different CPUs competing for different jobs, and they don’t all carry the same ASP or capture equal value; some sockets are near-monopolies, others are headed for a price war.

To cut through the positioning, I needed a mental model to help answer my questions. Which sockets matter? Which specs matter? Who competes where?

That model is what follows.

Framing: The CPU Sockets that Orbit the GPU

Let’s start with the GPU at the center of our model. Don’t let anyone tell you otherwise. CPUs are not the center of the AI universe; GPUs are. Also, I’m using GPU interchangeably for AI accelerator / XPU / GPU.

In this model, the orbits represent the CPU’s jobs to be done. Each job creates a socket a CPU can fill, and the closer the job is to the GPU, the more valuable that socket is. I know you have lots of questions. But aren’t CPUs general-purpose? Does “closeness” to GPU truly matter? We’ll get to those.

1) First orbit: the Host CPU

Every GPU server is built around a host CPU, usually one or two sockets, that the GPUs attach to. It’s also called the head node, and its job is to run the GPU driver, launch kernels, tokenize and stage data, manage memory, and keep the accelerator fed. Really, it’s just “do whatever’s necessary to keep the accelerator fed.” Some people joke the head node is a glorified memory controller.

The host CPU sits in the token path. Thus, the host CPU must NEVER be the bottleneck. Stalled GPUs are insanely expensive, and the host exists to make sure that never happens.

The specs that matter for a host CPU follow from that fact: high per-core performance (so it can keep up with the GPU and decide what to do next quickly), high bandwidth to the GPU (so it can move data without stalling it), and enough memory bandwidth and capacity to stage what the GPU needs. Core count is secondary.

Nuance: coherent host vs. standard host

Let’s get a bit nuanced.

The host orbit splits in two on the question, “Does the GPU need to use the CPU’s memory as an extension of its own?”

Early in the training era, the answer was no, and the link was PCIe. The host CPU stages data, the GPU copies it across the bus, the two keep separate memory pools. PCIe Gen5 moves about 64 GB/s per direction, Gen6 about 128 unidirectional. That’s fine when the host is just feeding tokens.

Reasoning models changed the calculus. When a model thinks before it answers, its KV cache (the attention state it holds while generating) balloons, and when it spills out of GPU memory, the GPU has to tier it into CPU DRAM and read it back fast, inside the generation loop. PCIe bandwidth is no longer enough to keep up.

So Nvidia built a coherent link, NVLink-C2C, that presents a shared address space, letting the GPU read CPU memory as if it were local. That is Grace Blackwell: a Grace CPU and a Blackwell GPU fused into one coherent module, sharing data at 900 GB/s, about seven times what PCIe Gen5 could move (Vera doubles it to 1.8 TB/s).

The coherent host is the head node rebuilt for the reasoning era, and as context grows, more of its value moves across that coherence line.

So if we split the host orbit to account for the nuance, it looks like this:

Closest in is the coherent host, tied to the GPU over a coherent link and built for KV-cache tiering and long-context reasoning. It is the most valuable seat and the most proprietary because that link exists only between an accelerator and the CPU it was designed with.

A step further out sits the standard host, tied to the GPU over ordinary PCIe and handling the tokenizing, batching, and feeding. It is less tightly coupled, but it has one thing the coherent host does not. It can sit in front of any accelerator, since PCIe is universal. That makes the standard host open, modular, multi-vendor territory.

2) Second orbit: the Agents CPU

Agents are the new workload, and today’s agents are mostly not a GPU job. An agent is given a task, and it loops. It asks the model what to do, gets back an instruction, executes it (run some code, query a database, search the web, call an API), observes the result, and asks again.

The model’s thinking is one step in that loop (on the GPU).

Everything around it, the parsing, the tool calls, the code execution, the state management, runs on a CPU. That work would overwhelm the host CPU if you let it.

The host is in the token path and cannot afford to be busy doing anything else, and there are now thousands or millions of agents pinging the model, not a handful of humans. So the agent work gets offloaded to standalone CPUs dedicated to running agents. The goal there is different from the host, namely to run as many agents as possible in a given power footprint.

The metric is threads per watt, not raw per-core speed. Lots of cores, lots of threads, low power, enough cache to keep each agent’s small working set on-chip.

This orbit is where the CPU demand is exploding. It is already most of the reason the CPU-to-GPU ratio in AI infrastructure has moved from roughly 1:4 toward 1:1.

Nuance: thinkers vs. doers (GPU-coupled vs. CPU-bound).

Not all agent work is the same; some should run in the same datacenter as the GPUs, but much needn’t. Vik’s Yellow Pages splits CPUs on exactly this axis, reasoning vs. action, and the same split applies to the work itself.

The doers (CPU-bound, action agents). Here, the heavy lifting is on the CPU, and the GPU sits idle while it waits for the CPU. Coding is the most popular and a great example. Agents write code, compile it, run it, open a pull request, etc. Compiling and running can takes seconds, which dwarfs any network latency. So the extra hop for the GPU to talk over the front-end network to a CPU rack in another datacenter is no big deal. Hence, the doers can leave the GPU data hall and run on cheaper CPUs somewhere else.

The thinkers (GPU-coupled, reasoning agents). Here, the heavy lifting is the model’s thinking, and per step the agent is mostly waiting on the GPU. A reasoning-heavy task that ships a large context to the model every step, then waits, is GPU-coupled. This work aims to stay close to the GPU, on the scale-out backend network inside the data hall, so round trips and large context transfers remain fast. And because each agent mostly waits on the GPU, one fast core can host many of them, so this socket prizes per-core speed first and core count only second.

Take, for example, agents that drive real-time generation and world models. Fei-Fei Li’s World Labs frames a world model as a renderer, a planner, and the real-time loop connecting them, the same perception-action cycle that drives an embodied agent. The GPU renders a frame, a CPU reads the user’s or agent’s input and conditions the next step, and around again, all inside a frame budget of 16 to 33 milliseconds (30 to 60 FPS). A single network hop eats that budget, so the controlling CPU has to sit on the backend, next to the GPU. It is a reasoning job, not an action one, so the CPU does little computing but has to be fast and close. Thus, we need high per-core performance and low latency, not necessarily high thread count.

And as the world’s persistent state grows the way a long-context KV cache does, it pushes toward the coherent host. As AI moves from words to worlds, more of the agentic workload lands in this tightest, most GPU-coupled corner.

Side note: The strongest case for keeping even the doers on the backend is the agent’s growing context. A long session accumulates a large context, and every turn the model attends over all of it through the KV cache, the attention state it builds from those tokens. Recomputing that cache each turn is wasteful, so the inference system keeps and reuses it, tiering it out of GPU memory into a dedicated context-memory rack as it grows (Nvidia builds this with BlueField DPUs, fast storage, and KV-aware routing in Dynamo). But that pulls the KV cache onto the backend, not the doer. The agent’s memory is really just tokens, and its durable memory (files, a vector database, retrieval stores) never touches the GPU at all. The doer ships a prompt and a tool result, text in and text out, and the inference system turns that into KV and caches it. So the context-memory rack is real backend infrastructure, and the doer’s compute can still leave.

One caveat: if the work produces a very large artifact the model then has to reason over, you keep it near the backend so the big payload does not crawl across a slow link to reach the GPU.

3) Third orbit: traditional cloud CPUs

The outer orbit is the CPU as we have always known it, the general-purpose server CPU running web services, databases, microservices, and VMs.

Cloud CPUs are not in the AI loop directly. But agents often reach out to all of it; the ERPs, the CRMs, the databases, the web servers, and so on. And the more that agents multiply, the more ordinary cloud-CPU demand they pull along behind them.

Nuance: there is a portfolio here too

I know you all know this, but to be pedantic:

Traditional cloud buyers have always chosen cloud CPU SKUs by workload. General-purpose, compute-optimized, memory-optimized, storage-optimized, etc. A per-core-licensed database wants fewer, faster cores, because the license is priced per core. A web tier wants density and throughput per watt. That same split, fast cores vs. dense cores, runs straight through every vendor’s lineup, and it is why “the cloud CPU” was never one SKU.

Where is the Value and Who Competes at Each Socket

Put it together and you have a spectrum of CPUs that all benefit from agentic AI, but not in the same place or to the same degree. Map them onto the orbits above, and each lands in a different socket: coherent host, standard host, thinker, doer, or traditional cloud, and some in more than one.

Those sockets are not worth the same. One is close to a monopoly; the rest will be in a price war.

This is what current head-to-head comparisons miss. Every vendor counter-positions, framing the whole “agentic CPU” question around the socket where its own part wins. Nvidia leans on the coherent host and thinker, AMD and Arm on the doer’s rack-scale density, and so on. The marketing then reads as if everyone leads, because each is measuring a different socket.

To know who actually captures value, you have to ask two things of each chip. Which socket is it really built for, and what is that socket worth?

Behind the paywall, I answer both.

First, the money. What each of the five sockets is worth, which one is a near-monopoly, and which are headed for a price war.

Then the field. Where Nvidia’s Vera, AMD’s EPYC, Intel’s Xeons, Arm’s AGI rack, and Qualcomm’s CPU each land, who owns the moat, who is the most complete, who is shut out of the one socket that actually pays, and who is renting a way in.

Read more

Why does the loss compound? Conduction loss in copper scales as I², not linearly. Triple the current, ninefold the loss.

Let’s pencil it out. Vcore (the compute logic supply voltage) is locked at 0.6 to 0.8 V by transistor physics; we’ll use 0.7 V. Given that P = VI, we can estimate the current draw for a few Nvidia GPUs:

Plug the currents into I². Hopper to Blackwell is ~3x the loss (1,714² / 1,000² = 2.94). Hopper to Vera Rubin is ~11x.

And conduction loss is only half the story. Transient voltage droop, which gets harder as workload transients steepen, adds a second loss term on top.

The only way out is to shorten the high-current portion of the path.

Power delivery is splitting into two domains. The rack-to-board step (48 V to 12 V) stays where it is, because at those higher voltages, the current is still low enough (tens to a few hundred amps) for standard motherboard copper to handle without overheating. The board-to-die step (12 V to ~0.7 V) is where current explodes past 3,000A for a Vera Rubin, and that’s the step that has to migrate from the motherboard onto the package substrate, and eventually under the die itself.

Intel said as much at ISSCC 2026 in February:

Last week, Analog Devices (ADI) agreed to pay $1.5 billion in cash to acquire Empower Semiconductor. Empower ships kilowatt-class power delivery as SiP modules that mount on the package substrate next to the compute die. Short lateral paths, in-package magnetics, but the active silicon still sits beside the SoC rather than embedded inside the substrate. A real step in the right direction, and ADI just paid $1.5 B for it.

Twenty years of IVR attempts have left one architectural step still open beyond Empower: a merchant, on-package, in-substrate, monolithic-magnetics IVR chiplet. A startup called PowerLattice is aiming at exactly that slot.

In this post, we will look at:

• How AI accelerators lose power before they compute. The two loss mechanisms worked from first principles.

• Intel’s published ISSCC 2026 loss budget for a 5 kW SoC.

• Why ADI just paid $1.5 billion, and the impact of Nvidia bumping Vera Rubin from 1.8 kW to 2.3 kW.

For paid subscribers:

• What Intel and Empower build today

• How PowerLattice’s architecture goes further

• Why transient response matters a lot

• Which power-IC incumbents lose their AI socket if the architecture lands, which are partially protected, and which benefit either way

• What about Nvidia?

How AI Accelerators Lose Power Before They Compute

To keep current manageable, data centers step voltage down in a cascade so the highest-current section is the shortest:

Yikes! That last step, 12 V → 0.7 V at the Motherboard Voltage Regulator (MBVR), has crazy current flowing through it! In current designs, the MBVR sits outside the compute die, either on the PCB just outside the package, or on the package substrate flanking the SoC. Either way, thousands of amps travel horizontally through copper traces to reach the compute die. That lateral path is the Power Delivery Network (PDN), and every millimeter of it loses power two ways:

1) I²R conduction

P = I² × R. Focus on current squared.

Say you have a 200 microohm section of the PDN carrying 1,000 A. That dissipates (1,000)² × 0.0002 = 200 W.

Next generation you roughly triple the current. Same path, same resistance: (3,000)² × 0.0002 = 1,800 W. Nine times the loss!

2) transient voltage droop

AI workloads jump from idle to full power in tens of nanoseconds. The MBVR sits centimeters of high-current lateral copper away from the die. Inductance in that lateral path means voltage at the die sags before the MBVR can respond. If the sag dips below the logic minimum, errors are introduced.

To avoid this, designers add a guard band. So the logic might need 0.75 V, but the MBVR supplies 0.95 V (200 mV of margin):

Because dynamic power scales as V², that means we need say ~1.6x dynamic power to avoid droop problems. (0.95² / 0.75² ~ 1.6x)

The ideal fix is move the regulator from centimeters of lateral substrate copper to micrometers of vertical pillar directly under the die.

Putting the IVR (Integrated Voltage Regulator) directly under the load collapses both losses at once. I²R drops because the high-current path shortens by orders of magnitude. Droop drops because proximity allows much smaller, faster inductors and capacitors, so the regulator responds much quicker

Intel’s ISSCC 2026 Chart Puts Numbers on the Problem

At ISSCC 2026 in February, Intel’s Kaladhar Radhakrishnan presented “Integrated Voltage Regulator Solutions to Enable 5 kW GPUs.” Check out this slide that shows the waste that happens to a conventional MBVR architecture as GPU power scales from today toward Intel’s end-of-decade forecast of 5 kW per chip:

At 1 kW per GPU (Blackwell B200 territory), today’s MBVR architecture is fine. The system draws 1.25 kW from the wall to deliver 826 W of useful compute, for 66% efficiency. I²R loss is 89 W. Droop waste is 174 W.

But at 5 kW per GPU, the architecture struggles. The system now pulls 8.3 kW to deliver just 3.5 kW of useful compute. Efficiency has fallen to 42%. I²R loss has grown from 89 W to 2.2 kW! That’s a 25x jump from only 5x more current, because the loss scales as I².

Droop waste has grown from 174 W to 1.5 kW too.

Roughly half the system input is now burned as heat, not as useful compute.

Intel’s has ideas for alternatives, for example an in-package landside IVR. Landside meaning mounted on the bottom of the package, opposite the compute die, where the BGA balls connect to the PCB.

Intel calls one version a C2VR (Continuous Capacitive Voltage Regulator).

Applied to the same 5 kW SoC, the losses are so much smaller:

I²R loss on the input path is reduced from 2.2 kW to 476 W. System input falls from 8.3 kW to 6.8 kW. Useful compute rises from 3.5 kW to 4.1 kW.

Same compute job done with 1.5 kW less wall-socket power!

This is a $1.5B problem

Just last week, ADI announced an all-cash $1.5 billion acquisition of Empower Semiconductor, a maker of silicon capacitors and the Crescendo kilowatt-class vertical power delivery platform. It’s built as System-in-Package (SiP) modules (multiple silicon dies bundled into one package) with integrated magnetics that mount on the package and scale to 2,600 A+ peak current:

My initial take is “Nice! A $1.5B price tag! IVR is an important problem space!”

Of course, Nvidia faces the same problematic physics. Vera Rubin was originally specified at 1.8 kW per chip. In late 2025, SemiAnalysis reported

Supply chain rumors have indicated that there are 2 different “SKUs” with different power and performance profiles: a Max-P variant at 2,300W and a Max-Q variant at 1,800W. However, these are not distinct hardware SKUs but the 2 default power profiles that Nvidia is offering users based on their workload needs. Max-Q is what Nvidia believes offers the best performance per Watt. Max-P offers the greatest absolute performance though this would come with an efficiency penalty. Running the Max-P setting results in a 20% increase in rack power draw but the performance gain fall well short of this 20% power consumption increase.

As we discussed, there are tradeoffs with the 2300W TDP. More watts at fixed Vcore means more amps. More amps mean more I²R and more droop. Which means they really need on-package IVR!

PowerLattice

In November 2025, a startup called PowerLattice emerged from stealth with a $25 million Series A jointly led by Playground Global (where Pat Gelsinger is now a General Partner) and Celesta Capital. The three founders came out of the Qualcomm/NUVIA group.

Peng has a long history of power delivery design, including a 12-year stint at Intel with many patents awarded. Per Pat Gelsinger, it’s a dream team:

“There are very few teams and people that can do it,” said Pat Gelsinger, general partner at Playground Global. “We have assembled what I’d argue is the dream team of power delivery.”

PowerLattice’s innovation is the Rainier micro-IVR, a monolithic, vertical-design silicon die combining proprietary on-die magnetic inductors, advanced control circuits, and a programmable software layer. The chiplet brings voltage regulation from inches away on the motherboard to within hundreds of micrometers of the compute die, eliminating most of the lateral substrate copper that bleeds power to I²R and droop.

The architecture scales by ganging multiple chiplets in parallel, each at a 5 A/mm² current density, with a low-hundreds-of-micrometers z-height. Z-height is thin enough for land-side mounting, substrate embedding, or interposer embedding.

PowerLattice claims >50% reduction in effective compute power, an order-of-magnitude lower power noise, lower cooling, longer processor lifetime, and 2× or more performance per watt where AI compute is data-center-power-constrained.

First chiplets are being produced at TSMC; customer testing planned for H1 2026. Which should be roughly now.

Can PowerLattice compete? What does this mean for ADI+Empower? And other public power-delivery semi companies?

For paid subscribers: Intel and Empower architectures, how the PowerLattice architecture goes beyond, and what it means for publicly traded power-delivery incumbents.

Read more

Deepu’s team builds the platform that essentially every robotics company on the planet uses across the three computers that physical AI requires: training in the data center (GB300, Vera Rubin), simulation (RTX Pro 6000, Omniverse), and the runtime at the edge (Jetson Thor and Orin). Roughly 2.5 million developers and more than 10,000 companies are building on Jetson today, but the industry is still shipping only a million or two robots a year against an opportunity Deepu pegs at tens of billions.

Subscribe now

In this interview, we walk through physical AI from first principles. A few things that surprised me:

• Agentic AI isn’t just a digital (agent) story. The industrial business of the future runs on fleets of robots of different embodiments, people, and digital AIs — all needing orchestration. You don’t validate that orchestration on a live manufacturing line, which is why Nvidia built Mega: a blueprint for simulating an entire factory’s worth of agents in a digital twin before you touch the real one

• The industry has marched from VLMs to VLAs to world models in the past few years. World models matter because when a robot moves an atom, the rest of the world reacts, and you need to model that reaction, not just the action. “Necessary but not sufficient” is the new consensus

• Edge robots won’t run in isolation. Deepu expects hybrid edge-cloud as the default, where the robot does as much as possible locally but can “phone a friend” to the cloud for long reasoning. You never have enough compute at the edge

• Every robotics application has a “10-second mark” — the qualifying time before you’re even in the competition. Self-driving cars have hit theirs in the last six to twelve months thanks to two unlocks: end-to-end models replacing stitched-together specialist models, and reasoning that lets a system handle scenarios it never saw in training

• Most of the spend at robotics startups today is not on edge deployment, it’s on training and simulation. Until accuracy is solved, there’s no point scaling deployment, so the action sits in the first two computers

• Simulation finally works for robotics because the sim-to-real gap has closed enough to be useful. Nvidia open-sourced Newton — a physics engine built with Disney Research and Google DeepMind specifically for robotics — to push that further

We also cover Nvidia’s new data-center-vs-edge reporting structure and why Deepu thinks the next manipulation and locomotion tasks will hit their 10-second mark in the next year or two.

This interview is lightly edited for clarity.

Why Robotics Is Suddenly Possible

Hello listeners, today we have a special guest, Deepu Talla, VP and GM of Robotics and Edge AI at Nvidia. Welcome, Deepu.

DT: Austin, hi, good morning.

Good morning, thanks for coming. I’m super excited to talk to you today about physical AI and robotics and all things edge AI. A lot has changed in the past two years or so. Listeners both understand that physical AI is here, physical AI is coming. Jensen has said the ChatGPT moment for physical AI is here and coming. On the other hand, most people aren’t experiencing it in their day-to-day life. And so there’s differences from the data center portfolio to the edge portfolio. I look forward to unpacking it with you just to help everyone have a better intuition, understanding, pulse of what’s actually happening out there.

DT: Yeah, absolutely, sure thing.

So I teed up a list of questions, so we’ll jump right into these. My first question for you, set the stage for us. A lot has changed obviously with the rise of LLMs, which everyone knows, but more specifically at the edge with VLAs, vision language action models. Could you talk to us about why is robotics suddenly possible? What are the technical changes that have happened that make it so that all of a sudden there’s all these humanoid startup companies and self-driving taxis are suddenly really good? What has changed?

DT: If you think about the opportunity itself, we’ve known it for, gee, what, 50 years, growing up watching Star Wars and Star Trek and the need for robotics, physical AI has always existed. Whether it’s for doing dangerous jobs or labor shortage and so on. But the technology has not been good enough. There’s fundamentally two things that you need for bringing physical AI and robotics into the real world.

The first one, of course, is the model or the algorithm or the technology needs to be accurate enough. Intelligent enough. If it’s not able to do the job well, accurately, then what’s the point? In the physical world, because there’s no human to back it up, the accuracy requirements are extremely high compared to the digital world. For example, if you’re using ChatGPT or Claude to summarize an email for you or compose an email for you, it does a pretty good job. It’s getting better and better, but you will go off in the last one or two percent. You’re going to tweak it and make it right and ship it. But in the physical world, that can happen. If you want a robot to do some manipulation tasks and finish some things, humans are not going to be backing it up. So the accuracy requirements are 99 point — how many nines after the 99 point? Depending on the application, a self-driving car is probably somewhere between eight and 10 nines of accuracy that you need. A surgical robot, surely we would want to be much more than that. If it’s a consumer robot in the home, maybe four or five nines might be okay. So that’s the first thing. That’s why it’s been super hard. Technology has not been good enough to make it accurate, number one.

Number two, let’s say you created a super accurate robot. Now that needs to get integrated into the physical world. In many times what happens is that robot is working with other robots or humans or other processes that are happening in the real world. For example, in a manufacturing setting, there’s many other things that are happening. It needs to integrate very well into that existing workflow or system. It’s kind of like you hire an engineer or an employee into your company. They’re brilliant. That’s why you hired them. But they also need to be equally good at integrating with the rest of your employees so that they can be more productive.

And that is actually a pretty big problem we haven’t been able to solve, because typically what happens is when you integrate a robot or autonomous operation into your existing workflow, you have ERP systems, you have warehouse management systems, you have security systems, you have PLCs, programming logic controllers for these robots, many of which could be 10, 20 years old on different software. And it’s super hard for humans to build that glue logic, if you will, to bring all those things together. Now, luckily, in the last three months with the rise of agentic AI and coding basically becoming easier and easier for these agents to solve, it brings us great hope that once you solve the accuracy, the integration piece is also going to be solved reasonably well.

So what I hear you saying is, for LLMs when humans were using them to start, the human’s in the loop. Maybe you don’t have to have quite as high of a need for accuracy. Even with some of the integration pains, the human could copy from here and paste over there. But it sounds like a lot of physical deployments, maybe you don’t have a human in the loop, so you do need better accuracy or more intelligence, but then also yes, there’s an integration challenge of how do you actually make this useful in the workplace or in an industrial setting or something, as opposed to just the toy chatbot stuff that we did early with LLMs.

DT: Yes, right. The analogy that I use with my team in general is imagine you are a 100 meter racer. Your goal ultimately would be to win the Olympic gold. But before you win the Olympic gold, you need to qualify for the Olympics and you need to hit a certain time. In the case of men, it’s roughly 10 seconds. It’s incredibly hard to hit 10 seconds. But unless you hit 10 seconds, doesn’t matter. You’re not going to qualify for the Olympics and you can’t get there. And of course, in order to win the Olympics, you probably have to hit 9.7, 9.6, eventually. But 10 seconds is the golden mark.

So for each application, I believe there is a 10-second equivalent. Until you hit that, you’re not in the game. You keep trying. And if you look at all the physical AI and robotics applications, almost in every case, we haven’t hit the 10-second mark. I believe we have recently hit the 10-second mark in autonomous vehicles. You’ve got to keep asking yourself, how is it that suddenly in the last six months to a year, there are so many Waymos out there, suddenly Tesla self-driving has hit that 10-second mark, if you will. Now that doesn’t mean the 10-second mark is good enough. It’s just that it puts you in the game. Now it’s all about scaling to hit that 9.7 and really go off. So you’ve got to ask yourself what changed? We’ve been trying this for 10, 15 years, but suddenly something happened to hit that 10-second mark.

I think there’s two things that really happened for self-driving cars. Number one, end-to-end models. Until very recently, until a couple of years ago, since 2015 to 2023-ish, it was all about building specialist models for whether it’s lane detection, whether it’s path planning, whether it’s for sign detection, whether it’s for all those kinds of models. You would have these 20 different so-called specialist models, and you’d put them together, and they would kind of work. They would be brittle because you would never be able to solve the long tail problem. They’re not quite the 10-second mark. They’re probably the 10.5-second mark, which is great, but not good enough. So end-to-end models is one.

And then secondly, what we’ve seen in the digital AI world in the last one year is reasoning has become extremely important, kind of like humans today. How is it possible that some 16-year-old who gets a license, you can practice for 10 hours and you are on the road, and you’re driving similar to somebody who’s been like 30 years experience, with millions of miles experience potentially on the road? Because we do reasoning. We haven’t encountered all the scenarios in our training dataset, but we are able to have some intelligence and then we reason about it and then we act appropriately. So because of that, end-to-end models and reasoning, self-driving cars hit the 10-second mark.

Now, then you expand it into what are the robotics applications similar to that that we can get there. You go to the ultimate application, the extreme right goalpost — humanoid robotics, let’s call it general purpose with fine-grained dexterous manipulation with so many degrees of freedom. You can navigate anywhere. You can manipulate any object from rigid bodies, which is easier, to soft bodies and fluids which requires extreme physics simulation, and you need to do all that analysis. Those are the increasingly hard problems that we need to solve and technology is evolving to get there.

But the left goalpost, if you think, is the self-driving car. We’ve gotten good enough. You’ve got to ask yourself, what is the next one that feels like we are reasonably good enough technology between end-to-end models, between reasoning and simulation, of course, because you have to test it in simulation. It’s too dangerous and too expensive and too slow to test it in the real world. What are the applications? You can start to feel like you can start to see that a lot of off-road delivery robots, you can see things like autonomous mobile robots and industrial environments. You’re starting to see this kind of getting deployed.

And then the next wave you can think of like video analytics as an application, which is cameras and outside-in. We think of them as robots. Typically when you say robot, most people think of a robot like a human, a humanoid or an AMR — sensors and actuation are on the robot, and we do perception inside out. Because we have sensors on us, we look inside out and then we process it and then actuate. But there’s also an outside-in robot, kind of like a traffic controller. Kind of like, GPS in your car somehow tells you even though you don’t know what the route is 500 meters away or 500 yards away, what the traffic looks like. That’s coming because of outside-in from other agents that are being spatio-temporally analyzed and you’re combining all of that information. If you did that, you can solve all of the safety applications. You can do situational awareness using cameras and other sensors in a building or a factory or a city and so on. So that’s how we are seeing it.

And it’s amazing right now the pace at which these models are evolving — what started with language three and a half years ago, moving to vision, vision language models, to multimodal, add reasoning on top of it, went to vision language action models, and now we’re seeing world models. Because an action model takes an action and does some manipulation, but when you take an action and do some manipulation in the real world, the real world is moving. Atoms are getting moved, something is getting changed. And then the world reacts appropriately because of the change, and you need to be able to model that. That’s where the world models are coming in. I’m sorry, long answer, but I’m just super excited as you can tell how fast this technology is evolving.

From Specialist Models to World Models

No, this is really, really helpful. I heard a couple key unlocks that have happened recently. You talked about simulation and world models, which we can touch on, but you also talked about end-to-end and reasoning. Let’s dive into the simulation and world models. Tell the listeners a little bit more about that, because I don’t think a lot of people have spent a ton of time thinking about these. What is happening in that space over the past two years where that’s also enabled a key unlock, for example, for the sort of easier or the first thing that we’re all experiencing, which is the self-driving cars. What’s different from 2020 or 2023 and today in the simulation world model space?

DT: Until ChatGPT happened in November 2022, the technology we had mostly was convolutional neural networks and transformers, of course, but we were building so-called specialist models for a specific task. They would kind of work, but the world, especially in physical AI robotics, because the accuracy requirement is so high and also it’s very hard to maintain the world in a structured manner. If you maintain the world in a structured manner, you exactly define each component is arriving at a certain time and you program that robot and put a model, you kind of solve it. But that’s really solving 1% of the real opportunity. That’s where technology was.

When ChatGPT came about, it fundamentally transformed from so-called a specialist doing a very narrow task to a reasonably good enough general purpose model. In the case of ChatGPT, it was trained on everything that we had on the internet using language and so on. It could do many jobs because it was a good generalist. It’s kind of like, you could potentially train a 10-year-old human to do some very narrow task and they’d be special and they could do it. However, they’re not very good generalists because they cannot do much more.

In humanity, we define a good generalist as somebody like getting an undergrad degree, for example. So a 21-year-old. That’s a reasonably good enough generalist because they have knowledge on multiple things. And then what happens is if you want to solve really important problems after that, you take that reasonably good enough general purpose brain and then you derive a specialist from them. It’s kind of like you hire an employee at 21 years old, very good generalist, but for the next 30, 50 years, they’re going to train in a specialty using the general purpose capability, not losing the general purpose capability, but becoming increasingly specialized in something. That’s when you can solve really difficult problems.

So until 2023, the technology was, before that, so-called specialist models, you would gang multiple of them together. You could solve some problems, but 1% of the opportunity and very brittle if you take it to anything else. Once ChatGPT moment happened, applied to language, what the physical AI robotics folks realized is, okay, hold on a second. We can use that same technology and use multimodal, because in the case of ChatGPT it started with language, but in the case of physical world, vision is one of our most important sensors. And of course there’s sound and then there’s touch and then all the other things, but vision is one of the biggest sensors that we use. So can we add video camera, of course you can add radar, you can add lidar, you can add ultrasonic, you can add speech, in addition to text as language as the modality? That’s what researchers started doing for robotics.

What came out of it in 2024 was so-called vision language models. And then they said, okay, you can analyze, understand the scene using computer vision. But what’s a robot if you don’t take action? Ultimately, you can analyze everything that you want, but you have to take some action. So they said, okay, fine, we have vision language model that’s understanding the scenario, but we need to take action. That’s when VLAs came about — vision language action model. You combine language, combine vision, you analyze it and then you take action. And that unlocked quite a few number of use cases in the last 12 months, especially use cases as it relates to relatively structured world and doing some sort of manipulation with rigid bodies. For a rigid body — if you look at this thing for example, it’s a rigid body and I can hold it this way, I can hold it this way. It’s not that complicated because I can apply a force or torque that’s between 1X and 10X and it kind of works okay. But when it’s a soft body, you can do it of course, because if it’s deformable, it squishes or it breaks and so on.

So today with vision language models, action and VLMs, we are able to solve those types of use cases which are relatively structured, but rigid body. So that’s where we are today. But then the realization is, well, that’s good, but still not good enough. Because even solving rigid bodies, you probably solve from 1% of the problems to 2 to 5% of the problems, let’s say. We need to get to 100. We want to solve general purpose robotic problems so that we can really expand.

So this is where world models come in. Because when I picked this bottle and when I moved it somewhere here, physics changed. The atoms got moved. The robot did it, but the rest of something changed in the world too. So you need to model that. What’s happening in the world because of this actuation needs to be modeled. If I place this bottle on a table here right beside me, but if I placed it at the edge of the table, it can fall and something happened as a result of that. So all of that scenario also needs to be modeled. This is why the industry believes — if you look at all the researchers right now, they started with language, went to VLM, went to VLA and now they’re all saying, necessary but not sufficient. We need to add a world model.

So now you hear these things called world foundation models. You see things like world action models, that’s one of the latest terms that you hear about. Which is essentially combining, in addition to the VLM, VLA, you add simulating the environment around you. In order to truly solve this problem, you need to install what you’re doing in the robot, but you also need to understand what’s happening in the world because it’s always going to be interactive. That’s where we are today.

The Three Computer Architecture

That was a great little history lesson that got us to today. So my question is, given that we’ve moved from specialist models to these generalist models that are now multi-modal, they can take action, ultimately to world models that can represent not only the action you’re taking in the world, but what’s happening in the world around you. What does that mean for edge computing? Does that mean we need way more memory, way more compute, more memory bandwidth, impacting your portfolio and your roadmap?

DT: If you think about robotics or for that matter edge AI, there’s fundamentally four steps. If you walk backwards from the last to the first, the last step, of course, is the deployment. That’s the runtime. In the case of a robot, the robot is operating at the edge, at the point of action, like a car or a humanoid robot. There are sensors and actuation. So you need a computer there. And because for latency reasons, for cost reasons, for connectivity availability reasons, safety and all of that decision-making, especially for physical AI robots, you want to do as much as possible at the point of action where there is sensing and where there’s actuation. So that’s the edge computer. And there’s a lot of work that we’ve been working on making that happen.

But before you do the deployment, the third step is you’ve got to test it. Until you’re sure that it’s good. And then the best place to test it is in simulation because it’s faster, safer, cheaper. That’s the third step.

Before you test it, you’ve got to train it. And this training is no different than training large language models. It’s typically done in a data center. That’s the second step, is training.

And the first step before you train is you need to have data. And data, unlike ChatGPT, large language models where there’s a corpus of everything that humanity has created in the last 50, 100, 200 years is reasonably well represented, it’s not well represented in the robotics world, especially when you’re talking about — you can see YouTube videos of dances, but you don’t see YouTube videos of extreme fine-grain, precise manufacturing tasks and so on. And even if you see that, there’s no physics modeled in that. You kind of see how it’s being done, but you don’t know what’s the force, what’s the torque, what’s the angle, what’s the best way? The trajectory planning? You don’t see any of that. So that’s the problem.

So we’re working on all of these steps right now because we at Nvidia, we don’t build robots. We’re building a technology platform that helps everybody building robots. We provide the core infrastructure and provide the acceleration libraries and workflows for data generation, training number two, testing and policy evaluation and simulation number three, and the last step is the edge computing deployment.

So your question of what does it mean for the edge computer? In order for robotics to really take off and scale, first you have to solve the accuracy problem and the integration problem. And today much of the action is in solving the accuracy problem. Until you solve the accuracy problem, there’s no scale out happening in the edge and deployments. That’s where we are today. Now let’s imagine we’ve solved the accuracy problem and the integration problem is going to be solved increasingly with agentic AI. Now you essentially come to the edge computer and you need to make it scale out from whatever, hundreds of thousands of robots today, maybe a million robots, to ultimately, if the vision comes true that there should be multiple robots per human like a C-3PO and R2-D2, kids will grow up with a robot and the robot keeps changing over time and the memories will stay forever. Billions of robots, if not tens of billions of robots, possibility in the future.

So if you look at it from that perspective, we are in less than 1% of that. Because we are barely shipping a million robots, the industry is shipping barely a million or two today, but the opportunity is in tens of billions. So you are 10,000 times away to get there. So what needs to happen? Of course, there’ll be many different embodiments as time goes by. But especially for the robots, like humanoid robots, that need to be reasonably general purpose and have good enough general purpose intelligence to do multitask and then also have a component of the brain that’s going to be super specialized in doing certain tasks better than anybody else, because you don’t need every robot to be good at everything at super everything. It is going to be super expensive and the mechatronics may not even allow it because you have to make some trade-offs. If you don’t have the right mechatronics on you, it’s unlikely for you to be able to do all sorts of jobs.

So you need faster edge compute, of course. Memory, especially in today’s world, you see with all the supply chain issues, memory capacity is a problem. So in terms of using the right amount of memory, optimizing, using the right numerical precision for getting the right accuracy, but at the same time, because edge computers are more constrained in terms of area, in terms of cost, in terms of power, you need to be much more efficient. And we’ve been on this journey for over a decade. In fact, the first Jetson was 2014. So 11 years plus into this journey.

And interestingly, Austin, the thing that I realized in all of this journey is, remember the four steps that I mentioned from data to train to test and simulation or deployment? When we started, the only technology we had was the deployment technology. In fact, that is the destination. The ultimate destination is to have the physical robot. And what I realized is that the slowest way to get to the destination is to work on that problem. Because there are not that many robots. But we didn’t have the technology for training at that time. We didn’t have the technology for simulation at that time. So these got built now. And we are seeing, as we work with literally every robotics company on the planet, of course, everybody has to build a physical robot to test. But there’s 1,000 times more activity happening in training and testing and simulation today.

Okay, this was really good. I asked really about the edge computer, but you zoomed me out, which was good, and said, hey, don’t forget it’s not just about the deployment, but it’s about collecting the data so that you can train a model, so that you can simulate it, so that you ultimately have the confidence to go deploy it. And to your point, you guys have been doing a lot of work in the simulation world, because obviously it’s cheaper and ultimately makes a more robust deployment if you can simulate all this stuff instead of having robots go out in the real world and either have to wait around for conditions or just get broken. So there’s a lot of learning in simulation. Can you maybe walk listeners, remind them, so this ties into the three computer business model — computers for training the model, for simulating, and then ultimately you’re deploying at the edge. Can you walk us through each of those and remind us what kind of platforms are people using? I think a lot of people are familiar of course with training but what about simulation and then could you walk us through maybe the portfolio at the edge?

DT: Absolutely. So you’re right. Robotics, we think, needs three computers, as you mentioned, because you need the third computer, which is the brain inside the robot — that’s the runtime. And we’ve been working on it the longest, believe it or not. So we have this portfolio of products called the Nvidia Jetson. It’s incredibly popular — over close to, I think, two and a half million developers on the platform. Our current generation is Thor and Orin. More than 10,000 companies have been building robots either shipping or in the process of developing and about to ship robots. Incredibly robust ecosystem with so many partners. And they go into all sorts of form factors, all sorts of embodiments from humanoid robots, to agriculture robots, to medical robots, to delivery robots, to drones, to video analytics appliances, to telepresence type of devices, you name it. The breadth of end equipments and industries that companies and developers have been leveraging has been amazing. So that’s the third computer.

And then we keep on improving the performance. If you think about when we first launched the first Jetson, let me see, it was 192 gigaflops of processing. Today the latest generation is two petaflops. So that is 4,000 times performance increase in roughly 10 years. And then along the way came AI and we support all the latest, greatest models. The beautiful thing about Nvidia, we’re fortunate because we share the same architecture, what you’re running in the data center, what ChatGPT or Claude or Gemini or anyone, Qwen or you name it, any model or Nemotron from us, everything runs on our GPU because it’s fully programmable. That same GPU is in our Jetson portfolio as well. So as a result, we can run data center type of models at the edge. It’s just a question at that point of do you have, what’s the number of tokens per second? How fast is it? What are the different trade-offs that you need to do? So that’s our third computer.

The first computer is where we do the training. You mentioned it, most people are familiar. It’s exactly the same computer that are in all the different clouds and different enterprises and all the different neoclouds, sovereign clouds and so on. Same GB300 is our current latest shipping product. There’s a lot of Hoppers and Grace Blackwells and then Vera Rubin about to ship in next quarter, coming up fairly soon. So that’s first computer. You train the data and training happens in that computer.

And then there’s the computer in the middle, which you asked, which is where you need to test it. And you want to test it in simulation. And people wonder about when I say you must test in simulation, people would be like, no kidding. Nothing new. In fact, we’ve been building chips for 30-plus years now and every chip before we send it out to tape out to manufacturing, we have 100% simulated, emulated it and we know it’s going to work. Without simulation, it’s impossible for us, because if you don’t simulate and test it and if the answer comes out wrong out of the manufacturing fab, you’re one year away. And can you imagine if you’re one year late on any of the products that we’re doing? We are making products every one year now, going into hundreds of billions of dollars of infrastructure and eventually trillions of dollars of infrastructure. So we know that it works. That’s why we do simulation, emulation for chips.

Then you ask the question, why are you telling me in robotics that simulation is so important? It’s a no-kidding. It turns out that the simulation in robotics, the technology was not as good — the sim-to-real gap is sufficiently large until recently that you can simulate all you want, but it’s not exactly representative of what happens in the real world. So you’re almost throwing it away. That has been the problem because remember in the case of robotics, the simulation, the physics and all of that is extremely complicated.

So now the technology has become reasonably good enough thanks to AI and thanks to our investments in the Nvidia Omniverse, which we’ve been working on for well over 15 years for all sorts of simulation, started with games first and then went into all sorts of general physics and chemistry and all the high-performance computing modeling. Because of that, we’ve been able to build this platform that now the sim-to-real gap is manageable for many tasks and increasingly that gap is getting closed with thanks to reinforcement learning and new physics engines. Recently we announced this open source physics engine called Newton — work with Nvidia and Disney Research and Google DeepMind, and it’s completely open. So this is truly the first physics engine being built for solving robotics problems.

A lot of work had to be done to create this second computer in the middle, and it’s Omniverse. And so our best computer today is RTX Pro 6000, and there’s different flavors of it, different RTX Pro versions of it, but that’s our flagship. And it’s available also through different clouds. It’s available in workstations and computers from all the different OEMs. So that’s the three different computers for training number one, simulation number two, and then the runtime.

And the last thing I would add is once you deploy a robot, your journey doesn’t end. That’s actually the first, because these robots are going to be in the field for 5, 10, 15, 20 years in some cases, and you would expect them to get smarter over time. Just like you hired an employee and they’re good to go on day one, but they’re going to be learning new skills and important things and new problems need to be solved in the next 20, 30 years. Which means this loop of data generation and training and testing and deploying, this is a forever loop. This flywheel is forever. Deployment is just the first step.

Where the Spend Goes Today

Okay, wow, this is so cool and so interesting. So RTX Pro for simulation, that’s interesting. Do customers — you mentioned they’re in various clouds. It probably depends on the customer and on the domain, but are customers ultimately buying a lot of this hardware or are they just renting it as the simulation needs are on demand? And I assume that because you said there’s this loop of you’re always trying to make things better, are robotics companies just kind of constantly training, simulating, deploying, iterating?

DT: Absolutely. Much of the spend, if you look at all the latest, greatest robotics labs or startups, who have raised hundreds of millions of dollars, a billion dollar valuation because it’s the toughest problem. Much of their spend today goes into training and simulation, because until you get a reasonably good enough, accurate model, why bother deploying at the edge and scaling out? You do want to deploy at the edge to make sure you’re testing right. But the scale of deployment at the edge initially is going to be limited until you get accurate. So much of the action today is happening in training and simulation. And it depends on — so the compute is absolutely available in all the clouds and neoclouds. So that would be renting, on a demand basis. And some of these companies are also able to build their own local on-premise cluster for both training and simulation. So it’s going to be a hybrid model depending on how much compute is required.

Sure, that makes sense. So maybe a timely business question. On the earnings call last night, Nvidia introduced new business units, kind of rolling things up differently. So there’s the data center and then there’s the edge. And data center was hyperscaler and non-hyperscaler and then there’s the edge. But when you’re talking about the three-computer business model, it sort of spans both of those. And you talk about, like early, like a startup, maybe they’re raising a ton of money and right now they’re investing a lot in training and simulation and then small in deployment, but eventually that will ramp. How do you sort of track that across the different ways that it rolls up?

DT: So the announcement is not about new business units, it’s a new way to report so that investors and analysts and everybody can understand our business better. Today, much of the action is happening in the data center and especially for digital AI, for enterprise AI. And then increasingly we’re starting to see, even though we started investing in this well over a decade ago for physical AI, in the next three to five years, we are expecting major unlock in technology for the whole industry. And as a result, the amount of compute that’s going to be consumed, whether in the cloud or an enterprise cloud or on-prem edge, physical AI robotics will span across all of these computers.

So my job and my team’s job at Nvidia is to essentially create the workflows, create the technology that all these companies that are building robotics — they could not be building a full robot, they might be just building a brain, they might be just doing simulation, or they might be doing sensing actuation — is to provide the technology that they can use to essentially build their product and our solution. And the technology that we deliver is going to span across a cloud — could be AWS or GCP or Azure or OCI — or it could be in a neocloud like a Nebius or a CoreWeave, or it could be through some local on-prem workstation or even clusters that are built, or it could be a Jetson Omniverse cluster. It doesn’t matter. So the way we think about it is, it doesn’t matter where the computer is. It’s all about creating the right workflows and that leads to the right computer usage appropriately.

Agentic AI and Fleet Simulation

Sure, totally, it makes sense. Going back, you talked about ultimately for robotics and physical AI to actually be used in the real world and useful — first, there’s the getting in the game, which I love the 100 meter dash analogy by the way. I love track and field athletics. So first, you’ve just got to get to the 10-second mark just to get in the game. And you talked about a big part of that being essentially how performance, how good, how believable the model is. And then beyond that, you talked about integration and you mentioned agentic AI. There was reasoning. These models just have to be good, have to give good responses. But then you talked about reasoning, taking it to the next level, but then you also mentioned agents and agentic AI, and I’m super curious to unpack — what does agents at the edge look like? What do you even mean there?

DT: That’s a good question. So most of the things we talked about today is about making a robot extremely useful. But ultimately, if you think about many of the applications in industry, in enterprises, it’s not going to be just about a robot. It’s going to be about fleets of robots. Just like if you have a company, it’s not about an employee, but it’s going to be about all of them working together to create something. Imagine a factory. A factory in the future will have robots of different embodiments, different levels of intelligence. There’s likely going to be people too. And there’s going to be digital AIs. And you have a game plan of a manufacturing plan, which includes doing it in a safe manner, improving throughput, and you have to manage all the inventory of supplies coming in and supplies going out, and all of that is going to have to be orchestrated.

So how are you going to combine each and every one of the robots with different capabilities and somehow integrate them all together and evaluate scenarios and what is the best way to do it? This is where agentic AI comes in, because it will integrate each of these different digital AIs and physical AIs to have Uber policies. And now the next question is, how do you validate that the policy is the best one? You don’t want to stop your manufacturing line to test these policies. And it turns out you actually want to do all of this in simulation too. This is where a digital twin of an environment of a factory matters.

And one of the things that we’ve been working on, there’s this technology called Nvidia Mega, which is a blueprint for doing fleet simulation of a complete factory level or a city level or a building level, if you will, doesn’t matter what that abstraction is, and simulating all the different physical agents, digital agents, AIs and orchestrating all of that for testing. This is why I keep talking about agents is going to be super important, because ultimately it’s not going to be about a robot. A robot needs to be good, but it’s about how you integrate all of them together to create a bigger job, bigger task.

Fascinating. So that’s super interesting. I think a lot of people are starting to think about agents. I’ve got OpenClaw running and I’ve got the agent that checks my email and the agent that does this and that and it summarizes things and whatnot. And you’re saying, yeah, those are digital agents, but we’re going to also have physical embodied agents. And so this orchestration plane in an enterprise or in an industrial setting is going to need to not only be able to interact with an orchestrator, digital agents but also physical agents and orchestrate across them. And then you made the super interesting point, which is okay, well you ought to be simulating and testing a digital twin of that too. Because how do you know that — yeah, because of course there’s going to be some sort of supply chain manager, logistics person, or factory manager that’s going to want to play around with this and it’s going to be a lot easier to test all of that in simulation again rather than just willy-nilly try something in the factory and have the line shut down or whatever. Super, super, super interesting. Yeah, you guys are definitely living in the future.

DT: That’s right. And the joke I have is, I’ve been living in the future for more than a decade, but for the first time I feel like the future is coming to the present.

The Road Ahead: 2029–2030 and the Edge Roadmap

Yeah, absolutely. So look ahead three years, 2029, 2030, let’s say. How close are we to that sort of world where there is a business and it’s orchestrating across humans and digital agents and physical agents?

DT: I think it’s going to happen. It’s going to happen in stages. So the first step is a robot. What sort of jobs or tasks is it able to do at the right level, sufficient level of accuracy, throughput, cost and so on. That has to happen for those kinds of robots to scale up. And then increasingly robots will become good at solving more and more of these tasks. So what we’ll see is it’s going to be like a continuous journey where, just like autonomous vehicles have hit the 10-second mark, we’ll see that the manipulation, rigid bodies and locomotion type of tasks the next year or two will hit the 10-second mark. So you’ll see more of those being deployed in factories and warehouses. And once they’re deployed, agentic AI will absolutely be required to orchestrate all of that. So that’s likely going to be deployed by 2029, 2030.

Now, as time goes by and as we solve general purpose intelligence and solve more dexterous manipulation, fine-grain with soft bodies and deformables and all of that, then you can imagine, you’re going to unlock more and more use cases. And that really is going to be technology — you got to the 10-second mark. When you hit the 10-second mark, you would know. And then the next two, three years after hitting the 10-second mark is trying to win that Olympic gold.

Yeah, fascinating. So last question. As you’re talking about this future where you have different embodiments and different use cases being solved, that are eventually, they’re kind of like point solutions until they hit this 10-second mark and then it’s like, good, throw them in the mix, start orchestrating across them and more and more sort of unlock that level of sufficiency to be deployed and orchestrated across. Ultimately, what does that mean for the edge computing portfolio? Are there going to be tons of different SKUs because people need different memory and different compute for those different work cases? Or do you ultimately see that actually a lot of this is still solved by maybe a tighter compute portfolio just maybe deployed at different power levels? How are you guys thinking about the future of the roadmap?

DT: So we already have different SKUs. We have in the Thor family, we have two SKUs. In the case of Orin, we have like six different SKUs, same software, but depending on the performance required, depending on the cost, power, and functional safety and industrial grade, because the number of applications that you think about is so broad. So we have a pretty broad portfolio.

And then ultimately, we will address portfolio where extreme high compute and all of this is important, but there will also be applications where maybe you don’t need all the performance that we’re providing because a lot of these would be hybrid edge-cloud. Because there’ll be a lot of intelligence at the edge, but there’s no reason why you wouldn’t want to phone a friend and call into the cloud to get some answer for something, especially for long reasoning, long thinking type of things, which you never have enough compute to do locally at the edge.

So our portfolio is, we have a pretty vibrant portfolio from, same software but leveraging the same compute architecture, but scaling in performance, price points and power. And then we’ll continue to do that.

Nice, that’s cool. I hadn’t thought — it’s good to be reminded that these robots don’t need to exist in isolation, but they can always call the local cloud computer or whatever and kind of phone home or phone a friend. That’s interesting. Alright, well, that’s it for today. A lot to chew on. Thank you so much. I learned a lot. I think listeners are gonna love this. Appreciate you spending time with us, Deepu. Thanks.

DT: Yeah, my pleasure, Austin. Take care.

Chipstrat is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

There are physical limits at higher bandwidths too, and that changes what technologies matter over time. EMLs, SiPh, VCSELs, AECs, ACCs, AOCs, microLEDs, etc.

On top of that, capacity constraints are in play (e.g. InP wafers), so decisions made 18–24 months earlier determine who can actually ship when the doubling lands.

Timing is everything.

But there’s a lock to unpack. Where to start? Let’s wrap our heads around the cadence the industry is chasing by reading recent earnings call from the public semis, optics, networking, and contract manufacturers in the stack.

Here’s what they laid out, at the highest level:

• 400G: mature.

• 800G: volume cycle in 2026 and 2027.

• 1.6T: production scale in 2027.

• 3.2T: launching in 2028, volume ramps 2029–2030.

In chart form:

One caveat on reading the chart. We have to infer the mix shift ourselves. No CEO says “our 400G sales are declining” on the call.

Subscribe now

So that chart captures the cadence at the highest level, as decoded from ~15 companies’ earnings calls. But there’s a ton of nuance to unpack. First, there are differences between roadmaps for scale up/out/across. Plus, the further out stuff is fuzzy, so we should compare what the CEOs optimistically said against actual OFC (R&D) announcements.

For paid subscribers who want to go deep, we’ll hit on

• Year by year: Scale-out, Scale-up, Scale-across

• The receipts. ~50 direct CEO quotes from ~15 companies grouped by generation and year

• Timing nuances, especially 3.2T

• OFC 2026 reality check

We’ll also check back in quarterly to see how the communicated roadmaps change.

Read more

Natalie is a co-founder of Gimlet, alongside CEO Zain Asgar (a Stanford CS professor). Beltir spent years at Intel before joining Gimlet five months ago, after Gimlet had been one of her portfolio companies. The company was founded in 2023, has raised $92M (Series A this March), reports more than $10M in annualized revenue, and runs a two-track business — deploying its orchestration software inside customers’ data centers, and operating its own neocloud with mixed silicon.

In this interview, we walk through how Gimlet thinks about both the architecture and the business. Important insights:

• Most neoclouds are backed by one silicon vendor and gave significant equity in return. Hardware amortization is ~70% of their annual costs, leaving very little room to optimize bottom line. That equity entanglement means they can’t diversify their silicon, which is why the only software innovation they can ship is disaggregation on top of a single vendor’s stack — never across vendors

• Gimlet’s two-track business model is the answer to that constraint: deploy software inside customer data centers (frontier labs, hyperscalers, sovereigns) and operate their own neocloud with mixed silicon for AI-native customers. Supply-chain diversity optimizes the bottom line, differentiated token performance commands a price premium on the top line, and one track funds the CapEx of the other

• Hyperscalers and frontier labs already run multi-vendor silicon (NVIDIA, AMD, in-house ASICs), but the orchestration layer is getting more complex faster than internal teams can keep up. They’d rather spend engineering attention on next-gen training and product differentiation, so some outsource orchestration to Gimlet — and some go further, having Gimlet take on the CapEx and data-center burden so they can experiment with hardware combinations without staffing a forever-team

• AI-native customers aren’t just price-sensitive — they have product latency budgets (e.g. one-second response windows, voice agents) where faster tokens unlock entirely new user experiences, not just cheaper ones

• Sovereign clouds are a prime customer segment — Europe, the Middle East, India, Asia, and Korea have government funding and some have emerging local silicon vendors, but lack the in-house software talent to write optimized kernels across N chips. Gimlet’s pitch is “make an API call, not a porting project”

• On the architecture side, Gimlet’s stack traces a PyTorch workload as a graph, splits it at optimal points, then lowers each segment to the target vendor’s framework (TensorRT on NVIDIA, equivalents elsewhere). They don’t try to build a universal programming language across chips

• On GPT-OSS 120B with 8K input and 1K output, running the speculative decoder on a d-Matrix Corsair card while NVIDIA B200s handled the verifier delivered roughly a 4× shift in the throughput-vs-interactivity Pareto frontier compared to GPU-only speculative decode

They’re also hiring across the stack: scheduler, compiler, kernel optimization, and distributed-systems engineering, in person in San Francisco.

This interview is lightly edited for clarity.

Meet Gimlet

Hello, everyone. Today we have special guests from Gimlet Labs. We have Natalie and Beltir. And we’re going to talk all things heterogeneous silicon and rethinking the data center. So let’s start, Natalie, with you. Our audience probably doesn’t know you guys or Gimlet. So tell us more about you.

Natalie: My name is Natalie. I’m a co-founder of Gimlet. Gimlet, we can go more into it, but what we are is we’re an inference cloud built for agents. And one of the key aspects about our technology is that we’ve built this inference cloud across heterogeneous hardware. And we can get more into that, but we think that that is going to be the future of inference.

Nice, exciting. And Beltir, who are you and how did you get to Gimlet?

Beltir: I’m Beltir. Nice to meet you. I joined Gimlet roughly five months ago. Before joining Gimlet, I was at Intel, and Gimlet was one of my four portfolio companies that I’ve been working very closely with. I’m amazingly excited about what we’re building at Gimlet, so I ditched my corporate job and jumped onto the startup again and trying to build a very exciting business. It’s been an amazing five months so far.

The Case for Heterogeneous Infrastructure

Okay, I’m just going to try taking us through some of your slides that I clipped from blogs and found online to just get your reactions and have you talk through for our audience, in real time, what you’re trying to do, what problem you’re trying to solve, and why it’s important. I’ve written a lot about this shift from a GPU, one-size-fits-all GPU, to multi-vendor, multi-silicon environments. So I was excited when I saw that you guys are thinking about this too. Natalie, tell me — make the case for heterogeneous infrastructure.

Natalie: Just starting with the context that we’re all probably familiar with, it feels like every day you hear an announcement that one of the large frontier labs has made some kind of compute deal for capacity with some chip vendor, whether it’s Trainium, AMD, TPUs, or NVIDIA. And another piece of news you hear, it feels like almost every day, is that there’s a new accelerator company that just launched. They have amazing performance on inference. They’re designed for inference specifically.

What we’re basically seeing in the broad context is that everyone’s extremely capacity-constrained at this point, trying to scale out their inference. They’re trying to improve the performance of their inference. They need as much compute as possible. And then they also need specialized compute potentially to make it even faster. So how does that all fit together? Sometimes people ask, is this new chip going to be the GPU killer or something like that?

So the way that we see it at Gimlet is a little bit different. We think that all of these options are really great for different purposes. And that’s important because agentic inference is not a uniform workload. Different parts of it have different compute needs and different bottlenecks. So when you think about a really, really large-scale workload that you need to be very fast and efficient because we’re pouring trillions of dollars of CapEx into it, then you want to start thinking, how can I optimize the attention of this model? How can I optimize my speculative decoder or my tool calls? Each of those components actually benefits from a different type of hardware because it has different trade-offs. So what we see is that the industry is moving toward a heterogeneous stack for inference in order to meet the performance needs.

Yes. So I found you guys had this slide here, and this feels like it’s exactly what you’re saying, which is breaking down the workloads that are running at scale. It used to be there was a time when it was kind of like, let’s accelerate everything and we’re not sure what the dominant workloads are. So a GPU can run high-performance compute, scientific compute, or AI. But now obviously all of the inference that’s happening is really about LLM inference for these few frontier labs at scale, and so it feels like you can start to take that inference workload and ask, what is the right silicon for this workload? Maybe to the point of the table — what are the different parts of that workload, what are their system requirements, and how might those actually fit onto different hardware?

Natalie: I think one thing about GPUs is they’re incredibly versatile. So we definitely think they’re going to be an important part of the inference stack. When you look at this table, we have it broken down by different very high-level phases of inference, showing the resource needs for each of them and how they vary, and how you actually can’t have one chip that is optimal for all of these. It’s just literally not possible. But each of these is a critical stage.

So how do you solve that problem? Our core belief is that you solve that problem by disaggregating the workload and running each segment on the chip that’s best suited for it. The other thing I want to point out about this table is that even this is very, very coarse-grained. You can subdivide each of these components into more segments, each of which have distinct bottlenecks from each other. So it’s one of those problems that even at this level, people will benefit from disaggregating, but we’re thinking even more so than that — even within LLM pre-fill, how can we disaggregate that further?

Disaggregating the Workload and Orchestrating It

Say more here. You’re talking about, how can we split the workload up ever finer and finer? And then apparently in real time, being able to distribute that across the correct hardware.

Natalie: Right, and you also don’t want to indefinitely subdivide, because there is cost between sending the data from one chip to another. But it’s about expressing the workload, finding the optimal points to split it up, and then scheduling and scaling it across the available hardware.

Okay, so do you do that in advance of running it then? You look at the workload and figure out where those points are to break it up?

Natalie: Yeah, that’s a great question. Some of the other slides will go into it a little bit more, but the way to think about it is that we take the workload, we trace it. So if you give us PyTorch, we’ll trace that. It could be something else too. And then we’ll actually turn that into a graph representation. Our orchestrator and scheduler basically figures out how to segment it into its component parts for further compilation. So we trace it, we walk that graph and we understand what’s there, we break it up and optimize how we do those splits. And then for each of those segments, we’ll lower it to the target hardware.

One thing that I also like to point out here is we really work closely with our hardware partners because we’re trying to use the frameworks that they have available at the low level, not trying to create a stack that is a programming language for every single chip. So once we have those segments, we’ll actually compile them and lower them down to, for example, TensorRT on NVIDIA, or other similar frameworks on other hardware.

Fascinating. So I feel like what I’m hearing from you is that this is obviously more than just a hardware play, but obviously you’re doing a lot in the software stack to really orchestrate, is what I’m hearing.

Natalie: That’s right. We think of ourselves primarily as focusing on the software layer. We have to tap into hardware as well because we’re connecting these different platforms together. We’re connecting chips that have never been connected together before because no one has taken chips from vendor A and vendor B and plugged them together and orchestrated a single workload across them. So we end up having to play in that layer a bit too. But what we’re really emphasizing at Gimlet is the software layer for orchestrating across this hardware.

We think that, to the slide that you just pulled up, this is a problem that is going to compound, not ease, over time, because everyone is still coping from the massive scale-up of simple LLM inference. But what everyone’s moving to, and we see this with coding agents, is multi-step agents that are doing searches, running things on your machine, maybe calling out to other agents. These are even more heterogeneous than the LLM chat models, which were much more heterogeneous than people really even account for on their own. Once we start moving to background async agents that are all communicating with each other, they’re multimodal, there’s different model types — the whole problem of the inefficiency of a homogeneous stack is just going to get completely untenable.

Yes, that makes a lot of sense. We’re moving to where it’s not just the human interacting with the LLM, but you’ve got agents, the agents are doing different things, calling different models. So there’s lots of opportunity to optimize. Thinking like this sort of agent end-to-end workflow, what does that look like from an orchestration perspective? You have an illustration here, but in my head, it just feels very complicated when I’m thinking about agents, tool calling, and all that stuff. Talk to me more about this orchestration layer.

Natalie: We touched on it a bit before, but I think we think about optimizing and orchestrating across an entire agent, not just an individual model. We represent these things as graphs in our system. At the end of the day, we don’t really care what type of model it is, what things it’s doing, as long as we can represent it in our compiler’s framework and then figure out what its bottlenecks are and then schedule it on hardware.

So whether it’s one model, two models, models with functions — it’s all kind of the same in the way that we’ve designed our system. The important part is that we can trace that entire thing and then split it up and then, like this diagram shows, route it to the appropriate accelerator.

CPUs, Tool Calls, and the High-Speed Fabric

In this diagram you showed GPU, specialized accelerator (so maybe like an SRAM-heavy one), and CPU. CPUs are all the talk lately. Tell me how you’re thinking about CPUs. What type of workloads are you putting on the CPUs?

Natalie: That’s a great question. It’s been really awesome to see the excitement about CPUs recently because they are a really important workhorse of these agentic workloads. A pure LLM or a pure model only has so much capability unless you can actually connect it to the outside world and the ability to do general-purpose tasks. So the most obvious application of CPUs is things like tool calls, but you can also use them for things like smaller models or data processing and other types of things that benefit from the CPU’s trade-offs. But tool calls for me are the most exciting thing. When you actually run that tool call in the same place that you’re running the LLM, it really improves the end-to-end latency of the overall agent.

What do you mean by in the same place as the LLM?

Natalie: For example, when I’m using a coding agent today, the LLM is running on someone’s server. And then it’s coming back to me, saying to my machine, please look up the contents of this file, or please do a web search. And then that is executed from my laptop. This introduces a very network-bound aspect of the workload because it has to constantly jump back and forth between my laptop and where the model is running. So what I’m saying is that for cases where you can actually run those tools on the server side, you end up with much, much better performance.

Okay, so would you say then, especially in your architecture, the CPU rack should be in the same data hall on the same network, or is it just as long as it’s off of your laptop and running in the cloud, maybe there’s lower latency?

Natalie: It depends on the needs of the workload, but what we would generally say is that the way we approach it at Gimlet is we want to connect all of this hardware together through high-speed fabric. So that’s why we’re not just saying this data center is for hardware A and this data center is for hardware B — we’re actually physically connecting these racks together. In general, I think that it’s better the closer it is.

Beltir: The reason we want that proximity is actually latency, because there’s a big demand for really fast tokens and higher user interactivity. This today usually comes at the expense of a throughput hit. And in a world where everybody is power-constrained, capacity-constrained, people have to make really hard choices. Whether am I going to have a throughput hit but for high, low-latency tokens, or am I just going to optimize for throughput? By putting these different types of hardware in the same data center, interconnecting them, we’re trying to give customers a solution that actually expands that barrier where they can make these choices without as much of a trade-off on either end.

Okay, interesting. So at the end of the day, if we want as fast tokens as possible, you’re saying we should disaggregate the workload and put it on the right silicon for that shape of the workload. And we need a high-speed fabric, and ideally you would have all of the hardware that you’re scheduling across sitting on the same fabric to reduce latency.

Natalie: That’s right. For some types of disaggregation, this matters more than others. So for something like pre-fill/decode disaggregation, you might be okay with a hop, because that’s only happening one single time between the ingestion of the context and the outputting of the first token, then hop to emitting every subsequent token. But for more fine-grain disaggregation, it becomes more important.

Three Customer Segments: Sovereigns, Frontier Labs, AI Natives

Okay, so at a high level, we’ve talked through some of what you’re trying to do, which — reflecting back for listeners — you’re saying, hey, what if we built an inference cloud for agents where actually inside the data center, there’s lots of different kinds of hardware, and we’ll write a software stack that’s like an orchestration stack that looks at the workload, figures out where’s the right place to break it into little subtasks, and then we will give it to the correct hardware, whether that’s CPUs or SRAM accelerators or HBM accelerators, and we’ll have it all on a high-speed fabric so they can all communicate really well. So I guess that leads to the question — who’s this for? Who are the customers and why is your cloud going to be compelling for them? Beltir, I’ll hand it off — educate us on the customers.

Beltir: I put our customers in a couple of big buckets. The first bucket is frontier labs, in my mind, who are making all these contracts with many different silicon vendors. Again, everybody is power-constrained, capacity-constrained today, and as we talked about, they’re all trying to solve the problem of, how can I provide the fastest tokens — which is better user experience — without compromising my throughput, or getting as much throughput as I can from my existing investment. This is a never-ending problem as the baseline keeps moving and the capacity constraints become more and more of a bottleneck for everyone. That’s one bucket.

The second bucket of customers we get a lot of interest from is sovereign cloud vendors who are interested in supply-chain diversity, who are putting together multiple of these contracts in place, but lack the capability to be able to serve them at scale. Bringing up a new hardware vendor is a lot of work. Porting one same workload from NVIDIA to AMD to MatX, it’s a lot of work. What we’re talking about is not saying that we will take your workload and we will port it. What we’re saying is you shouldn’t be worrying about these different hardwares and porting your workload to each and every one of them separately. You should just make an API call, or you should have an intelligent software stack — if you’re deploying our software stack in your data centers — that actually takes your workload and figures this mixing-and-matching algorithm itself, rather than your engineers trying to write kernels for each and every one of these hardwares. This also creates a big bottleneck for them to get these new emerging architectures. Because who’s going to write those kernels for those? It’s pretty hard.

The third set of customers are what I call the up-and-coming AI natives who are buying tokens at scale. ElevenLabs, Notion, Glean, Harvey-type companies. Companies who are building the next-generation diffusion models, very latency-sensitive. They’re amazingly constrained by what the current infrastructure is offering them, which is, we have a good enough product but it doesn’t give the latency or the fast tokens that you need to be able to innovate the next-tier user experience. The first bucket for us is a combination of both — us deploying our software in their existing data centers. The second and third tier of customers is mostly around customers who buy tokens at bulk from our new cloud infrastructure.

Okay, this is super interesting. So I want to unpack this and go into each of them. Let’s start with sovereigns. So sovereigns, what I heard you saying is — you’re sovereign, you’re standing up your own data centers, you’re going to buy from different vendors over time so that you can have that supply-chain diversity. And then you’ve gotten yourself into a situation where you already have different hardware, but now you’re stuck with: man, that has increased the amount of software engineering we have to do, because now we have to decide maybe manually which workload goes where, and we have to write optimized kernels to run on the different hardware. So it’s like a software burden on maybe a customer who doesn’t have a huge software team. So you guys can come in and say, hey, we’ll take a look at your hardware and we will help you orchestrate across that hardware. Is that ultimately?

Beltir: That’s ultimately what we’re trying to go for.

Okay, that makes a lot of sense. You’re kind of like the cracked software engineering team that they need.

Beltir: Cracked software is the software platform that they need. Today, most of these infrastructures are set up as a bare-metal-as-a-service infrastructure, which has its own challenges from a software engineering perspective. What we’re offering them is not a set of software engineers — we’re doing this work for our own neocloud offering anyway. We are building this orchestration stack in deep partnership with those hardware vendors for our own business. What we’re offering them is a ready-made platform that we can deploy in their existing data centers for them to very quickly get to market with the existing investments they’re making, but not only time-to-market, also better throughput, better capacity, and better user experience from that as well.

Because all the sovereign clouds also, they just don’t want to build this for the sake of building it. They also want to be at the frontier of the innovation as well. If you look at Europe, there is a lot of government funding that’s going in this area for them to be part of the innovation ecosystem. Same in the Middle East, same in India, same in Asia. How can you give them an offering that actually helps them get there faster, differentiate them — is another part of the equation. And the other one is they are very keen on supply-chain diversity. Having NVIDIA and AMD doesn’t solve the problem. There is a lot of hardware innovation that’s happening outside of the US as well. That also has the same issues. If you look at Korea, there are really interesting chip companies that are coming out of the Korean ecosystem that we’re talking together with right now. They’re also thinking through how can these emerging hardware architectures be consumed without the software burden. Because being able to do this kernel engineering, the software model porting — it’s a lot of work.

Fascinating. I didn’t think about the point that we tend to focus on American chip companies, but there are actually other chip companies elsewhere. So not only for sovereigns can you solve the “hey, you don’t have to worry about software, our platform solves that for you” piece — and then I liked your point, which by the way, we will make sure that it’s highly optimized, so it’s not just that you got it to run, but we’re going to optimize it for you. On top of it, yes, you can as a platform take on the burden of getting comfortable and making sure that you work with all sorts of vendors from different countries, because that makes sense for you as a platform and then that’s something that you can offer to all of your customers.

Natalie: If you want the best performance, you really have to partner closely with the chip company. That applies to pretty much everyone. If you’re running a production-scale workload, you need to get a very close relationship with the hardware maker that you’re running it on. Doing so for N hardware platforms — and also keep in mind, it’s hard enough to get performance on one — moving it to another is another step up. Taking it and breaking it up and running it on even more, that’s something that we think is optimal from an efficiency standpoint, and it’s why we’re building Gimlet, but it would be very difficult for everyone in the space to replicate that.

Yeah, totally. Not to mention merchant silicon vendors only have so much bandwidth. I’m sure they can only help so many people that come to them. So I can see how it could be win-win for them if they can just work with you, and then you can make it work with everyone.

Natalie: One more point — the chip companies, GPUs are amazingly versatile. You have other hardware that’s really, really great at many parts of inference. By putting it alongside other types of hardware, it can really shine in the tasks that it’s best suited for.

Beltir: And it also de-risks from a customer experience perspective. You all are very comfortable with running on NVIDIA, running on the AMD ecosystem, but you will have a hard time porting your model on another vendor’s cloud-only option. Many hardware silicon vendors try to stand up their own clouds because customers were hesitant to use their cloud infrastructure. But what they’re also seeing is, even if they set up that cloud infrastructure, at-scale customers — for them, it’s also a lot of engineering effort to move their workloads to one vendor’s cloud only. So those clouds are not scaling. What we’re offering is a mix-and-match environment for the customers who are looking to benefit from these emerging architectures, and for the emerging silicon vendors, a way to go to market at scale without taking on the burden of building their own cloud infrastructure, because that’s not their core business.

Okay, so now let’s go to the hyperscalers or those serving the frontier labs. Hyperscalers, we know they have multi-vendor silicon. Meta’s always talked about a lot lately — they run NVIDIA, they run AMD, they have their own MTIA chips. Now, unlike the sovereigns, a hyperscaler has plenty of software engineers, even though this is a laborious task to optimize their kernels for all the different hardware. So tell me, why is it better that they would partner with you rather than just try — maybe they’ve already built this sort of orchestration themselves — or where is what they’re doing suboptimal compared to what you’re doing?

Beltir: I think there is no one-size-fits-all answer for a hyperscaler or frontier lab. There are different stages in this journey, because three years ago we weren’t talking about this level of disaggregation of inference workloads. We didn’t know what inference was going to look like. P/D disaggregation was a very early PhD-thesis-type of implementation. Today it’s becoming more commonplace and we’re talking about way more complicated disaggregation methods. There are different phases and different stages in their journey of figuring out how they’re going to serve inference at scale.

Some of them are trying to build this in-house with competing priorities. Some of them, the ones we’re working very closely with, are telling us that this is not their current strength or current focus. They’re in a place to meet the next-generation training wars, and in next-generation, differentiating their product. Rather than trying to bring up a different infrastructure, writing kernels, getting them up and running, they would like to outsource all of this so that they can experiment — because they know their workloads — so that they can experiment which of these combinations will give them the best alternative.

The other part of this is, as they’re investing more and more CapEx for their data centers, their margins are getting thinner and thinner. So what we’re also seeing is they’re trying to outsource some of these investments to companies like us, saying, okay, you take the data center burden, you take the CapEx burden, you bring it up and running for me. See how this will work for me in this particular hardware combination — because they already have those deals with the hardware vendors. So we see a couple of different reasons depending on where they are in their journey.

That really resonates with me, especially when you talk about what are their core competencies and what is their ultimate business model, and how can they spend as much time training a better model or whatever. It actually reminds me — when I was in grad school and when I was an undergrad and I did research, both times I benefited from people who came before me. A PhD student would spend like four years building a system and then they would only have two years left to quickly run some experiments on it. And then I would walk in and I just run experiments the whole time. And I’m like, man, I’m glad I didn’t have to spend four years building this. It’s kind of the same thing. You’re trying to say, let us build that infrastructure so that you can experiment on top of it. Let us handle optimizing and really focusing in this, and you guys just worry about your experiments.

Beltir: Exactly.

Natalie: No one really wants to go to all of those chip companies, optimize them for all. It’s a lot of work.

Totally, and you’re signing up to do that forever. You just built a team that is committed to doing that forever. I do like the idea of just outsourcing it, so you don’t let a company exist solely to solve that problem.

Beltir: Exactly. Think about every new hardware coming up — but not only that, the maintenance of an infrastructure like this is also a big ongoing commitment for them. Every new rack release you have to update. All of these create a lot of issues. And I think everybody is in a race to differentiate themselves rather than trying to figure out some of this plumbing in-house.

Yes, totally. Now, lastly, let’s talk about the AI native. So an AI native today, they don’t own their own infrastructure. They’re just trying to buy tokens as a service from APIs directly, or Amazon Bedrock or Google Vertex or something. And if I heard you right, what you were saying was today they can only get tokens. You can pay a lot for a fast token or pay less for a slow token, but maybe they don’t have enough fine-grain control. Or is it ultimately just like, by buying a token from you, it will be faster and lower cost? What’s the pitch?

Beltir: It’s a combination of both right now. I think there are two different types of customers. One of them are big enough so that the token cost is hurting their profit margins as they’re growing. So they’re more cost-sensitive and they’re looking for options to reduce that cost for them as they grow. The second one are emerging innovators that are building diffusion models, video-based solutions, voice-based solutions, where latency is a big, big bottleneck for them to bring a competitive product to a market. They have options, like I mentioned, on emerging new clouds’ own cloud solutions, but it comes at a very different trade-off for them to be able to do that. They have to spend their limited resources porting their models to those cloud solutions as well. So it’s a combination of two different customers that are approaching us right now.

Natalie: There’s another thing here — actually there’s two points I want to make. The first is that you get tokens from someone. At the end of the day, the limiting resource might be like power capacity. If we can deliver a shift in the Pareto frontier for the available power by leveraging heterogeneous hardware, we can translate that for our customers to lower latency, to higher throughput — it can be a variety of benefits, because you’ve actually shifted what’s possible by doing this. And what the folks in this bucket tell us is that, taking latency as an example, it’s not just that it’s better to get tokens faster. It’s actually that different product experiences have different latency budgets. The user can’t wait for a response more than one second. By making it three times faster, five times faster, what those folks tell us is it actually lets them enable new experiences that wouldn’t have been possible when using the providers that run homogeneous stacks.

So I only have a second to respond here so I can only do a couple of things — but if I could do a bunch of things in that second, then yeah, I can unlock a new user experience that is differentiating.

Natalie: This is especially important for things like voice agents.

The d-Matrix Partnership and the Pareto Frontier Shift

Totally. Okay, so you mentioned Pareto frontier. Let’s give one example before we end so people can understand what we’re talking about. Tell us about d-Matrix, your partnership with them, and then I’ve got the Pareto frontier slide after this.

Beltir: One of the things that we’ve been talking about is mixing and matching different architectures, but especially with GPUs and SRAM-based architectures. Without going into the technical details — this is how you can actually pair a throughput machine like an NVIDIA B200 or GB200 with an SRAM-based architecture, which are amazing decode machines and can push the latency frontier way more, multiples of what an NVIDIA or a GPU-based architecture can do.

We had this hypothesis that mixing and matching together can actually shift the Pareto curve faster. We partnered with d-Matrix. d-Matrix is only one of our partners that we can name publicly right now. We are partnering with multiple of these SRAM-based architectures. The d-Matrix team has been amazing from a time-to-market, speed, and partnership perspective in optimizing a software stack and a hardware for this. What we’ve done with them is basically putting together in our own data center a d-Matrix Corsair card in the same rack with NVIDIA B200s, directly connected to each other, to be able to test how much we can push the frontier curve. I’ll let Natalie talk about what it means and what we’ve done.

Natalie: Let me first orient the chart. I think your listeners are probably familiar with the classic chart that Jensen often shows, but just in case, let’s recap it. So on the y-axis, what we have is throughput per kilowatt in terms of tokens per second per kilowatt. This is basically saying, if I have a 50-megawatt data center, how many tokens per second can I push through that data center? Then on the x-axis, what we have is interactivity. So if I’m a user getting tokens being processed in that data center, how quickly can I get those tokens as my personal experience?

You would think those two things at first order would be very related, but they’re actually at odds. That’s because the longer you give me to serve a token, the more efficient I can be with how I generate that token. But if you say, no, I need this token really fast right away for Natalie’s use case, then you have to pull out all the stops to get that token to that user as soon as possible. So we show these things as a frontier where you can optimize for one or the other or somewhere in the middle, but you’re never going to get something that’s fully in the upper-right quadrant because they’re fundamentally at odds.

So let’s now look at what we did with the d-Matrix side of things. We show three different Pareto frontiers for three different configurations for the same workload. This workload is running GPT-OSS 120B, 8K input sequence length, 1K output sequence length. What we’re showing is the frontiers for that workload.

We have three configurations here. The green one is a traditional pre-fill/decode disag on GPUs. So we can see that that offers a certain tokens per second at a given interactivity level. Usually the way people think about it is, my requirement is that my users need at least X tokens per second. And then from there, I try to push the throughput as high as possible. So you would set a latency budget and then try to maximize throughput given that latency budget.

A common technique that people adopt to speed up their workloads is they introduce speculative decoders. What speculative decoders do is they say, wow, running decode is really slow and inefficient, because I have to run the full model for every single token. But sometimes I could maybe use a smaller model, or something like Eagle which works a little bit differently, to guess at the next token. And maybe if I could guess multiple tokens in a row, then what I could do is take my large model and verify if they’re correct. Because it’s a lot more efficient to verify five tokens in a row and say, are these correct, than it is to actually generate them one by one with that large model.

So what we have in the blue line is a GPU-only speculative-decode flow. What we can see is, compared to the pure pre-fill/decode disag, it offers a shift in the Pareto frontier that’s quite significant. That’s why folks are adopting speculative decoders — because it really, really helps deliver better experiences, have more capacity, et cetera.

But what we did is we decided to take this a step further and say, okay, what if we take that same speculative-decode setup, but instead of running all of those parts on a GPU, we’re going to take the spec-decode part and run it on d-Matrix Corsair? That’s because d-Matrix Corsair offers a lot of on-chip SRAM, and it’s really, really fast when you can store the model weights in memory. So by running that smaller 1.6B spec-decode model on the Corsair — even on top of the blue line, which is already quite optimized — we see a dramatic, dramatic performance benefit. At a reasonable point on the interactivity side or on the throughput side, you can get like a 4× benefit.

Awesome, interesting. Zooming back out for listeners — we talked about taking parts of the workload and scheduling it to the right hardware. So d-Matrix’s chip is one of those SRAM-heavy ones. And so if there’s part of this workload, which is like, can you do this really quick guessing and see if you get it right in advance so that you have to do less work, what if that just ran on an SRAM-heavy chip that could do this guessing really fast? And that allows — now I stole the chart — where it’s like pushing it horizontally, for the same throughput per kilowatt it would unlock a much higher interactivity. I know you had other charts that said, of course, depending on what people are trying to do, if they’ve got a latency budget, they could also stay at a fixed interactivity if they wanted and get a higher throughput. So serve more customers more efficiently.

Natalie: Right. You can choose: do you want your customers to get their tokens four times faster, or do you want to serve two or three times as many customers at the same latency?

Well said. I just clicked one other slide which you showed that you can get even more of an unlock if you use a verify stage of 20 tokens instead of five. But maybe on this slide, a point here is for someone like a sovereign — it shows that you guys are thinking a lot about how to tune infrastructure, how to run little experiments, how to take the latest and greatest like speculative decoding and then take the latest and greatest chips like a d-Matrix one and figure out what are all the right knobs so that your customers can just come to you and say, make it faster, and you say, we got you.

Natalie: We all have limited capacity. We need to serve a lot of tokens. Inference is supposed to become the dominant workload over training this year. So what we are doing here, this is one example of the type of disaggregation that we can do and the type of hardware we can deploy across, but it’s not limited to this. This is to illustrate what you can get when you adopt a heterogeneous stack.

Beltir: And this only starts from customer-back, because every customer has unique requirements. They have different workloads. They run MoEs, some of them sparse experts. That changes what type of disaggregation methods you need to apply. That also changes which hardware combination would be the best for that particular workload. That’s also something that we can help with the customers as we learn more about their workloads. Because giving them an unlimited end option is also not the solution. There needs to be a limited solution space also for it to be cost-advantageous. So what is that right optimal combination for that particular workload?

We start from the other way around, saying, okay, this is the customer, this is the workload, this is their constraint — either latency, or power, or throughput. This is the characteristics of the workload and their customer base. So based on that, we run simulations and tell them, here is what we think would be the best architecture mix of hardware for you, and based on your needs, this is how you can push the frontier and what limits you can get. And then based on that, we start designing what is the smallest data center stamp that is required, because it has to be a repeatable implementation to be able to scale. You need this to be in the same data centers, you need to network them. What is that network topology? And then build a path to scale that implementation. So this is an end-to-end partnership with the customer.

Why Gimlet Differentiates in the neocloud Landscape

Okay, that’s super interesting. I love that you start with the customer needs first and build to their needs to get the most optimal experience for them. Also, in the back of my mind, one of the things I’ve been thinking is — if you guys are essentially like a neocloud, there are tons of neoclouds, how do you differentiate, who captures that in the long run? Some are just bare metal. So it’s like, okay, how can you differentiate there? But what I heard you saying is, no, no, no, we are a full-service, almost consulting partner where we’re helping you design your data center footprint, and we’re helping you optimize it, and we provide the software platform that will help do this for you. So it’s very differentiable compared to others in the neocloud space.

Beltir: Correct. And also if you think about us versus everyone else in the neocloud space, most of them are today backed by one silicon vendor. And in return, they gave significant equity. So it’s hard for them to diversify their silicon ecosystem. So it’s hard for them to do the mixing and matching that we do. And if you look at what’s going on in this ecosystem, inference prices are coming down, so everybody is getting more pressure on their top line. They have very limited opportunity to diversify supply chain. They have no negotiation. Hardware amortization is usually 70% of their annual costs. So they have very little room to optimize their bottom line. This is why the software innovation you see them announcing all these disaggregation methods — because they’re trying to create a sustainable business model.

What’s slightly — I will say grossly — different is we have two different dimensions that we work with the customers. A is the end-to-end software service and the scaling motion with large-scale customers. But for customers who are up and coming, who cannot yet commit, we also build our own neocloud with fundamentally different economics. Because from a bottom-line perspective, we actually have a supply-chain diversity that optimizes our bottom line. From a top-line perspective, since we can offer very differentiated token performance, we can also command a price premium rather than trying to race to the bottom from a pricing perspective. So if you ask us, Beltir, are you differentiated? I think we are very, very much differentiated. And having these two different dimensions in our business model gives us the liquidity and the financial stability, because one can fund the CapEx investment of the other.

Fascinating. I really like it. You make the very interesting point, which is the incentives that other people have or are bound by that would prevent them from going in this direction. You can buy whatever silicon makes sense, and then of course you have the chops, the software chops in-house, to disaggregate whatever workloads across that hardware as you see fit. Control your destiny.

Beltir: The other thing — the neocloud, I see it in two dimensions. You have CoreWeave-type people whose core strength is buying GPUs, data centers, and offering that as a bare-metal-as-a-service, but they lack the full-stack experience today. They’re trying to acquire companies to figure that out, but it’s really a long journey, very hard journey to mix and match acquisitions to create a unified stack.

The other ones are like Together and Fireworks, who are only software and trying to acquire the capacity from usually one silicon vendor’s infrastructure providers. So we don’t want to be either of them. We want to offer the end-to-end experience with two different business models that are very complementary to each other.

Series A, Hiring, and What’s Next

Nice, I love it. Last slide. Earlier this slide when we talked about business models, it also had the headline that I think in March you announced you raised the Series A. And then I saw you obviously are hiring people — if people click on view open roles, there are several roles. So tell us a little bit more about what you’re looking for and what’s up next for the rest of this year.

Natalie: I’d love to talk about that. We are very focused on hiring right now. We’re set to — I forget how many X we’re going to, is it triple, quadruple? It’s something crazy like that by the end of the year, because we are scaling rapidly to meet the demand that we’re seeing. So if you want to join a company that is in crazy-scale mode, this is a good time to join, because you’ll still be part of the old guard because we’re in that rapid growth phase.

Who are we hiring? In terms of number of roles, engineering is the biggest one. We are looking for people who know how to do high-performance AI systems across the stack, whether that’s by working on our scheduler, working on our compiler layers, working on how do we monitor these incredibly complex distributed systems, how do we write optimized kernels, how can we leverage AI to automate some of the optimizations that we’re doing ourselves. And then also general builders — folks that are kind of Swiss Army knives that love to go up and down in the stack and contribute to different parts. Definitely reach out to us if you’re interested. I will note we are an in-person office based in San Francisco.

Beltir: Just final words from my end. This is a crazy fast-growing rocket ship right now, because in many startups there’s always a concern, do I have product-market fit. We proved there is product-market fit. We are very well funded. We’re on the fast pace to get accelerated capacity. Most people are struggling with supply-chain problems — given our value proposition, that’s the least of our problems. Right now, our biggest problem is getting the right people to execute and deliver the customer commitments we have. So we are hiring across the tech stack from low-level kernel engineering to higher levels of software engineering. We’re building an end-to-end cloud stack, not a bare-metal-as-a-service. So across the tech stack, if people are interested, roles are open. We are looking for creative and innovative engineers who are looking to jump on a crazy growing ship.

Nice. Good pitch.

Natalie: I’ve been at startups most of my career, and I’ve been blown away by the scale of the opportunity here, and I pinch myself almost every day. We really look forward to welcoming our new colleagues.

Yeah, awesome, I love it. Having product-market fit, understanding your business model and having it figured out, and then of course just the macro environment that we’re in where there’s so much demand and so little supply, and being able to come in and figure out a unique way to make the most out of the constraints. Pretty exciting. So hey, I learned a ton. Thank you so much, Natalie and Beltir. This was really engaging and I know the listeners will walk away having learned something. So thank you.

Natalie: Thanks so much, Austin. It’s been a great conversation.

Well... allegedly. There are rumors of a warpage problem on the 4-die package, and chatter that TSMC is leaning on panel-level packaging (CoPoS) to deal with it, maybe even a fallback to a 2+2 config. Hold that thought.

So how do you connect four pieces of silicon together such that they behave electrically like a single chip? That’s the question of advanced packaging. And as AI accelerators keep getting bigger, the packaging itself is becoming the dominant cost variable in the bill of materials.

Today we’ll cover:

• A primer on 2.5D advanced packaging, and the reticle limit that started the whole story

• TSMC’s CoWoS family (CoWoS-S, CoWoS-R, CoWoS-L)

• Intel’s EMIB

• EMIB vs CoWoS-L

What is the reticle limit?

The way you make a chip more powerful, historically, has been to make it bigger. More transistors, more compute, more parallelism per die.

The ceiling on “bigger” is the reticle limit: the largest area a lithography stepper can pattern in a single exposure. About 26 mm × 33 mm, or roughly 858 mm².

NVIDIA’s H100 was already pushing the reticle. Blackwell broke through by stitching two reticle-sized compute dies together into a single GPU:

Once you cross that line (i.e. when one die isn’t enough) you need a way to physically connect multiple dies so they behave electrically like a single chip:

That’s advanced packaging. And as accelerator sizes grow, the cost of the packaging itself becomes a dominant economic variable.

What is 2.5D packaging, and why is it called that?

2D is one or more dies sitting directly on the organic substrate. No interposer, no bridge. Routing runs through the substrate itself.

That covers classic monolithic packages and chiplet designs where the dies talk through substrate traces. The constraint is density. Substrate pitch is coarse, so you get moderate die-to-die bandwidth, not the tight compute-to-HBM coupling AI accelerators need.

2.5D adds a passive silicon routing layer between the dies and the substrate. That can be a full silicon interposer, a silicon bridge embedded in the substrate like Intel EMIB, or silicon bridges inside an RDL interposer like TSMC CoWoS-L. It carries fine-pitch routing and sometimes TSVs, but no working transistors. It moves signals, it does not compute.

That is what makes tight compute-to-HBM coupling possible, and it is the dominant architecture in modern AI accelerators.

3D goes vertical. Silicon stacked on silicon — AMD 3D V-Cache, Intel Foveros, TSMC SoIC.

To get nitpicky: 2.5D with CoWoS-S is also technically “silicon on silicon”, but the interposer underneath is passive. Think of 3D as active on active and 2.5D CoWoS-S as active on passive.

HBM stacks are 3D internally, though they usually sit in a 2.5D package.

TSMC’s CoWoS family: three variants

Chip-on-Wafer-on-Substrate (CoWoS) is TSMC’s umbrella for 2.5D packaging. There are three commercially relevant variants. They differ mainly in how much silicon is used for interconnect.

CoWoS-S: full silicon interposer

The original. Entered production with Xilinx’s Virtex-7 2000T FPGA around 2011, where four FPGA slices were stitched together on a passive silicon interposer:

How it works:

• Multiple active dies sit on top of a large passive silicon interposer

• The interposer sits on the organic package substrate below

• The interposer carries fine-pitch metal routing for dense lateral interconnect, plus Through-Silicon Vias (TSVs) that route signals and power vertically down to the substrate

Important nuance: the interposer is not a “logic chip” in the compute sense. It’s processed on a mature silicon node optimized for routing density and TSV formation, not transistor performance. No logic, no transistors doing work on it.

Think of it as a tiny circuit board made out of silicon. Same job as a PCB, just at lithography pitch instead of PCB pitch, with several fine-pitch metal layers plus a forest of vertical vias.

Electrically, this gives you tens of thousands of short, fine-pitch interconnects between neighboring dies, at far lower latency and power than routing the same signals through an organic substrate.

Here’s what that original Xilinx example looked like:

HBM made this silicon interposer the default approach for flagship AI parts. The HBM interface is too wide and too dense for conventional packaging. Once GPUs adopted HBM (notably AMD’s Fiji / Radeon R9 Fury X), large silicon interposers became standard across every high-end AI accelerator that uses HBM:

There’s an economic problem though. The silicon wafer is being consumed for routing, not compute. That’s expensive. As HBM stack counts grow and compute reticle counts grow, the per-package silicon bill grows right alongside them.

CoWoS-R: organic RDL interposer

TSMC’s response to the silicon interposer cost is to build the routing in Redistribution Layers (RDL) of organic material instead of silicon:

This is cheaper, but isn’t a silver bullet. Organic processes have wider lithographic tolerances than silicon. Trace pitch widens, layer count climbs, and the assembly can’t match the bandwidth density that an HBM-to-GPU interface demands.

Thus, CoWoS-R is useful for cost-sensitive products that don’t need the densest die-to-die interconnect. It cannot carry flagship AI accelerator workloads on its own.

Trade-offs!

CoWoS-L: local silicon bridges in an interposer

TSMC’s current frontier, which places small silicon bridges only where you need high-density routing (compute-to-compute, compute-to-HBM). Use cheaper organic RDL material everywhere else on the interposer.

The bridges sit inside the interposer. The interposer is then attached to the package substrate as one large piece.

This is the TSMC architecture used for Blackwell-class accelerators.

The move is elegant balancing of trade-offs, with silicon where you need bandwidth, organic where you don’t. Beautiful in principle. The catch is that you’ve still got a separate interposer to build (an RDL carrier with those silicon bridges embedded in it), and then you have to dice it and attach the whole thing onto the package substrate. Two pieces, two attach steps.

Intel’s EMIB

Embedded Multi-die Interconnect Bridge (EMIB) shares CoWoS-L’s central insight (silicon only where you need it) but resolves it very differently.

EMIB skips the interposer entirely. The silicon bridges are embedded directly into the organic substrate of the package:

Here’s a cleaner version from Intel’s foundry blog:

Note that EMIB has only two layers. Dies and substrate.

EMIB-T and EMIB-M

Worth noting quick: Intel has next-iteration variants. EMIB-T adds Through-Silicon Vias through the embedded bridges themselves, which lets power and high-speed signals flow vertically through the bridge, not just laterally. HBM-heavy designs increasingly need this. EMIB-M integrates MIM (Metal-Insulator-Metal) capacitors into the bridge for on-package power decoupling.

Both are direct descendants of the same “embedded in the substrate” architecture. Worth a watch on Intel’s recent EMIB-T/M explainer:

EMIB vs CoWoS-L, side by side

So EMIB and CoWoS-L are both bridges right? Which is better?

• EMIB: silicon bridges embedded directly in the organic substrate. One piece, one attach step.

• CoWoS-L: silicon bridges embedded in an RDL interposer; that interposer then attached to the package substrate. Two pieces, two attach steps.

  

That one difference leads to these value props:

1. Cost

EMIB doesn’t have a separate interposer to amortize at all. The silicon bridges are small dice embedded in a package substrate that already exists.

Let’s be precise about what’s being compared here. Every flip-chip package, EMIB or CoWoS, sits on a panel-made organic substrate. CoWoS-L additionally builds and attaches a separate interposer (an RDL carrier with small silicon bridges embedded in it) between the dies and that substrate. That extra interposer, plus the extra process steps and the extra attach, is the cost difference. And in current-gen CoWoS-L that interposer is built in round-wafer format, so the panel-vs-wafer waste from the next section applies to it too. EMIB just doesn’t have any of it. The bridges are cheap because they’re tiny, and you get thousands per wafer.

Process steps eliminated:

• Interposer build

• Interposer dicing

• The interposer-to-substrate attach

That’s three places where cost and yield could go wrong, but won’t for EMIB, because they don’t happen.

2. Panel utilization

This is a big one, even if it sounds boring, and it matters more the further along the roadmap we go.

Silicon interposers are cut from 300 mm round wafers. Packages are rectangles. Rectangles on round wafers leave significant edge waste, and the waste fraction grows with interposer size. The bigger the interposer, the more wafer area you throw away at the edges.

Substrates use rectangular panels (a common size is roughly 510 mm × 515 mm). Rectangles tiled into a rectangle. The math is much friendlier.

Intel cites approximately 60% wafer utilization for interposer-class CoWoS versus approximately 90% panel utilization for EMIB:

That’s the cost headline. On a flagship part with a multi-reticle package, you’re looking at a substantial cost-of-goods delta before counting anything else.

And remember the Rubin Ultra rumor from up top? The fix that keeps coming up is CoPoS (Chip-on-Panel-on-Substrate), which is TSMC moving its advanced packaging off round wafers and onto rectangular panels. CoPoS isn’t EMIB. TSMC keeps its carrier-and-RDL approach; it just runs it on a panel. But the package got too big for a round wafer, and the fix is panels. Intel’s package substrate was a rectangle from the start.

3. Scalability past the reticle limit

This one is just geometry.

A reticle is roughly 26 mm × 33 mm = 858 mm².

• A 5-reticle complex (think Blackwell-scale, roughly Rubin-ish): ~4,290 mm², or about 43 cm².

• A 14-reticle ceiling: ~12,000 mm².

A 300 mm wafer has π × (150 mm)² ≈ 70,686 mm² of total area, and the usable rectangular yield is meaningfully smaller once you account for edge waste and dicing kerf. At 14-reticle interposer sizes, you’re approaching one interposer per wafer.

One. Interposer. Per. Wafer.

At that point the interposer absorbs the entire cost of the wafer. Packaging cost stops scaling and starts cliff-diving in the wrong direction.

EMIB stretches in X and Y across say a 515 mm × 510 mm panel (≈ 263,000 mm² of usable area). The “one interposer per wafer” problem doesn’t arise.

So the cost curves diverge. The larger the package, the wider EMIB’s margin gets.

And packages keep getting larger every generation.

4. Yield through smaller bonded pieces

Bonding a single 5-reticle silicon interposer onto a substrate is a tough operation. You’re moving a ≈ 43 cm² silicon piece through reflow, and silicon and substrate have different coefficients of thermal expansion. Warpage at that size is a yield-limiting problem. Remember that alleged Rubin Ultra issue above?

EMIB attaches dies individually to the substrate. Each attach is small, locally thermally controlled, and decoupled from the others.

Small-piece bonding is inherently higher-yield than big-piece bonding. The yield advantage compounds with package size for the same geometric reason the cost advantage does.

That’s the case for EMIB on the merits. For paid subscribers: three forward scenarios with my thoughts, the “TSMC will just do CoPoS” pushback, the Amkor partnership that adds a second source for EMIB (and why it’s a 2028 story, not a 2026 one), and what it all means for how you read Intel Foundry’s hand. If those tickle your fancy, keep reading.

Read more

CEO Jim Anderson: A significant portion of the sequential growth we expect in the current quarter is driven by 1.6T adoption. As a reminder, earlier this year at OFC, we were the only company to demonstrate 3 different types of 1.6T transceivers based on 3 different types of laser sources; silicon photonics, EML and VCSEL.

My “pay attention” radar went off when I read this. Why all three? Are they hedging bets here? Which bet is better? What are competitors betting on? Feels like a tech inflection opportunity that could shake up industry and market dynamics. Will one or more of these fail?

Answering those questions first requires foundational knowledge in each technology and current market dynamics.

Let’s start with VCSELs for datacenter communications. We’ll start simple.

• What is a VCSEL?

• Why has it dominated the short reach data center market for decades

• Why is the 200G-per-lane jump giving everyone heartburn?

What Is a VCSEL?

VCSEL stands for Vertical-Cavity Surface-Emitting Laser. The “vertical” part is the giveaway. A VCSEL is a tiny semiconductor laser that fires light straight up out of the top of the chip:

This contrasts with edge emitters, which emit light from the edge of the laser:

Honeywell began VCSEL research in 1993 in Minneapolis, moved it down to Richardson TX in 1995, and launched the first commercial product in 1996, initially targeted for datacenter communications. The history is a fun chain of acquisitions: Finisar acquired Honeywell’s VCSEL Group for $75M in 2004 (what a steal), II-VI acquired Finisar in 2018, and then II-VI acquired Coherent Inc. in 2022 and renamed the resulting entity Coherent Corp. So when Coherent talks about VCSELs today, it is effectively the heir to the entire 30-year US VCSEL development arc!

Check out this great infographic for more history. And note the volume of VCSELs shipped even a decade ago.

Commercial adoption was largely propelled by the Gigabit Ethernet (IEEE 802.3z) and Fibre Channel standards. VCSELs were a significant improvement over edge-emitting lasers in reliability, speed, cost, power efficiency, and manufacturability at similar wavelengths. For the first decade of existence, VCSELs were mostly used for short-reach datacom.

In 2017, high-power 2D VCSEL arrays were incorporated into the iPhone for Face ID, and consumer applications have since become the primary drivers of VCSEL production volume. Today VCSELs are everywhere:

• Smartphone face ID and 3D sensing. When your phone unlocks by reading the geometry of your face, that’s a VCSEL array projecting structured infrared light at you.

• Automotive LiDAR. VCSEL-based LiDAR is one of the architectures that self-driving systems use to build 3D maps of the world.

• Optical mice and touchless sensors. The dot of light under your computer mouse is (very often) a VCSEL.

• Short-range fiber communication and Active Optical Cables

ams OSRAM has a nice walkthrough of VCSEL technology for various applications, and their BELAGO 1.2 dot projector is a representative product.

As you know, below ~10 meters, copper cables still dominate but struggle as speeds increase:

Beyond a certain distance, single-mode fiber takes over to avoid modal dispersion. That distance is data-rate-dependent and has been compressing rapidly, historically around 300 meters at 10G-era links, around 100 meters at 25G on OM4, and dropping to roughly 30-50 meters at 200G PAM4.

That middle slice (multimode fiber, between where copper gives up and where single-mode takes over) is VCSEL territory. Note that the slice itself is shrinking as data rates climb.

Another important note: What matters for the analysis later is that VCSELs have a very mature supply chain. Billions of VCSELs ship per year (mostly into consumer devices). The reasons VCSELs got pulled into all those applications is because VCSELs are compact, energy-efficient, array-friendly, wavelength-stable, and uniquely testable on-wafer before any packaging happens.

Why We Care About VCSELs Right Now

VCSELs are at an inflection point in 2026. AI data centers consume short-reach optical interconnect at a scale nothing in computing history has matched, and the industry is racing from 800G to 1.6T with 3.2T on the horizon. To get there, incumbents are trying to double each lane from 100G to 200G, and 200G is where the current path struggles. And how would it even double again to 400G/lane?

This post (the first in a series) will explain why VCSELs have dominated short-reach data center optics for two decades and what, specifically, is breaking at 200G.

We will cover:

• The value props that built VCSEL’s moat at short reach plus wafer economics

• What came before VCSELs, and why edge emitters still own long-haul

• The named market players holding up today’s 800G/1.6T short-reach supply chain

• Why the 200G-per-lane transition is breaking the moat

Read more

On Monday, Microsoft and OpenAI amended their partnership. OpenAI is no longer exclusive to Azure; the license is now non-exclusive; OpenAI can serve all its products on any cloud. Like AWS.

On Tuesday, AWS launched Bedrock Managed Agents (powered by OpenAI) along with the new Connect family of agentic enterprise apps.

The same day, Ben Thompson published a joint Stratechery interview with Sam Altman and Matt Garman explaining the partnership.

By the way, AWS CEO Matt Garman is on X now! Less than 3K followers at time of writing. That will surely change:

Lastly, on Wednesday AWS posted very strong Q1 2026 results.

What to make of it all?

I’ve long been a bit bearish on Amazon. AWS is a phenomenal business, but at the corporate level it pulls Amazon’s margin up because retail is a tough low-margin business. Google is the inverse: GCP pulls Alphabet’s margin down because Search and ads print money. I’d rather be Google.

And my bearishness persisted. I hoped Amazon would see the AI shift as an opportunity to pour resources into AWS and expand its dominance and profit dollars, but instead, Azure and GCP ran ahead in GPU capacity. Plus Azure had an exclusive relationship with OpenAI, and Google’s own Gemini models seemed to keep up.

Man. If alpha was in the model, and GPUs powered frontier models, AWS didn’t have enough of the asset that mattered. Although Anthropic was showing some early promise…

But the agentic era is quickly changing things. No truer words, right?

The agentic platform, as I’ll explain, is sticky and value-accretive. AWS has an opportunity here. Moreover, Amazon’s Graviton and Trainium are well-positioned to serve it cost-effectively. And most importantly, Amazon’s own e-commerce operations are the best possible first customer for agentic platform primitives. But a platform needs an ecosystem, and SaaS is dead right? Nope. AWS shows a survival playbook for other operationally-grounded SaaS companies. And Connect family revenue is gravy on top, too.

Right now, AWS’ future looks bright; if they deliver on the vision, we’re in a period with dual mispricing opportunities for investors:

Read more

Will this hold? Officially in the AI supercycle? Let’s dig in and think through the implications.

From the earnings call:

Wendell Huang: In US dollar terms, revenue increased 6.4% sequentially to $35.9 billion, slightly ahead of our first quarter guidance. Gross margin increased 3.9 percentage points sequentially to 66.2%, primarily due to cost improvement efforts, a high capacity utilization rate, and a more favorable foreign exchange rate.

Q1 is usually the quarter when margins decline as smartphone sales slow after the holidays, and thus fabs run a little cooler and gross margin gives back 100 to 300 basis points. We can even see that was true this quarter for smartphones, down 11% QoQ:

Read more

In this interview, we walk through Meta’s ads infrastructure from first principles. A few things that surprised me:

• Recommender workloads have a different compute-to-memory ratio than a standard LLM GPU, and this difference gave rise to MTIA custom silicon

• Retrieval isn’t a generic workload either. Meta’s scale makes it memory-bound, which is why Andromeda got its own custom NVIDIA Grace Hopper SKU that Meta co-designed

• Meta’s adaptive ranking model is an LLM-scale recommender (~1 trillion parameters) served at sub-second latency. It’s distilled from GEM, Meta’s Generative Ads Recommendation foundation model, and scales compute per user based on interaction history length

• Consolidating N ad ranking models into one (Lattice) improved performance, not just cost. A single model trained across varied objectives outperformed the specialized ones

• LLM-written kernels (Meta’s KernelEvolve) flip the economics of heterogeneous fleets. Demand for software engineering is going up as the price comes down, and Meta now wants ~100x more optimized kernels per chip

We also cover how Meta’s GenAI and recommender systems teams cross-pollinate inside Meta, and what Meta’s infrastructure looks like two years out.

This interview is lightly edited for clarity.

How Meta’s Ad System Works

Hello everyone. Today we have a special guest, Matt Steiner, VP of Monetization Infrastructure, Ranking, and AI Foundations at Meta. Welcome, Matt.

MS: Thanks, great to be here with you, Austin. Thanks for having me.

What I wanted to get out of this conversation is to better understand Meta’s core advertising business and then how that drives infrastructure decisions. I’m going to assume listeners know nothing and we’ll walk through from first principles. At the highest level, how do ads work? What are the backend models that power Meta’s ad stack?

MS: Maybe let’s start with a quick overview of how the ad system works. On a very high level, an advertiser shows up and they say, “I have some creatives with some copy and I want to show them to some people.” Sometimes they pick explicitly who they want to show them to. Sometimes they say to our ad system, “show them to whoever is most likely to convert for the objective that I specify” — whether the objective is the person visits my website, the person adds something to a shopping cart on my website, or the person actually clicks buy on my website. Those are all different objectives. Advertisers can optimize for different things.

Once the ads are created, it is our job to record who these ads should be shown to. So we produce a big database and it says, “here are all the people that the advertiser would have wanted their ad to be shown to,” and we record in each person’s little mini database, “this is an ad that could be shown to Matt the next time Matt logs in.” Of course, that list of ads that could be shown to Matt the next time Matt logs in is very, very long.

So when Matt logs in and our front end asks for an ad, whether that’s on your mobile device on Instagram or Facebook, on the web — each front end queries our backend system and says, “give me the best ads to show Matt next.” The request goes through our systems and arrives at our indexing system, and our indexing system fetches all the ads that could be shown to Matt. That is where a piece of technology that we’ve talked about recently called Meta Andromeda comes into play.

A long time ago, we had a much shorter list of ads that could be shown to Matt. Today that list is extremely long, and to be able to process all of the ads that exist in that list we need to use a fairly powerful system. We worked with our hardware partners at NVIDIA and designed a custom hardware SKU with some GPUs in it, and we co-designed a machine learning model that runs specifically on that hardware SKU for the purposes of best assessing which ads are the top N ads to rank for Matt.

In the ads serving process, the two large steps are basically: find ads that could be shown to Matt, and then rank them to produce the top ads to be shown to Matt.

Andromeda operates in the first stage, which we call retrieval, and it uses a powerful machine learning model that has embedded some of my interests and past interactions to personalize which ads should be retrieved for me. Because not every product that is advertised to me is going to be a product that is interesting to me. So we’re basically sub-selecting the products and creatives that might be interesting to me in order to return to the ranking system to rank those.

The next step is ranking, where we apply these large and powerful machine learning models to figure out what is the right order of these ads in terms of highest conversion probability times expected value for advertisers. The ad system has a number of ranking models and they rank different ads based on the objective functions for the advertiser, and we have been on a long journey to consolidate those into a single ranking model using a technology we call Lattice.

The advantage of combining ads ranking models into a single larger model is of course cost savings. You don’t have to keep N copies of user interests in each machine learning model. You can keep one copy of a person’s interests in that machine learning model, which saves memory. You can compute the subnets for a machine learning model once instead of repeatedly computing the same subnets across a bunch of different models. You just do one computation. It’s more computationally efficient to have a single model. And then the other advantage is performance — a machine learning model trained on more data with more varied objectives performs better than a smaller machine learning model trained on all the data for that objective, partly because of the compute advantages, partly because of the memory pressure advantages, partly because each piece of data has some additional signal associated with it that the machine learning model can use to improve its own performance.

So: Lattice, consolidation. And then further along in the consolidation journey, we have built GEM, our Generative Ads Recommendation Model, which is our foundation model that we’ve tried to train on all of the data that’s available for Meta’s ad system to use to improve the probability of accurately predicting what somebody’s going to be interested in, what they’re going to convert when we show them an ad for achieving an advertiser’s objective. This large foundation model was then used to distill into smaller models that we could serve for specific purposes, encoding as much information as we can from the larger foundation model.

Now, like with any system, some people use it less and some people use it more. There are people that are very interactive with brands and content and ads. They’re commenting on the ads, they’re liking the ads, they’re interacting with the brand, they’re buying things from the brand. Those power users actually have much longer interaction histories with a brand or with all the brands together. It turns out that in our original architecture design, we did not have enough compute available to process all of those interactions given our extremely limited latency budget. For example, when a person shows up in a Meta property, we want to make sure that their feed loads and their ad loads in that feed in a certain fixed latency budget — let’s call it roughly one second. We want to have sub-second latency for all of our average retrieval requests. That means we can only process so many interactions when evaluating or inferring that machine learning model.

Recently, we’ve built a new ranking model called the adaptive ranking model that substantially varies the amount of compute used to evaluate the model based on how long a sequence from a user is of their interaction history with a brand or all the brands that are advertising on Meta systems. That way we can use a dramatic amount more compute for users with longer interaction histories and meaningfully increase the accuracy of our predictions about what they’re going to interact with next. That drives better results for advertising partners and much better experiences for the people that are seeing those ads. It’s all through the magic of right-sizing the compute and memory associated with each one of those requests, and right-sizing the model based on the amount of data that’s available to evaluate for a particular person.

Okay, this is so fascinating, there’s so much here. At the highest level, you broke it down to retrieval and ranking — retrieval was Andromeda, ranking was Lattice. With Lattice, you talked about having lots of models but trying to simplify that down into one model for many reasons. And meanwhile, the whole backdrop here is — what kind of scale are we talking about again? Three-plus billion daily active users?

MS: That’s exactly right. More than three billion daily active users across Meta’s properties worldwide. A lot of people seeing a lot of organic content in their feed, a lot of paid content in their feed, and interacting with both.

Take me back to GEM and remind me — we have retrieval and ranking, and where does GEM fit in?

MS: GEM is our foundation model. It’s the model that we train with all of the data that we can use for training to produce the largest, most sophisticated, most prediction-accurate model possible. At the same time, the model is so large it’s not servable effectively. So the model has to go through a distillation stage where a lot of the core learnings of the model are distilled into smaller models that are servable.

The next step after that was to try and make the largest possible servable model on the most powerful inference hardware we have available, to produce the most accurate predictions specifically for those users who are power users. They have long interaction histories with brands and content and interests that we can really do a lot better for — deliver them much better experiences and deliver advertisers much better predictions and consequently return on advertiser spend.

Long User Histories and Adaptive Ranking

Nice, and that’s where adaptive ranking fits in. This is really interesting, because I think people are starting to get used to the idea of a foundation model that’s so big you can’t serve it, and then the consequences and trade-offs of having smaller models that are servable. For listeners thinking of generative AI, they might be thinking of smaller models that respond faster but aren’t as “intelligent.” Broadly when people are thinking about generative AI, they’re thinking about optimizing for intelligence or for interactivity — how quickly does it respond. You talked about latency, but you also talked about being willing to spend more compute at inference time to get a better outcome for the advertiser and a better experience for the user. Can you talk more about the outcomes? Why does adaptive ranking and spending more compute because you have that longer history yield a better outcome?

MS: Maybe one way to think about this is: imagine that you’re married and you have an anniversary, and every year you buy something for your spouse — something that they like that’s in their interest set that’s not necessarily in your interest set. If you can look at a long interaction history for a particular person, and you see, “every September they buy this particular class of item,” you don’t have to even know that it’s their anniversary, but you can see in that long interaction history, every September they buy something in this category. Then you can use that information to make a better prediction for what they’re likely to purchase in September.

That’s one example, but maybe you have a history of purchasing specific things in specific months corresponding to your children’s birthdays or a holiday or an anniversary. You can see how looking at longer sequences of interactions can deliver much-improved predictions about what a person is likely to want and then what a person is likely to purchase based on those longer interaction sequences.

But you can only process those longer interaction sequences if first, you’ve stored longer interaction sequences, and second, you have the computational power available at serve time to be able to process that whole interaction sequence when a person logs in. Not everybody has long interaction sequences. Not everybody interacts every month with an advertiser, but some people do, and where the data is available to deliver dramatically improved experiences for those people, you of course want to give them the best possible experience you can. That is a function of whether you have the compute available to be able to process all that information within that latency budget through parallelization, etc., that GPUs and large-scale GPUs in the inference stack now allow us to provide for people. Better providing which products and services people are interested in delivers better results for our advertising partners as well, because we’re just matchmaking. We are matching the person who wants to purchase a thing with an advertiser who has the thing to purchase.

Yes, that makes a ton of sense. For me, you’re saying: if I only look temporally at the last month of what you’ve been doing, I could give you some ads. But you’ve been on Facebook since back when you had to get invited — so if I could look all the way back, maybe there’s interesting trends. But of course the trade-off — I’m thinking about an analogy to generative AI, which everyone can relate to. It’s kind of like context. I want a big model, I want to give it a ton of context, but that’s expensive and takes time. And with user-centric social apps, you’re thinking a lot about latency. So you’ve got that constraint of what is the most context I can give it, the biggest model I can give it, but still do it in sub-one-second. That’s a perfect segue to ask you more. You talked about co-designing with NVIDIA, you talked about GPUs. Take me back — did this stuff run on CPUs at one point? How has that evolved?

From CPUs to Custom ASICs

MS: Back in the day, retrieval of course ran on CPUs. And back in the day, even ranking ran on CPUs. There was always a push to deliver more compute for both retrieval and ranking, because the more compute available, the larger, more complex machine learning model we can evaluate, the larger the user history long-sequence context windows can be passed into those models, delivering better predictions.

We’ve been on a long march through smaller CPUs, medium-sized CPUs, larger CPUs, custom ASICs, GPUs, more sophisticated and powerful GPUs, more sophisticated and powerful custom ASICs. This is all in service of delivering better results for our customers at a reasonable cost to our business so that the ROI works out on both ends for both our advertising partners and Meta.

Okay, that’s amazing. What I heard you saying was: it’s been a long history for Meta of asking “how can we get more compute to serve better ads?”, which is a win-win — you’re in a marketplace with users and businesses and you’re sitting in the middle. This idea of using compute to do predictions better has been the story of Meta’s business for quite some time.

MS: At least the last ten years, we’ve been investing really deeply in performance-optimizing the hardware, the networks, the data center designs, the silicon chips themselves, the machine learning models, the software infrastructure, the tooling associated with them. It’s a very large, complex optimization CP-SAT problem that we have to satisfy to deliver the best results for our customers and for the people that use our products and services. It’s a really fascinating technology problem in addition to a business problem.

Co-Designing with NVIDIA

Yes, indeed. It’s an intersection of both. What did the practical process of hardware-software co-design look like when you were developing the retrieval engine, like with the NVIDIA Grace Hopper?

MS: We sit down with our partners and we say, “this is the amount of compute that we want to target for this particular use case. This is the latency budget. What are the configurable blocks you have in your portfolio that you could considerably make into a SKU, whether it’s a chip level or a hardware level, that would work for this particular use case?”

Our hardware partners have various configurations of machines and chips and boards available that they are willing to build in certain configurations. We looked at that and we said, “given the retrieval problem itself, it’s going to require a huge amount of memory. It’s maybe a little bit more memory bound than it is compute bound. So we need a lot of memory. We need a lot of specifically high-bandwidth memory, so there’s enough memory channels to keep those GPUs saturated when they’re doing that computation.” We wind up with a SKU design that is optimized for the retrieval space where it has the right amount of memory, the right amount of high-bandwidth channels between the memory and the compute, and the right amount of compute that is effectively balancing that for that particular use case.

That design is maybe different than the hardware SKUs that you would use in ranking broadly or in serving a web page. But we had some great partners to work with on the hardware side. And of course, we have truly brilliant AI researchers on the modeling side, and software engineers for distributed systems that are optimizing the software infrastructure layer, and networking engineers who are optimizing how these machines talk to each other so that we can minimize end-to-end latency while maximizing the parallelism and compute we have available to deliver the best results for people and businesses.

So you sit down with your partner and say, “hey, we are a large customer. We have particular workloads that we run at scale and we know the shape of those workloads really well. This one with retrieval has these characteristics — memory bound, needs high memory capacity and bandwidth.” Does that lead you to look at those certain workloads that you have and ask, “what is the right shape of compute? What is the right SKU for retrieval versus ranking versus GEM training versus adaptive ranking?”

MS: That’s exactly right. We are always trying to work both sides of this problem. One problem is: how do we influence evolution of the hardware to better meet the needs of the software stack, and where we anticipate the software and AI stack is evolving over the next couple of years? Because — you’re probably familiar — hardware has relatively long lead times compared to software. On the other side of the problem, we are trying to influence the software stack evolution in a direction that is going to meet the hardware and maximize the potential of the hardware that’s going to be delivered to us this half, this quarter, this year, next year, and the following year.

We’re always trying to evolve them in similar directions. Sometimes there are hardware breakthroughs and we evolve our software stack to take advantage of those hardware breakthroughs. Sometimes there’s new software breakthroughs and we try to influence the hardware design in that direction to support those software breakthroughs. There’s a big discussion about this constantly across the industry. It’s particularly important given the rapid pace of innovation in the AI space — how quickly machine learning models are evolving, how quickly they are improving their performance and cost characteristics. It’s a wild time to work in the hardware-software intersection space.

MTIA: Recommender Systems vs LLMs

Totally. And obviously with transformers coming into existence, you’ve probably gone from more traditional ML into evolving toward transformer-based ones, and we’ll get there. But first, take me to MTIA. You talked about CPUs, you talked about GPUs, and how with the Grace Hopper that fit nicely into particular workloads. What leads Meta toward MTIA? There’s been a lot of announcements on that front lately — showing a roadmap, partnering with Broadcom. Can you tell us the business and economic rationale for moving in that direction?

MS: We tend to think about this in terms of the evolution of our heterogeneous hardware fleet over time. We can see the offerings that are available from our hardware partners that have various configurations of memory and compute and memory channels and different ratios. Some of them work really well for a particular use case. Some of them work really well for a different use case. There are different trade-offs with running different machine learning models on each of those hardware configurations. Sometimes the trade-off is latency. Sometimes the trade-off is cost. Sometimes the trade-off is power. In this very complex constraint satisfaction and optimization space, you’re trying to figure out what is the best offering that maximizes your returns for your advertising partners and for your business as well.

That’s where sometimes we have a use case that is different from your standard use case in the space. That was the initial impetus for the Meta Training and Inference Accelerators. Ads is a recommender systems class of problem, which is a little bit different domain than your large language model class of problems. The large language model problem is what’s known in the industry as an embarrassingly parallel problem. You can process a bunch of stuff in parallel. It doesn’t have to have super-effective high-bandwidth communication to be able to sync up the weights at periodic intervals.

At the same time, in the recommender systems space, all of the data is personalized. In the large language model space, if I was to say to somebody, “complete the sentence ‘to be or not to…’” there’s an objective correct answer — the highest probability answer that almost everybody who speaks English and has taken high school English classes could guess what the next word is going to be. A machine learning model similarly can learn there’s an objective highest probability answer to that blank in that sentence.

Now in recommender systems, the world is not objective and highest probability like that. The question is: what is the next best ad to show Matt? And it is not “what is the next best ad to show,” because who’s looking at the ad slot dramatically determines whether the ad is going to matter to them. There’s no objectively correct answer to what is the next ad to show, but there is a highest probability answer to what is the next ad to show Matt. Every example that is fed into our training systems for recommender systems has to have that personalization attached to the example.

What does that personalization look like? Well, Matt likes gardening and cycling and seems to buy a lot of stuff for toddlers, a lot of cleaning products. As a result, things that fit in those domains may be much more appealing to Matt than things that are outside of those domains. I used to have hobbies, now I have young children. That’s changed what I purchase quite a bit. The machine learning model can encode that, and it changes what the correct answer is to that question of what ad should be shown to Matt next.

That changes the size of the data packet associated with each of those examples. You have to pass in this personalization blob for the example of “we showed this ad to Matt and Matt clicked on it,” or “we showed this ad to Matt and Matt didn’t click on it. Here’s Matt’s big personalization blob of things he’s interested in.” The machine learning model can learn, “with this kind of personalization blob associated with Matt, he likes cycling and toddler toys and gardening equipment. These kinds of ads are good ads to show Matt and these kinds of ads are not good ads to show Matt.”

But that literally changes the hardware characteristics that you want when you have a very different I/O ratio associated with each example. If your examples carry a lot more data with each example, then you have to have a much fatter network pipe to keep the chip fed. You have to have more memory on the hardware SKU to keep the chip fed. You have to have a lower ratio of compute to memory — and high-bandwidth memory at that — to be able to effectively utilize the compute. So the optimal hardware SKU for training recommender systems may not be the same as a GPU that is optimized for training large language models. There’s obviously pros and cons there, but you may want to build a SKU that fits that particular workload really well.

Now that’s not all of our workloads. We obviously use GPUs in a lot of places. We use them for a lot of different parts of the recommender systems problem. But for some types of models, we have a use case for a hardware SKU that has a different configuration than what’s commonly offered as a GPU-packaged SKU. For some circumstances, a custom SKU with a different compute-to-memory ratio makes a lot of sense. For other applications, the GPU SKU is much more performant or much more cost-effective for that workload. We’re really trying to optimize the available compute and memory to the available models that need to be trained and the data size with each of those models. It’s a fascinating, challenging technology optimization problem.

Heterogeneous Hardware and LLM-Written Kernels

Yeah, that was really helpful. I like how you illustrated the problem to show that there’s specific I/O requirements and memory requirements, and how that could lead you to think about what, of all the possibilities out there, what SKU would fit best for this particular type of workload — and that might involve making your own. Now that’s talking about recommendation systems, which is really useful, and it’s a good reminder that the business involves training and inferencing recommender systems. Now, you did talk about GEM as a foundation model and needing to train that, and it being so big that it’s not cost-effective to serve. Can you tell us more about the compute challenges and the infrastructure demands on creating GEM and serving GEM?

MS: GEM as our foundation model is the largest model that we train in the ads recommender space. We try to feed it as much of our data as we can feed into the model to produce the largest, most complex, and best-predicting model that we have available. Some of the parts of the model are not super efficient, and that makes it not very effective to serve, particularly if you’re latency constrained. That’s why we had previously done this distillation process.

Now we’re using this distilled GEM variant that we’re calling the adaptive ranking model, where it’s distilled to be efficient enough to be served, but it’s not nearly as distilled as prior models, which were much smaller. The adaptive ranking model is an LLM-scale and complexity recommender model for Meta, with roughly one trillion parameters in this inference-time model. And it gets evaluated at sub-second latencies, which is a pretty fun and interesting software and hardware challenge.

Sub-second latencies — that’s amazing. You’re talking about different SKUs and different workloads, and I’m tracking all that, and you mentioned at the end of the day you have a heterogeneous silicon environment — different vendors, some home-brews, some off-the-shelf, some custom. You talked about software, and obviously having to work internally to make sure your software is going to work with the hardware and vice versa. Can you tell me more about how you manage software across all that hardware? Because to the layman, that sounds like a lot of added complexity — but I don’t know how many different levels of abstraction you can have that makes it easier.

MS: In general, heterogeneous hardware is a challenging problem to solve because you have to make sure that each of your binaries not only is capable of running on that hardware, but is performance and cost effective on that hardware. This is where folks have historically been forced to choose between custom optimization of a binary on a particular hardware type, or translation layers, which abstract away a lot of the custom features of the hardware but also abstract away a lot of the performance improvements of the hardware as well. There was a very clear spectrum of performance trade-offs between abstraction layers, which make it simpler to deploy hardware but less cost effective, and customization of binaries for hardware, which is slow and costly to implement but much more performant and cost effective once implementation is done.

Recently, machine learning models have enabled really cool abilities to customize specific binaries for hardware such that you can now at scale deploy binaries that are custom modified and performance optimized for specific types of hardware rapidly and easily, without having an expert software engineer do those performance optimizations for you. We recently put out a paper, I believe we called it Alpha Evolve or Alpha Kernel, where a machine learning model — a large language model — will write a custom performance-optimized kernel for a particular binary or machine learning model and a particular hardware pair.

If we have a large number of machine learning models and a large number of heterogeneous hardware types, writing the custom hardware kernel that would optimize the performance of this binary on the hardware was very time consuming before. It’s effectively a matrix of custom software that had to be written and hand-tuned by an expert software engineer. Now we’ve entered an era where large language models with coding capabilities can produce these optimized kernels at extremely low cost, way, way, way cheaper than having someone sit there and meticulously pick through the various optimizations necessary to make this binary or model run on this particular type of hardware.

It’s a real breakthrough in the technology industry and it’s going to enable a lot more of that cost-effective optimization that allows you to take much more advantage of all of the hardware available to you. Now we’re thinking through all of our deployments of all of our binaries to all of our hardware. Whereas before we wouldn’t necessarily move a binary that was adapted to a particular type of hardware to another type of hardware because that would be high cost and maybe it wouldn’t be worth it — now we can ask the machine learning model to produce an optimized kernel for this binary or machine learning model on this hardware, and we can do a lot more active management of software running on hardware. Which is going to both lead to better performance for our advertising partners, better experiences for people, and of course lower costs for Meta, as we get to take more advantage of the hardware we have available to us and really right-size the hardware and software use cases together. It’s a long journey, we’re not done by any stretch, but some of the new breakthroughs here in AI are having really beneficial effects on our ability to optimize our hardware and our software for our business.

GenAI Cross-Pollination and the Road Ahead

Amazing. What a world we live in. Reflecting back a bit — where my head is at is, back in the day it used to be software engineers were very expensive, and obviously Meta has probably always bought a lot of compute. But I could see the rationale for not having heterogeneous silicon because then you have to hire a bunch of software engineers if you want to optimize it for every different piece of silicon. Or on the other hand you just say, “software engineering is expensive, so we’re not going to perfectly optimize.” But at your scale you want to perfectly optimize everything so that you can eke out lower latency or better results. And interestingly, now we’re in a world where you need to buy lots and lots of hardware for your business, but the cost of software engineering has gone down to some extent with the help of generative AI LLMs, letting you still have a fleet — a matrix of different tasks and different hardware — and yet you can use LLMs to help optimize and fill out that spreadsheet in a cost-effective way. Which is very awesome. That leads me to the question about generative AI. How is Meta thinking about the relationship between its core recommendation systems and infrastructure and the investments in generative AI? Not only using generative AI in your core business, which alone is really cool and interesting, but also I know that you are training generative AI and offering that to customers.

MS: There is a lot of crosstalk between our various AI experts in the generative AI / large language model world and in our recommender systems world. Not only is there crosstalk and collaboration on hardware and data center design and performance optimization for the distributed systems, including things like the model trainer — we are both really focused on optimizing the machine learning model trainer and optimizing various aspects of the performance that the system needs to be able to train much larger models and serve much larger models. There’s a huge amount of joint investment that effectively benefits both sides of the house, the large language model side of the house and the recommender system side of the house.

We have experts in both types of ranking on both sides of the house so that we can improve the performance using both domains’ techniques and capabilities. We are — maybe as evidenced by the pace of breakthroughs that we’re able to deploy in our services here — really seeing the benefits of the innovation in the AI space across both parts of the business today. That’s obviously very exciting. This is the weirdest, wackiest, most fun time to be a software engineer ever.

Yes, seriously. It’s fascinating to think about those different sides of the house and how they cross-pollinate and impact each other, and just how fast both are moving. What an awesome time to be at Meta, and what a crazy time. Last question — looking forward, maybe two years because the rate of change makes it hard to look further than that — what do you see as the primary infrastructure needs for the next generation of AI-driven advertising?

MS: You can see we are all investing very heavily in building out data centers and purchasing large quantities of compute and memory and storage so that we can build better machine learning models, so we can find better machine learning models. The process of identifying performance improvements is really training a lot of machine learning models, tweaking various optimization parameters, coming up with new architectures and testing those to really drive maximum performance benefits. So, large investments in machine learning model training, machine learning model research that leads to performance improvements for training, that lead to performance improvements at inference time, substantial investments to make sure that we can infer these large language models and other generative models and ranking models both more cost effectively, but also driving more compute available at serve time and more memory available at serve time so we can feed things like longer sequence histories and larger context windows into these models.

The overarching theme here is end-to-end optimization. We’re trying to optimize the data center designs with the networking designs and the SKU designs and the software infrastructure designs for the distributed systems and the machine learning model infrastructure, the machine learning models themselves, the data that goes into them — all jointly, so we can drive maximum performance together.

Maybe to your point earlier, the demand for software engineering has effectively gone through the roof as the price has gone down. Whereas before we would invest in a limited number of hardware optimization kernels to run software on, now we want 100 times as many software optimization kernels for each piece of hardware because it’s available now. We can have machine learning models produce that, and now we have our expert hardware performance tuners supervising these models instead of writing the optimizations themselves. The same thing is true at every layer of the stack where we’re doing this optimization now. The demand for custom software that is more performant than a generic abstraction layer has gone through the roof. Every team at every layer is trying to do much better optimization to produce better results per dollar, better results per watt of power used in these data centers. That’s really leading to these meaningful breakthroughs that you’re seeing in terms of performance all across the industry, but particularly for the business as well.

Yeah, what a wild cross-optimization problem, being vertically integrated in some respects from hardware through data center design all the way to the software, to the training and the inference. And then being able to use LLMs to help with all this super fast. What I like about what you’re talking about here is: you have to make all of these trade-off decisions, but there’s a clear optimization function that you’re solving for when you’re thinking of an ad-space business — an ROI, how much are you willing to spend, how much are they willing to pay, and how can better results lead to potentially paying more or the pie growing bigger. I’m just thinking out loud, contrasting that to maybe other players in the generative AI space where the economics aren’t quite as straightforward in making these decisions. Anyway — you guys have a lot to think through. My very final question for you personally: how do you stay on top of it all as it’s changing so fast up and down the stack?

MS: That’s a great question. I don’t think I have a fantastic answer. The rate of change is amazing. I try to use all of the AI tools available, including large language models, to summarize papers, produce a list of all the latest papers that have come out with breakthroughs that are relevant to the domain that I work in. I rely on a brilliant team of expert AI researchers to summarize the progress that’s happening in the space, how that should influence the roadmap that we’re building for the future. But the amount of information and the progress in the space is just wild. It’s really amazing and something to behold.

Yes, totally. Well, you don’t sound bored, that’s for sure. Awesome. That’s it for today. Thanks so much, Matt, for taking the time to educate us. I’ve learned a lot and I know everyone will really get something out of this, so thank you.

MS: Definitely not. Thank you for having me, Austin. Great to chat with you.

Substrate is controversial. The debate tends to focus on individual objections, such as a lack of industry experience or the impracticality of particle accelerators. But I want to zoom out and look at the elephant as a whole:

To do that, let’s use a simple framework to test whether Substrate’s strategy is actually sound.

How to Solve Problems

A while back, I was listening to an old Lex Fridman podcast with Ray Dalio. And Ray was talking about his “5-Step Process”:

1. Set your goal

2. Identify the problems blocking it

3. Diagnose the root cause

4. Design around it

5. Follow through

Not rocket science. But there’s power in simple frameworks as a lens to view the world.

These five steps resonated with me, as they pattern-match well with my lived experiences (entrepreneurship, engineering, academic research, home projects, and so on). Once I heard it, I started seeing it everywhere. Listening through the How I Built This backlog, I found it in the Crate & Barrel founding story from the 1960s. Let me show you how it applies there, and then we’ll try it on Substrate.

Crate & Barrel

Quick context: Summer 1961. Gordon and Carol Segal are getting married. Both 23. They registered for stylish European housewares, but nobody in their lives had the money or taste to buy it for them. The Segals couldn’t afford it either.

But on their honeymoon in the Caribbean, they found the same products at a fraction of US prices:

Gordon Segal: There was one Scandinavian store in the Virgin Islands. My wife picked up some of the items and said, “How can you have Danish 18/8 stainless at the $2.95 a place setting? It’s so much more expensive in America!”

And the Danish merchant there said, “We have salesmen from Europe come here, and we buy direct from factories.”

Ah! A goal, a problem, a root cause. Gordon’s wheels started spinning:

We got back to Chicago, and I was in the real estate business. She was teaching school. We were both sort of bored. And then, one night in February of ‘62, I was washing these dishes we had bought. I said, “You know, Carol, there had to be other young people like ourselves with good taste and no money. We should open a store.”

Through the Dalio lens (and listening to the rest of the podcast):

• Goal: Sell beautiful European housewares to young people with good taste and no money.

• Problem: Those goods were priced out of reach in the US, even though they were affordable in Europe and the Caribbean.

• Root cause: Middlemen in the US import distribution chain.

• Design: Skip the importers. Buy direct from European factories.

• Follow through: Hard work and grit.

The Segals didn’t pencil out this strategy cleanly from Day 1. But the Dalio process was at work.

Now let’s apply it to Substrate.

Substrate

Substrate CEO James Proud made the goal very clear in this Stratechery interview: revive American chip manufacturing leadership.

WTF? REVIVE AMERICAN CHIP MANUFACTURING? WHO DOES HE THINK HE IS?

Before you grab pitchforks, let’s work through the reasoning. We’ll address “yeah but no industry experience” and the rest later. First, the 5-step process.

Goal: American leading-edge semiconductor manufacturing

What obstacles stand in the way?

Money and talent come to mind first. But those aren’t fundamental bottlenecks. Think about Elon and Terafab. Money and talent are tractable.

Dig deeper. Assume you’re well-capitalized and talent-rich. OK, this sounds like Rapidus. We can cover them in another article.

Now what?

You know what’s actually hard? Creating a customer.

Problem: No one will work with you

Customer acquisition is the problem.

How could you possibly incentivize chip designers to use a brand-new, leading-edge foundry? You need a reason so compelling that it overcomes the risk premium.

On what plane can you even outperform TSMC?! They have 30+ years of process knowledge and relationships with every major chip company on Earth. And don’t say supply chain security. TSMC has n-1 capacity in Arizona and Intel Foundry is getting its swagger back…

Hmm…

Well, what pain points do chip designers have with TSMC today?

Cost is a big one.

The Cost Problem

Many companies can’t afford leading-edge nodes, and even those that can only use them for a select few SKUs. Design costs run in the hundreds of millions. Mask sets cost tens of millions. Only products with massive volume (smartphones) or high ASPs (Nvidia GPUs) can amortize that cost. And variable costs compound it. Leading-edge wafers are north of $20K and all signs point toward $100K by the end of the decade. So even for the highest-volume products, where fixed costs amortize to near zero, wafer price still matters.

Why is the leading edge so expensive? Lithography.

EUV tools cost hundreds of millions each and the cost is only going up. You need dozens per fab, a significant driver of leading edge fab requires tens of billions to build. Deeper background reading here:

But… what if you could somehow reduce the cost of lithography drastically?

What if you could offer a value proposition of “2nm wafers at 28nm prices”?

You could create customers. Existing medium and low volume SKUs that would love to use smaller, more power efficient transistors or build their own custom ASICs instead of using less efficient/broader off-the-shelf options.

Yeah, but EUV LITHOGRAPHY IS MAGIC! TIN DROPLETS! 30+ YEARS! ASML! NO WAY!

Yes, I know. Suspend disbelief for a bit and just follow Dalio’s process. Let’s pull on the thread more.

What is the root cause of the lithography economics problem?

XLight has rightly pointed out that Laser-Produced Plasma (LPP) is very expensive. The “shoot tin droplets and hit them twice with a laser to generate EUV light” part. And every ASML EUV machine needs one.

XLight says, “Why not use a much higher power free electron laser (FEL) and share that light source amongst many EUV scanners?” This unlocks much better economics, not only from decoupling the light source from the scanner but also by increase the dose which impacts productivity and wafer economics.

See more here:

But what if you pull on the root cause even further? Could it lead to a different solution?

The entire EUV system (light source, optics, scanner) is incredibly complex with known inefficiencies (many mirrors, lots of lost light…) all driving extreme cost.

Is that cost due to physics, meaning this is the globally optimal solution, and there is a physical limit preventing lithography from ever working differently?

Or did path dependence lead us to a local minimum, not a global one?

Is the root cause physics, or path dependence?

Retracing The Path

Substrate believes it’s path dependence. And you can actually retrace the history to test that claim. In the ‘80s and ‘90s, the industry was actively researching X-ray lithography, which has a natural resolution advantage from its shorter wavelength.

Hmm… maybe we could learn what the shortcomings were and if they are till true today!

There are great old papers on this, especially from IBM Research. Check out this paper “X-ray lithography in IBM, 1980-1992, the development years”. It’s super enlightening. Let’s keep pulling on the thread, from Alan D. Wilson’s paper:

Optical lithography, in 1980, was very poorly understood by the experts. On the basis of historical trends and current difficulties with existing tooling and technology, the limits of lithography were thought to be about 1–1.25 µm for optics. X-ray lithography, consequently, was targeted for entry around 1 µm, the perceived limit of optical lithography.

At the onset of the program the strategic advantages of X-ray lithography were stated to be high resolution (better than optical lithography), throughput superior to that of e-beam technology, better resist-processing characteristics, and potentially lower defects (no multilayer resists).

X-ray seemed promising at the time. What did they learn? How did they learn? Were they thinking about manufacturability or just science?

We spent the remainder of 1980 developing a financial and technical program plan for X-ray lithography based on synchrotrons, considering a number of basic questions: Where were exposures going to be done? What should the mask be made from, and how would it look? How would we develop a stepper/aligner system, and what would be the role of vendor assistance in this regard? What would be the staffing needs (the initial group included only six people) as the program progressed? And finally, what should the test vehicle be?

It was recognized early in the drafting of our program that we were targeting manufacturing, not device prototyping, but full manufacturing. Our manufacturing divisions would eventually be our customer.

That’s really sound and reasonable thinking. They were thinking about full production scale and manufacturability, not just the science. And if you read the rest of the paper, the X-ray lithography (XRL) technology actually worked:

We had made complex, fully scaled CMOS devices with 0.5-μm ground rules before our optical counterparts had reached the same level using a long-established technology. Our yield was also acceptable: not 100%, but acceptable. The principal goal of the X-ray program had been achieved.

There are even tips for Substrate and us to think consider regarding a practical particle accelerator:

For X-ray lithography to be viable in IBM, we needed to explore acquiring our own X-ray source… Early in 1982, Grobman and I were starting to learn about the physics of synchrotrons. Our interest in a synchrotron ring for IBM was kindled by reports from Munich of the design of a tabletop machine called Kleine-Erna…. We began this serious inquiry by visits to established rings in this country and in Europe. Perhaps not being part of the synchrotron establishment was beneficial: We could ask questions and find out what really made rings good and what did not, as well as who the real experts were.

A quick aside… Agree. Perhaps not being part of the establishment is a benefit at times. … And figure out who the *real* experts are 😂

This IBM researcher Alan D. Wilson is my dude.

Anyway, he continues

Armed with this information, we asked ourselves what the specifications for the ring should be… I was invariably asked “What should a ring for industry be?” My answer was this: It should fit on a truck; plug into a wall socket; be reliable and available to operate 20 out of 21 shifts per week; have sufficient average output capable of sustaining a stepper throughput of more than 30 wafers per hour using an insensitive (having a sensitivity of -100 mJ/cm^) X-ray resist; be capable of being debugged/commissioned and assembled at the vendor, shipped intact to an IBM site, and rendered operational in a reasonable time at full specifications.

Make sure it fits on a truck and plugs into a wall socket. Quite practical!

And that kind of thinking is a lot different than the picture of particle accelerators I had in my head (i.e. CERN).

Oh last quick note from the paper, it seems that folks exploring XRL have always been doubted.

The establishment of a program was a formidable task, since half of this distinguished group seriously questioned the need for and the viability of X-ray lithography.

But IBM actually built a synchrotron (“Helios”) that fit on a truck and worked:

Oxford proposed a superconducting dipole system with a cold bore. The magnet turned out to be difiicult to construct but had excellent performance. The Helios 1 ring was completed and commissioned at Oxford, England, in October 1990. During the design and building of the synchrotron we visited Oxford and the Daresbury team every four to eight weeks over a period of three and a half years. The ring was shipped to IBM in March 1991 and arrived at East Fishkill on March 29, 1991. It fit on a truck, as shown in Figure 12(a), and we slid it into ALF that same day [Figure 12(b)].

The first beam was stored on or about May 20, 1991, and final specifications were met in January 1992. Our goal had been met and, in fact, exceeded, because the ring performs beyond specification [36]. Critics who thought design alone would not work were wrong. IBM and Oxford as a development team commissioned the Oxford ring in record time and with a very high level of performance.

If it worked back then… why don’t we have X-ray lithography today? XRL wasn’t without practical shortcomings at the time. This IBM paper from 1998 said that XRL was mature enough to possibly be introduced at 130nm node, but admitted manufacturing issues, for example with masks.

Nonetheless, there are challenges still to be met. Among the most important are the development and commercial availability of an improved e-beam mask writer; the ability to fabricate defect-free masks satisfying the image placement and critical dimension control requirements with good yields; the stability of the masks in usage (including the issue of possible radiation damage); the ability to correct for magnification errors; and the ability to satisfy the industry’s desire for a technology extendible to 70 nm ground rules. These issues are primarily manufacturing issues, as opposed to issues related to demonstrating proof-of-concept or feasibility, although demonstrating extendibility is still needed before the industry can commit to using XRL at 70 nm ground rules

XRL was physically feasible but had engineering problems to solve at production scale.

Meanwhile, optical lithography kept working far beyond the ~1μm wall Wilson predicted. So the industry kept pushing optical. First DUV, then EUV.

But a lot has changed in 35 years. US National Labs have spent years advancing particle accelerators and sources are now brighter, more reliable, and more compact. Computational lithography has improved significantly, too, and can help overcome the mask and proximity challenges that plagued IBM.

Substrate’s Design

So back to the Dallio process. The bottleneck is EUV-based lithography economics, and Substrate’s approach is to go back to the fork in the road and choose a different path. Could there be a global optimum, and could it be XRL?

It would require co-designing the light source, optics, and scanner as an integrated whole from scratch. But to me, there seem to be many cost savings possibilities on the table:

• One particle accelerator source feeds many scanners, like IBM’s Helios ring which had 16 beamline ports. The light source cost amortizes across many tools.

• No multilayer mirrors means no compounding reflectivity losses. Almost all generated photons are available at the wafer, vs. single-digit percent for EUV.

• No tin-droplet plasma source means no tin contamination, no collector degradation, no droplet generator maintenance.

• Single-patterning at leading-edge nodes. Fewer exposures = fewer masks, fewer etch steps, fewer defect opportunities, faster cycle time.

• In theory, the particle accelerator shouldn’t need cleanroom space. Only the wafer-handling end of the beamline sits in the cleanroom. This is contrary to EUV scanners which are massive and require substantial fab and subfab infrastructure:

So here’s the chain of reasoning so far:

• Goal: American leading-edge semiconductor manufacturing

• Problem: No one will work with a new foundry

• Root cause: The only lever that overcomes the risk premium is dramatically lower cost, and cost is dominated by lithography

• Root cause (a layer deeper): EUV’s cost comes from path dependence, not physics

• Design: Go back to the 1990 fork. X-ray lithography with modern sources.

This is sound. It sure seems XRL could genuinely untangle the lithography cost problem.

It’s believable. And it’s fundable. Substrate raised $100M from Founders Fund, General Catalysts, In-Q-Tel, and more. Founder’s Fund’s thesis is to invest in smart people solving difficult scientific problems where, if they succeed, the technology would be extraordinarily valuable. Substrate is a perfect fit. A 1% chance of reshaping the $1T+ semiconductor industry seems like just the type of asymmetric bet FF was built to make.

Of course, sound strategy doesn’t guarantee success. Execution is everything.

So can they actually pull this off? Behind the paywall I address the biggest objections head-on, work through whether XLight and Substrate can both win, and discuss the impact to TSMC and ASML.

It’s really, really interesting.

Read more

In this interview we discuss why Pope left Google to start a chip company, how to overcome the CUDA lock-in, and why frontier labs are the natural first customers. We get into the chip itself: a hybrid SRAM-HBM memory architecture that combines the low latency of Cerebras and Groq with the throughput of traditional HBM designs, and why that unlocks advantages across training, prefill, and decode. We also cover how agentic AI changes hardware requirements, how MatX uses AI internally in chip design, and the biggest skepticism Pope hears: can a 100-person startup manufacture at datacenter scale?

This interview is lightly edited for clarity.

Subscribe now

Origin Story

Hello listeners, we have a special guest today, co-founder and CEO of MatX, Reiner Pope. Welcome Reiner, for listeners who haven’t heard of you and MatX, who are you, what is MatX, what are you guys trying to do?

RP: Thanks, very happy to be here. As you mentioned, I’m CEO at MatX. What we’re doing is making the best chips for LLMs that is allowable by physics. My co-founder Mike Gunter and I, prior to MatX, were working at Google for a long time. Most recently, I was on the Google Brain Team training one of the LLMs at the time, and Mike was on the TPU team. There were a lot of things we wanted to do to make the TPUs much better for running LLMs. Things like running at much lower precision, having much more compute performance based on large matrix support, and generally optimizing for LLMs, reducing a lot of the other circuitry that was needed for non-LLM workloads. At the time, this was in 2022, and it turned out the best way to do this would be by starting a separate company, which is MatX.

So take me back, you mentioned 2022, you came out of Google, which I will say, it seems like everyone came out of Google that’s at the forefront of AI and hardware.

RP: It’s like the Bell Labs of the time.

Yes! There’ll be a book written 10, 15 years from now that we’ll get to go back and read and it’ll be fun for us to remember the good old days.

But okay, back to 2022. I think it was November 30th when ChatGPT officially launched. How much ahead of that were you guys thinking about this direction? Did you launch before ChatGPT? And how did that inflection point—the general public becoming aware of transformers—how much did that change your life in terms of fundraising, vision casting, hiring?

RP: As it happened, we left Google one week before ChatGPT was released. We did not know it was coming. But the historical context was that GPT-3 had been released more than a year earlier in this developer demo.

It was really hard to use. You had to get in the mindset of “I am writing a document and I want the rest of this document to be the response I’m looking for.” It’s not a chat interface at all, totally different. But if you were paying a lot of attention, you could see the potential. A lot of insiders in the industry were appreciating something big is happening here.

And the question, really, pre-ChatGPT was: these models are incredible, but they’re 100 times more expensive than the models we’re used to running. There are 100 billion parameters instead of under a billion. Can we even afford to run them? The simple economics doesn’t work out if you’re used to running software as a service where every query is free and now you have to spend cents per query. When you’ve got millions of queries per second, it doesn’t pencil out. The big question prior to ChatGPT was: okay, cool demo, but it’s too expensive. Can you actually productize it? And there was a lot of skepticism that you actually could.

ChatGPT demonstrated that you can, and not only that, but the product is incredibly valuable. What that meant for us was we had already seen that prices are going to be high. If prices are high, how do you make them cheaper?

It turned out to be quite difficult for us to fundraise even after ChatGPT. It took about two quarters for that to really land, when the impact on Nvidia’s stock price showed up. Then there was the realization: okay, this is using a ton of GPUs, everyone is buying a ton of GPUs. Eventually Nvidia reported these gangbusters quarters, and at that point investors started seeing the potential.

Okay, interesting. So you started by saying this is really transformational, but on the current hardware, it’s going to be too expensive. So there’s got to be a better hardware solution. Then ChatGPT launches a week after you guys leave, and I would expect investors to say, “I can see this is going to be productized!” But at the same time, Nvidia is the one capturing all the value and selling GPUs. So was the early skepticism just around: why will anyone buy hardware that’s not a GPU? Or did they quickly connect the dots that GPUs aren’t necessarily the most efficient?

RP: Some of the skepticism is definitely about why you would buy hardware that’s not a GPU. And then the other one is just: how do you compete with the world’s biggest company?

On the “why would you buy something that’s not a GPU” question, the big consideration is the software moat that Nvidia has. Everyone writes CUDA. Historically, we’ve seen how much software lock-in there is in so many businesses. Why is this one different? Isn’t there going to be software lock-in here? Would everyone really rewrite their software onto a different hardware platform?

CUDA Lock-In

Is there lock-in? How are you thinking about it from a software perspective?

RP: At this point, I think it’s proven that the lock-in is pretty weak. Barring Google, who has been on TPUs forever, all of the other frontier labs are multi-platform. OpenAI, Anthropic, Meta, X — they are all on Nvidia, many of them are on TPUs. There are Cerebras announcements, AMD, some Broadcom-developed chips as well. All of these players are multi-platform. That is the proof already that the software lock-in is not that great.

If you want to think about the first principles reasons why, it’s because software versus hardware lock-in is really a question of how much spend you’re putting on the hardware versus how much you’re putting on software engineering to support the hardware. This is really the first time that balance has changed, and it has violated a lot of people’s intuitions.

Historically, the whole history of software as a service is you’re paying really large salaries to a large software engineering team, and the compute spend is a small fraction of that. Engineering time is precious is the mantra. Of course there you have to prioritize the ease of software.

But this is totally turned around now. All of the frontier labs are spending tens of billions of dollars on compute. The salaries of the people writing software for that compute are very high, but still small in comparison to the compute spend. So the rational choice is to do anything you can to get hardware costs down, be multi-platform, get the negotiating power that comes from that.

I see, interesting. From first principles, it makes a lot of sense. Now that you’re going to spend so much money on hardware, how can you spend it correctly on software to unlock that? Even if it means you have a team writing kernels specifically for this architecture.

Fast forward from 2022 to now and we’re seeing everyone has multi-vendor silicon and it’s made your point. It’s very easy for you now. But back then, when you’re just starting and trying to raise that Series A, you clearly were trying to articulate that and hope that it came to fruition. Of your early investors, some of them must have believed. What got them to believe you in a world where it looked like Nvidia had all the GPUs and had the lock-in?

RP: Ultimately, all of early investing is primarily a bet on people rather than on technology. There’s a bit of both — you can have the best people in the world and have a business plan which doesn’t make any sense at all. But the premise that there is a physical product that we make that we will sell for dollars is a very easy business plan. It’s clear how you can make margins off of that.

In some sense, that’s even an easier business plan than starting a frontier lab. A frontier lab is like, “we’re going to make a model, we hope we can sell it in a product that hasn’t been defined yet.” With selling hardware, at least the business case is clear.

And for early seed stage investors, it’s primarily going off who we are, our backgrounds, and folks we’ve worked with who have vouched for us.

The Chip

And of course you have the credibility of having been TPU people at Google. Tell me, actually really quick question. I don’t know if I’ve heard you say this anywhere. Explain the name MatX.

RP: Matrix multiply. One angle is you remove “ri” from matrix. Another one is the X is a “times.”

Nice. So now take us into the first chip, the MatX One. I know you raised $100 million to get started, and then just a couple of months ago raised $500 million. We talk about [using that money to build] a chip, but I know you’re actually building a system. The goal is data center deployments. So with all of that context, tell me about the chip, but I want to get into the bigger system.

RP: A few of the core bets of the chip: primarily very high matrix multiply performance, higher than anyone else has announced in the market. There’s a whole story there, but in summary, the marginal returns on having more matrix multiply performance seem to be much higher than marginal returns on more HBM performance or other considerations. So you’ve got to invest in that first.

And then there’s this thing that had been like free money sitting on the table: get your memory system right. That is a combination of seeing two good ideas in the market. Nvidia, Google, Amazon have been all tensors in HBM — HBM first. Cerebras and Groq have been weights in SRAM. That gives you very low latencies, but it has some capacity problems. You can put those two together. It takes careful engineering and you need to balance the system right. It’s hard to balance the system right. But it is totally doable. That is the other thing we’ve done, and it gives some really big advantages in both latency and throughput.

I think a lot of people are now starting to connect with that as they see the Groq LPUs and Cerebras; they see the benefit of weights in SRAM for low latency. But of course you need HBM for high throughput and KV caches. Everyone’s starting to realize that context is awesome — the more context you can give a model, the more interesting insights you get. You made the right bet. Was that an architectural bet made from day one, based on first principles?

RP: Yes. One of the things we’re very good at is workload mapping to hardware, and creative new ways to do that that are more optimal, especially when you consider the space of what potential hardware could be. This combination of different memory systems was a core idea going in.

One of the things it really enables — if you look through the list of parallelism and partitioning techniques: tensor parallelism, expert parallelism, pipeline parallelism. The last one is the ugly stepchild in some sense. It misses a lot of the advantages of optimizing latency and memory footprint that the other ones do. It turns out that’s actually a memory system choice. This combination of SRAM and HBM actually makes pipelining work as well as the other techniques for the first time ever. We understood that, and that was what we were going after.

So back in 2022 when you’re making these early architectural decisions—the big systolic array, the right memory choice—you’re also thinking about mixture of experts and how different parallelism strategies require tuning those memory choices correctly. That’s IP and a differentiator for you versus someone who just says, “oh, weights in SRAM and HBM, let me go do the same thing.”

But reflecting back to 2022, I’m not sure mixture of experts was even out yet. So how much are you reading papers as stuff was happening in ‘22, ‘23, ‘24 and saying, do we need to tweak the architecture?

RP: We’ve been reading papers since 2017. I think the big and disappointing inflection point in 2022 was when Google stopped publishing. We were talking about how Google is where all the researchers came from. They had an incredible team in Google Brain and they were publishing everything, all of the good work they did. Very vibrant place to be. They stopped doing that in 2022 because of seeing the competitive market playing out. You could get all of the trend lines of where the best models are going until then, and then that stopped. DeepSeek publishing has been a pretty good reboot of that, but it’s sad that the volume has not been so large.

Research and Publishing

Totally. I will admit I haven’t read all of your papers on your website, but I see that you guys do some publishing still. How are you thinking about that fine line of what to publish and what not to? Because for talent, it is exciting to get to publish to the world and share what you’re thinking about.

RP: The ability to publish neural net papers is a differentiator for us in terms of hiring. We have two different areas of neural net research. We’re a small company, especially our ML team is very small because that is part of what we do, but it is not the main thing we do. We’re not selling ML, we’re selling GEMMs.

But the agenda of our ML team is twofold. First is attention research, specifically focusing on memory bandwidth efficient attention. That is quite aligned to where we see the future of hardware being. The second is numerics. Numerics has been the single best improvement in chip performance over the last decade. I think we have some of the best numerics talent and IP here.

In terms of what we publish: we don’t currently publish the numerics that goes into our chip. We will probably publish it on a one or two year delay after releasing the chip. But we do publish all of the attention research, because what we’re doing there is advocacy. We’re saying: hey, model designers, you should probably have these considerations in mind, especially when you think of future hardware that’s going to have a ton of flops but is going to be somewhat more memory bandwidth constrained.

Product Positioning

So you’re making hardware to sell at the end of the day, but you have ML researchers working on attention, memory bandwidth-efficient attention, and numerics. That informs your own architecture — extreme co-design. But you’re also trying to show model labs—the end customer—what’s possible. If they adopt your chips, how much will that change how they think about training or inference?

RP: We’re trying to not go too far outside of the comfort zone. If you want product-market fit, you have to mostly meet the customer where they are.

The way to quantify that: you can look at the chip specs and there are maybe five most important ones — HBM bandwidth and capacity, matrix multiply throughput, SRAM bandwidth and capacity, interconnect performance. Our attitude is we want to be at least on par with the best competition like Nvidia on all of these, and then substantially ahead on at least a few. The substantially ahead for us is obviously matrix multiply performance, also interconnect performance and SRAM.

There is no place where we are substantially behind in these big considerations. Maybe in some less LLM-relevant considerations we’re behind, but in these big five, we’re at least on par everywhere. That means the opportunity cost of switching to MatX is never too large.

But then the headroom you can get — if you want to maximize the benefit, you can tune your model. That means things like changing the balance between the MLP layer and the attention, more MLP less attention, or using some of our lower precision arithmetic. We have a range of precisions to get the biggest advantages out.

Gotcha. So you make sure that in these five most important areas, none of them are too weak to prevent a customer from switching. You’ll be there on every front. But then if customers take a step further and optimize for your chips, they’ll have more headroom, they can do more.

RP: Yeah, that’s it.

Customers and Workloads

Let me segue into who are those customers in broad strokes, the target customers for this chip system.

RP: The most interest has been from frontier labs, which is as expected. That is who we are designing for, and why they’re most interested is their spend is biggest. That also means the economics of being willing to tolerate a new software stack is biggest there too.

They also have this longer-term vision of three to five years out, which is where you need to be when you’re buying custom hardware. If you want to do really good co-design with your hardware provider, you need to be thinking on that time scale rather than just “I’ll buy what’s on the shelf today.” That’s where we’ve seen strong interest.

And this has shown up across all of the workloads — training, reinforcement learning, and inference both prefill and decode.

Nice, okay, let’s talk about those workloads.

Let me reflect it back. Your customers are going to be the frontier labs. They have the most compute spend, they are the most incentivized to squeeze as much intelligence as they can out of that. They’re thinking three to five years ahead. They are incentivized to not only work with all their current partners, but to always be listening and see what else is out there.

The market is telling us the defining workload of our time is LLM inference. You can optimize around the transformer, around splitting it into prefill and decode. We see that with Nvidia and with Dynamo. Everyone’s getting used to that concept.

The market narrative has gone from GPUs for everything to actually at the rack scale, maybe it makes sense to have some SKUs that run prefill and some that run decode. This is their way of saying those sub-workloads have different constraints — if it’s memory bound, have the right hardware versus compute bound. But I know you had a great podcast that everyone should go listen to with John Collison and Cheeky Pint. You talked there about being competitive on all those workloads — training, prefill, decode, RL. And it kind of felt like going back to the days of a GPU can do everything. So how are you talking with these partners about their different workloads, and how do you not feel like a salesman just saying “yeah, we can do that, we can do that, we can do that”?

RP: We just have to be honest about what the strengths and weaknesses are. Let’s give that a shot here. Our product has a really large amount of compute. Traditionally, training and inference prefill are the compute-intensive workloads, and decode is memory bandwidth-intensive. So you might think, MatX has a lot of compute, why would we use that on a memory bandwidth intensive workload like decode?

That’s where the joint hybrid SRAM-HBM design really shines. You spend none of your HBM bandwidth on loading weights. All of that bandwidth is spent entirely on KV cache. So you can get better use out of your HBM bandwidth than you can with Nvidia. But you also get the very low latency because the weights are stored in SRAM, like Cerebras and Groq.

Digging into that further: low latency means small batch sizes — that’s just Little’s law. The number of things in flight are smaller. The memory occupancy in HBM is proportional to batch size. So you can actually fit longer contexts in HBM than you could if the latency were larger. Low latency is not just a usability win, but it actually improves your throughput as well.

This is similar to what Nvidia is now doing with the Groq and Nvidia racks side by side, but there are some taxes you pay by them being in different packages. Putting the whole thing in one package is the first principles way to do that and gives you the most advantages.

Sure, that makes sense. You have a lot of compute. You make the right memory choices. Therefore you can do low latency and high throughput. And there are even benefits in the small batch size, low latency with respect to how the HBM is used. You talked about how Nvidia has essentially separate racks, the Groq rack in there, say Vera Rubin. You’re making one chip with benefits to both types of workload. How are you thinking about rack scale, interconnect, scale up, scale out?

RP: We have a lot of interconnect in the product. I think it is the most of any announced product, in fact. The reason: so you can support mixture of expert models with fairly small experts without becoming communication limited. Very sparse mixture of expert models are what primarily drive the interconnect requirements.

We deploy very large scale-up domains as well as supporting scale-out. The sizing of your scale-up domain is really driven by the sparsity and the kind of mixture of expert layers you want to support. You want to do the mixture of expert routing within your scale-up domain as much as possible — that is how everyone does it. Bigger scale-up domains allow bigger mixture of expert layers.

On topology, we do some interesting things with network topology. I won’t go into huge specifics, but contrasting what is in the market: Nvidia has done things like running everything through the NV switches. Google has these torus topologies. If you think about what you really want for mixture of expert layers, you can design something very custom for that.

I see, nice. That again aligns with the idea of designing not just the chip but the whole system for the specific workload, even down to network topology. That makes a lot of sense.

The Team

So how many people, even hand-wavy, do you have at MatX? We’re talking about networking, ML, hardware. Probably you have to think about cooling and operations and all sorts of stuff because it’s really data center design. Tell me more about the company. It must be very cross-functional — what’s it like there?

RP: For a product like this, it’s a relatively small team. It’s over 100 people. But some of these projects — Nvidia has 10,000 or 20,000 people.

Most of the team is hardware, which includes the core chip itself, the logic design, design verification, physical design, and so on. We designed the rack in conjunction with a partner as well. So we have folks looking at what is the insertion force of a board into a rack, cable density, power delivery, thermals. That’s going down the stack.

Going up the stack, we have a really strong software team writing the software stack that runs LLMs on our chip. And then we have the ML team doing exactly the research agenda I described. Very cross-disciplinary. I think it’s a super fun place to work because in one day you’ll have a conversation about physical insertion forces and at the same time functional programming or SAT solvers for compilers.

Agentic AI

Nice, sounds fun. So I’m thinking about your interdisciplinary team, everything you’re trying to build in your first system. And at the same time, the world is constantly changing. We’ve got agentic AI, Claude Code, OpenAI Codex, maybe an explosion of inference tokens needed. Opus is awesome but expensive. I can’t use my Mac subscription for Claude Code. All of a sudden Mythos has come out. And I’m wondering as a chip designer with ML researchers, how are you staying on top of all this? Are things changing that make you think in the next version of our chip we should do things differently? Or are you seeing it play out and feeling pretty confident, like, we can help this problem of awesome but expensive inference?

RP: Halfway through your question, I was like, is this going to be about how do we use agentic AI versus how do we serve it? Both are interesting.

How do we serve it: there is this ongoing trend where you see the incredibly fast pace of change in models, how people are using them, how they’re training them. But when you filter that through the lens of what does that mean for the hardware, it’s almost all noise — 95% is noise. The rate of change for what you need in hardware is much, much slower.

As that applies to agentic AI: what is it doing? It’s still doing decode. It’s still doing prefill and decode. Some things that are different: it has increased the demand. When the agent goes off and thinks for a long time and the user is sitting there waiting, you would like them to wait for 30 seconds instead of five minutes. So the demand for performance has gotten higher, but that’s within expectations. Demand is always going to get higher. That’s a great place to be.

One place where it’s actually a difference is sizing. Sizing exercises are what we do every day. One example: how long does the model sit idle while it’s waiting for a response from an outside system?

In a chatbot context, the model has responded to you, and then you as a human are thinking, maybe you’re going to type another message, maybe you never do, maybe you leave. That’s on the order of 30 seconds or a minute. The context for the model has to be kept in memory somewhere during that time, and you have to size that memory.

That has changed meaningfully in an agentic context where now the model is mostly waiting for tool calls — run a compiler, do a web search, check your email. The times for those are very different. Checking your email can run in seconds rather than waiting for a human to think. So the memories in service of that end up being smaller.

But then there are things like long-running jobs — running a compiler or running a place and route tool, which can take hours. I think that’s actually the biggest place it’s turned up: there is now increasing demand for storage systems for when the KV cache isn’t actively being used but is waiting for a response from an outside agent.

Yeah, interesting. So tell me, how are you guys using agentic AI?

RP: Most of chip design is actually software development in practice. The way you express a chip is you write Verilog, which is a programming language. It’s an unusual programming language because it’s massively parallel, but it is a programming language. Can you write that better with AI?

One of the things we look at: the places where AIs are most effective is when there is a well-defined objective function. Does this compile? Is the area good? Is the power good? How many tests does it pass? We look at our processes and say, can we do development in a way that puts it in that regime, which is really the sweet spot for AI development.

The other thing we do: in addition to Verilog, we use other languages. There are popular ones like Rust and Python, but also some less popular ones — in our case we really like using BlueSpec. It’s a hardware description language that comes from functional programming. We are looking into how we can make sure AI is really good at BlueSpec even though it’s a niche language.

Cool, interesting. I’ve never heard of it. Is that something you think about as a competitive advantage, or just generally wanting to make AI models better at BlueSpec and share this with the world?

RP: There are so few BlueSpec programmers in the world that we just want a higher pool of them, and then it becomes a competitive advantage.

Go to Market

I love that. Okay, since you’re the CEO, I’m going to go back to talking about customers, route to market. On the one hand, it’s kind of nice because maybe there’s only five or six customers that would be great, so any one of these would be a great anchor customer. On the other hand, probably everyone in this space is wanting to talk with them and work with them. What does it look like to say, we’re a startup, trust us, we’re building this thing, it’s going to be awesome? How do you have those conversations to address their concerns, and ultimately, how will they end up buying your first chip or your roadmap of chips?

RP: “Trust us” goes as far as your word goes, right? Not very far. So you need to prove it.

For us, proof means a lot of detail on the artifacts we have. What is the core architecture? What are the very specific details inside the chip? How do we organize the chip — we talked about this splittable systolic array, these are the different compute units inside the chip, how do they connect to each other? What is the instruction set? What is the software SDK?

We give all of this information to customers under NDA. It is a lot, and it is uncomfortable for us to give that information, but it goes a long way towards proving credibility.

Yeah, that makes a lot of sense. As far as the software, what is the level of effort they’ll have to commit to when they say, here’s yet another vendor, we’re excited about everything they told us, we believe them, but there’s probably still some effort to port?

RP: For sure. If you look at the sizes of teams supporting each of these multiple platforms, it’s on the order of 50 to 100 people per platform. Really good people doing kernel development, maybe building compilers, building debugging tools. I think that’s the ballpark of what folks should expect on our platform as well.

We want to help and we’ll do as much as we can to do that work for you rather than you needing to start it all yourself. But ultimately a frontier lab wants to protect its own IP, especially the model architecture. The last mile of kernel development is always going to remain in the frontier lab so they know specifically what they’re doing rather than giving it to us.

The first miles — giving a strong compiler and debugging infrastructure — is something we can actually do for you though.

One or two last questions. What is the biggest skepticism that you hear from people?

RP: One of the things we’re focusing on over the next few years is: how can we as a relatively new startup manufacture in massive volume?

It’s a really exciting opportunity. The projections for data centers over the next few years are in the many gigawatts, tens of gigawatts. I don’t know when we’re going to hit a hundred gigawatts. Nvidia chips sell for about $15 or $20 billion a gigawatt. You might multiply that by 10 or 100. It’s a really large commitment.

The opportunity is really large, but being able to get very quickly to selling such a large volume is also a substantial challenge, and some big parts of that are ahead of us. I think that’s a really exciting thing for us to do over the next year and a half.

Yeah, that’s a good point. It’s not just about building the system, it’s about can you scale it, can you production ramp it, can you get to huge deployments that people are comfortable with, that work, that are reliable. Okay, last question. Give me a hiring plug. You’re 100-some people, it’s very interdisciplinary. Why should people come work with you?

RP: Ultimately you have to believe in the product vision, and I think we just have the best product in the market. It’s designed from first principles for what LLMs really need, keeping in mind years of know-how and techniques of what is the right way to map applications to hardware. That’s the company vision. But the way we operate, it’s a very friendly and high-trust team with a ton of incredibly smart people. I think that’s the day to day of why it’s a really exciting place to be.

Sure, A-plus people enjoy working with A-plus people. Awesome, Reiner, this was great. I learned a lot. Thank you for the time. I’ll be fascinated to check in over time and see how things are going with you.

RP: Yeah, thanks Austin, it was really fun talking.

One might be arriving, but it’s not what anyone expected. In 2017, Benedict Evans predicted augmented reality would be next. Quite reasonable. But three months later, the transformer paper dropped. It took a while for that to change the world, but nine years on, the new device isn’t something even more mobile than a phone. It’s a box on your desk for your AI agents to live on:

Nvidia sells the DGX Spark as a “personal AI supercomputer.” AMD calls it an Agent Computer. Perplexity is shipping Mac Minis as a Personal Computer service.

Is this the beginning of a new S-curve? Or is it another iPad? If the agentic computer takes hold, it’s additive TAM. It carries meaningful ASP because it needs serious memory and compute. Does every knowledge worker’s desk eventually have two computers on it?

But there are headwinds. Recently, Anthropic banned always-on AI agents from using its Claude subscription plans:

That means running OpenClaw just got a lot more expensive. And that’s not the only headwind.

Read more

• Coherent makes its own EMLs, VCSELs, silicon photonics, and finished transceivers. It’s hard to find another public company that touches this many layers of the optical stack.

• Six-inch InP is ramping across four fabs with yields management says exceed 3-inch. The performance comparison against Lumentum is still playing out.

• Five growth vectors stacking: transceivers, OCS, DCI, CPO, thermal. Management says CY2026 is mostly booked and CY2027 is filling fast.

• Breadth vs. depth is the real question. Does a hyperscaler want one partner for everything, or best-in-class at every layer?

We’ve covered Lumentum’s and Broadcom’s AI optical infra businesses so far. Lumentum is a shooting star thanks to its laser performance plus tailwinds of industrywide supply scarcity. Broadcom dominates the datacenter networking silicon (Tomahawk switches, 1.6T DSPs) and is pushing direct-attach copper in scale-up for as long as physics allows, even as it builds CPO technology (lasers included) for optical scale-up.

Time for Coherent.

Coherent’s angle is vertical integration across the photonics value chain. The company designs and manufactures InP-based EMLs and CW lasers, VCSELs, silicon photonics, detectors, and finished transceiver modules in-house.

That puts it in competition with Lumentum, Broadcom, and Sumitomo at the component layer, and with module vendors like InnoLight and Eoptolink at the transceiver level. The stock has run from $45 to $250 in fifteen months and still trades at a lower forward multiple than Lumentum.

In August 2025, Coherent began production on what management calls the world’s first 6-inch indium phosphide production platform in Sherman, Texas and Jarfalla, Sweden:

If yields hold, that should materially improve Coherent’s cost structure and add meaningful InP supply to an industry that is currently constrained. How that affects pricing dynamics across the laser supply chain is one of the key tensions we’ll explore below.

Coherent’s vertical integration means it can supply components, modules, or systems across virtually every optical architecture a hyperscaler might adopt, from pluggable transceivers today to co-packaged optics and optical circuit switches tomorrow:

The bull case is that this flexibility becomes increasingly valuable as datacenter optical needs diversify. The bear case is that hyperscalers prefer to unbundle and multi-source each layer, buying best-in-class lasers from Lumentum, DSPs from Broadcom, and modules from whoever is cheapest.

Let’s walk through the business, the growth vectors, and the tensions that matter.

NFA, DYDD.

Coherent Corp

Coherent’s backstory begins with II-VI Incorporated, founded in 1971 and named after the II-VI compound semiconductor groups on the periodic table. Naming is hard.

II-VI’s original business focused on supplying engineered semiconductor materials and optical substrates that form the foundation of lasers and other photonic devices. Thus, the company operated at the lowest layer of the value chain, upstream of components and with limited exposure to finished products.

There have been many acquisitions along the way:

A few notable ones.

Finisar was acquired in 2019 for ~$3.2B. It was a leading manufacturer of optical transceiver modules at the time and brought significant VCSEL capacity for 3D sensing (for Face ID). The deal extended II-VI from raw materials into finished modules and added scale in hyperscale networking.

Coherent Inc. was acquired in July 2022 for ~$6.6B. Coherent was a storied laser systems company, founded in 1966, that built the first commercial CO2 laser and grew into a global leader in industrial laser systems, especially after acquiring Rofin-Sinar in 2016. It brought complete laser systems for cutting, welding, semiconductor lithography annealing, and display manufacturing, moving II-VI from components into full systems.

So II-VI started as a materials maker and eventually expanded to add photonic products and laser systems capability. They rebranded the parent company II-VI as Coherent Corp. in September 2022.

The result is a company with two distinct lines of business. ~72% of revenue comes from Datacenter & Communications (DC&C), the high-growth segment riding the AI optical buildout. The remaining 28% is Industrial; lasers for semiconductor equipment makers like ASML and Applied Materials, excimer lasers for OLED display fabs, silicon carbide substrates, and specialty materials.

The industrial business is slow growth, but it’s profitable, sticky, and generates nice margins. The slide above suggests Industrial is shrinking QoQ, but the 2025 Analyst day suggests Coherent thinks it can still grow 5-10% CAGR over the next 3-4 years.

Drag on the AI story? Or a diversified base? I tend to be comfortable with the latter — good source of margin dollars.

Coherent CEO Jim Anderson’s mandate is to reshape Coherent from a leveraged, margin-constrained conglomerate into a focused AI photonics platform, and the company is making solid progress. Revenue has grown from $4.73B in FY2024 to a ~$6.7B annualized run rate, while gross margins have expanded by roughly 500 bps to ~39% and EPS has scaled from $1.21 to over $5 on a run-rate basis.

At the same time, leverage has been reduced from over 3x to 1.7x, alongside a series of divestitures and footprint reductions to simplify the business:

Coherent bonus points: Was recently included in the S&P 500 and a $2B equity investment plus multibillion dollar supply agreement with Nvidia.

Vertical Integration

Coherent claims to have the broadest photonics portfolio in the industry.

Let’s walk through it.

First, indium phosphide (InP). Coherent has over 20 years of in-house InP capability spanning epitaxial growth, laser fabrication, modulators, photodiodes, and integrated subsystems.

One important clarification: Coherent does not grow its own raw InP crystals. It purchases InP substrate wafers from external vendors under 3-to-5-year supply agreements, then performs epitaxy and device fabrication in-house. This is the same fundamental supply chain dependency that Lumentum and Broadcom face, though Coherent has locked in multi-year contracts with multiple 6-inch substrate vendors to mitigate it. Worth noting that Coherent does grow its own SiC and GaAs crystals internally; it is specifically InP substrates that are sourced externally:

Gianmarco Conti, Analyst: Gianmarco from Deutsche Bank. You’re expanding indium phosphide capacity, but the raw indium feedstock is roughly 70% sourced from Chinese zinc smelters, which are now subject to export permit requirements with multi-month processing times. I guess my question is, how much visibility do you have on indium supply for the next 12 to 24 months? And are you actively diversifying sourcing away from China? Or do you hold strategic inventory buffer?

James Anderson, CEO: We actually have a very diversified supply chain for indium phosphide substrates. We have -- and I think I’ve shared this in the past, we have over five different substrate suppliers today, and we work with those suppliers, not just on the next -- you mentioned next 12 or 24 months. We don’t work on just next 12 to 24 months. We work on the next like three to five years of capacity that we’re going to need. So we have, in some cases, very long-term agreements in place. And that includes not just the substrates, but all the key inputs that go into that. So we believe that we have very good visibility into substrate supply. And so that capacity expansion that Beck showed is we have commitments from our suppliers to supply the necessary indium phosphide substrates to support that.

The 6-inch InP production platform is an important topic. Management calls it the world’s first, and it is now ramping across four sites: Fremont, California (3-inch legacy); Sherman, Texas; Järfälla, Sweden; and Zürich, Switzerland (newest addition). The economics, per management, are roughly 4x the devices per wafer at about half the cost compared to legacy 3-inch lines, with capacity doubling this year. If yields hold at scale, this could result in competitive cost structure relative to Lumentum (which is migrating from 3-inch to 4-inch) and Broadcom (believed to be on 3-to-4-inch). The timing of those yields is one of the main questions we’ll look at below.

On EML laser chips, Coherent manufactures 100G EMLs for 400G and 800G transceivers, 200G EMLs for 800G and 1.6T, and demonstrated a 400G Differential EML for 3.2T and 6.4T at OFC 2026.

Coherent currently uses a mix of internal and external laser sourcing. Anderson confirmed at the Morgan Stanley conference that Lumentum is both a customer and a supplier, suggesting that Coherent’s internal EML capability has not fully displaced external supply across all performance tiers. The 6-inch cost advantage may be closing the gap, but the performance comparison against Lumentum’s epitaxy is still playing out.

For CW lasers, which power silicon photonics transceivers and are critical for co-packaged optics, Coherent is in full production and ramping on 6-inch InP in Sherman. At OFC 2026, it showed a 400mW high-power version for CPO applications.

CW lasers are one of the key product families covered by the Nvidia supply agreement. On the OFC status update call, Beck Mason (EVP Semiconductor Devices) set out to reassure everyone that CW laser yields on 6-inch wafers are looking promising:

BM: We’re currently running three main categories of devices on 6-inch indium phosphide, EMLs, high-power CW lasers and high-speed photodetectors. And
all three of those categories, we’re seeing higher yield and better throughput efficiency on our 6-inch lines than we’ve been able to achieve even on our very mature 3-inch production lines.

On VCSELs, Coherent manufactures GaAs-based vertical-cavity surface-emitting lasers. These serve Apple under a new multiyear 3D sensing agreement, and Coherent plans to launch a VCSEL-based 1.6T transceiver in the second half of calendar 2026.

VCSELs are a lower-power alternative to InP-based solutions, but with shorter reach. CTO Julie Eng explained the tradeoff at the OFC briefing:

“The VCSEL actually is an interesting potential for silicon photonics because the power is very, very low… it’s basically between 4x and 5x lower power than the silicon photonics solution. But it doesn’t go as far. It’s shorter reach.”

That means VCSELs could be used for in-rack and near-rack scale-up, where power and density are prioritized over distance, typically under 100 meters on multimode fiber.

Management says Coherent has ample gallium arsenide capacity, which is relevant because GaAs circumvents the industrywide InP bottleneck. Anderson expects VCSELs and InP-based approaches to coexist rather than compete:

“In the VCSEL-based solution, you usually have the laser in it. And so there’s some pluses and minuses of that. But I do think that these will coexist in CPO/NPO just as they have in pluggable transceivers.”

Coherent also has its own silicon photonics PIC platform and demonstrated a 400G pure silicon PN-junction Mach-Zehnder Modulator at OFC 2026. This is a pathway to 3.2T transceivers via silicon photonics rather than InP, giving Coherent optionality across both technology approaches.

At the transceiver module level, Coherent ships full OSFP modules at 800G and 1.6T. At OFC, it showed 1.6T transceivers built with three different DSP solutions from three different industry leaders. That is notable because it positions Coherent as technology-agnostic at the DSP layer, in contrast to Broadcom, which makes its own DSPs and can offer a vertically integrated laser-plus-DSP package. Coherent is effectively saying it will work with any DSP partner, giving hyperscalers flexibility to choose.

Beyond transceivers, Coherent makes optical circuit switches (OCS) using digital liquid crystal technology, which is non-mechanical and has no moving parts.

Anderson said at Morgan Stanley that Coherent is engaged with over 10 customers and that “multiple customers have already deployed in real data center applications”. Revenue shipments began in Q4 FY2025. This is a different technology from Lumentum’s MEMS-based OCS, and the two approaches are competing for the same emerging market.

Finally, on the industrial side, Coherent has some interesting datacenter-adjacent materials. Thermadite is a proprietary material with what management describes as exceptional heat-transfer characteristics, which is being evaluated by large customers as a replacement for copper heat sinks in data centers. Coherent also has a thermoelectric material that can convert waste heat back into electricity.

Both are early-stage, but Anderson highlighted them at Morgan Stanley as longer-term growth opportunities that bridge industrial materials expertise with datacenter demand.

The bull case for all of this vertical integration is greater internal control over cost, supply, and iteration speed. Customers value having a single partner that can build EML-based, VCSEL-based, or silicon-photonics-based solutions, depending on the application.

Plus supply chain resilience across 60-plus manufacturing sites in 14 countries, more than 20 of which are in the United States (Q3 FY25 transcript).

JA: But the other -- the second point I would make in terms of supply chain resiliency is around vertical integration. And this applies to not just our data center business, but also to our industrial business, for instance, our laser business is if you look at a lot of the very key technology in feeds for whether it’s a data center transceiver or an industrial laser, we make ourselves, manufacture ourselves a lot of the very key components that go into our transceivers or laser systems or other products. And so that’s an important part of our supply chain resiliency and flexibility. So to the extent that there are changes in the landscape, the tariff landscape and to the extent we need to adapt manufacturing, move manufacturing to different places for the benefit of our customers, we certainly feel like we’ve got a very good, resilient, adaptable supply chain to leverage.

The bear case is essentially “doing everything means doing nothing best”. Lumentum’s epitaxy appears to be ahead (Coherent still sources some lasers externally). Broadcom’s system integration is deeper (laser plus DSP plus switch on-package). The risk is that Coherent ends up as a jack of all trades competing against specialists at every layer.

Growth Vectors

Coherent frames its growth story in two layers. The existing engines, pluggable transceivers (800G through 3.2T), DCI coherent transceivers, transport/transmission equipment, and optical components, collectively address a $50 billion-plus SAM per management’s estimates. These are shipping now and growing.

On top of that base, Coherent identifies four new growth engines that add over $20 billion in incremental SAM by 2030:

The stacking of these new engines on top of an already-growing base is why management says CY2026 is mostly booked, CY2027 is “filling very, very quickly,” and FY2027 revenue growth will exceed FY2026. Each new engine is at a different stage of maturity, so inflection points are spread over the next 18 months rather than concentrated in a single quarter.

Open Questions

So… Coherent has a very broad photonics stack with promising growth vectors at various stages of inflection. The stock has run from $45 to $250. The sell-side is overwhelmingly bullish.

Yet Coherent still sources some lasers externally, including from Lumentum. Broadcom has the full-stack CPO lead, even if its CEO says CPO is “not anytime soon.” Chinese module makers are winning volume at 800G. And the BIS Huawei investigation is still unresolved.

Which of these growth vectors holds up under scrutiny? Where is management credible and where are they hand-waving? How does Coherent stack up head-to-head against Lumentum and Broadcom across each product category?

I went through all of this against Q2 FY2026 earnings, the Morgan Stanley TMT Conference (March 3), OFC 2026 announcements, and sell-side research, then put together a three-way comparison with Lumentum and Broadcom.

Here’s what’s behind the paywall:

• COHR vs. LITE vs. AVGO: Head-to-head across every product category

• Six things to watch, each with a bull case, bear case, and what to look for next: the 6-inch InP bet, OCS liquid crystal vs. MEMS, CPO positioning, the margin path to 42%, BIS risk, and valuation

• How the latest quarter stacks up against each of those

• What the Street is saying and where analysts disagree

• Catalysts for the rest of CY2026 and into CY2027

and more!

Read more