Enter High Bandwidth Memory 4 (HBM4), a 3D-stacked memory technology that promises unprecedented bandwidth per chip. This could determine which companies dominate – or disappear from – the AI landscape. This isn’t just another incremental upgrade; it’s the difference between training the next breakthrough AI model in weeks versus months, between profitable inference and burning cash with every query.
I’d think that the limiting factor would be the memory controller and bus, not the memory chips themselves. It’d be less cost-efficient, maybe, but I can just throw more physical memory chips into a server to buy more bandwidth at the memory level.
You can only get so much memory so physically close to a processor, and that places latency constraints, but I don’t believe that the parallel-heavy compute from AI workloads is particularly latency-sensitive, unlike many traditional workloads.
I’d think that the limiting factor would be the memory controller and bus, not the memory chips themselves. It’d be less cost-efficient, maybe, but I can just throw more physical memory chips into a server to buy more bandwidth at the memory level.
You can only get so much memory so physically close to a processor, and that places latency constraints, but I don’t believe that the parallel-heavy compute from AI workloads is particularly latency-sensitive, unlike many traditional workloads.