Scale Up, Out Challenges Amplified For Clusters

Key Takeaways: Software can identify the efficient way for clusters to use hardware such as GPUs and then optimize the output against the power consumption. The CPU is playing a larger role in the shift from AI training to inference. While this reduces GPU workloads, it places more pressure on memory speeds. Agentic AI workflows and mixture of experts (MoE) models are driving innovations around the ability to keep KV cache in the larger memory when the agent is not active, and then push it back