DGX GH200, each DGX Chassis will have 36 OSFP 400Gb port, which belong to 72 NVlink

Here are the timestamps and question summary, followed by a summary of answers: 15:10 On queueing packets into accelerator memory: Cary - Hybrid approach of on-chip vs. off-chip. Data paths are being built with batch packet processing in mind. Jim - Temporal locality of the packets and flows to compute engine, especially during initial flow setup. Mario - SmartNICs need to work at line rate; they should not queue packets, so most queueing is on the chip. Queueing may be more critical when working on packets at the messaging level, and then using off-chip memory is appropriate. Nick - Coming from an FPGA point of view, supporting high-speed external memory is critical for flexibility. Michael - Queues shouldn’t be kept on a chip; use host memory, particularly on entry-level NICs. You need to design your NIC, so it is balanced. The attached memory, DDR, is used on DPU to retain context. On-chip, there should be caches that are as small as possible. Rip - Keep expensive on-chip memory as small as possible. There is a case for slower and larger capacity memory; most systems will leverage tiered memory, where the host memory is the final tier. You’re only as good as the primitives you have for sorting, managing, and moving data.

27:35 How should classification and action on packets be handled? Jim - In any CPU architecture, you have caches, L1-L3-DDR, and as a result, added latency as you work on data; this is not the case with FPGAs. Based on the temporal locality of the data to the compute, you’ll find that these SoC-based solutions take much longer to process packets. Mario - It should not be fixed logic, as requirements and needs change over time. Our approach is to use ASICs with software running on processor cores to manage packets; that way, things are reconfigurable. Nick - We want to provide the user with maximum flexibility, as we know that protocols keep evolving. The FPGA allows you to change the way the hardware is programmed. Michael - Most things should not be done in the CPU; the majority of the traffic should go through the fast path you’ve designed for. Once you have the experience, you can use the CPU, which is on the slow path, the exceptions, for when you need the flexibility to do something not found in the ASIC. It’s a design that’s balance; the art is using CPUs with ASIC blocks for the best performance for what is required. Rip - Your most precious resource is your CPU, so by the time data reaches the CPU, your accelerators should have done everything they could. That way, the CPU could just do the thing that it's supposed to do. Cary - You don’t want to do classification on a general-purpose CPU. It shouldn’t be hard-coded logic as too much is changing in the market, something that is both flexible and programmable, almost on a minute-by-minute basis. Microcode engines, not CPUs.

36:00 On-chip network bandwidth, what is your approach, and how is it adapting? Mario - The architecture has to be hierarchical; you cannot possibly have all components talking to all components. You’ll need some coherent caches, and the design is not trivial; you may have more than one network on the chip. On-chip bandwidth has a huge impact on the performance of the DPU. The key is how you put together the various components, how tightly you integrate things. Nick - Flexibility is very important when tying the pieces together. We enable our customers so they can configure their architectures the way they want. We strive to offer the most flexible interconnect to meet their needs. At Achronix, our 2D NoC (Network on Chip) is finely tuned for this purpose. Michael - On the SmartNIC, the way you set up the on-chip network is much more straightforward than FPGAs. The right caching and buffering structure is one of the key things to having a successful product. Rip - With a hierarchical structure, as clock speeds and volumes go up, you’ll replicate more of the same components. We buy into that approach. What we aim to do with our architecture is to move data around faster; it turns out that half the time is spent moving data around. When things need to take a step up, make sure the core logic provides for that. Our challenge is, can we scale the way people write applications so that those applications scale up with increased traffic. A 10 GbE application is completely different than a 100 GbE application. The basic structure is the same, it’s the optimizations that matter. It’s not just about the hardware itself, it's about the software, the way you structure the software will ultimately influence how the hardware is designed. Cary - There isn’t an on-chip network per se; there is a series of optimized sub-systems that are interconnected via a crossbar switch. Taking an SoC DPU approach means getting these interconnects optimized and done right is hard work. Jim - It’s our job to figure out how to handle the internal plumbing. The strongest vendors in the market will be those offering customers the most choice. Working with hyperscalers, we’ve worked hard to tune systems to their specific configurations. We harden certain functions, like crypto, with that hardened data path, we can further tune and balance the system.

46:05 Balancing what we can do versus what we should do in hard logic as opposed to soft logic or programming, what are three DPU functions do you feel should be in hard logic? Nick - Interface logic like Ethernet and PCIe as well as TCAM and a classifier. Michael - Classification, data processing, compression, DMA. Rip - Assuming PCIe and DMA are hard, with regard to applications packet classification (L3-L4), crypto, both storage, and networking functions, and basic packet processing functions like LRO, checksum insert, checksum verification, those three are must-haves. Cary - Crypto, IPSec, adding something different, work scheduling across multiple components on the chip. Jim - On-chip data movers, compression, RDMA for, say iWarp, RDMA, etc...

49:25 Now that we’re thinking beyond 100 or even 400 GbE, how do you intend to scale your architecture? Michael - 400 GbE is kind of history; we’re working on 800 GbE, you scale vertically and horizontally. Vertically, you do things that use to be programmable now in hardware. Concurrency and more sophisticated data handling. Rip - Increase bus width, increase line rates, take most common soft logic functions and move them to hard logic. Adding more cores to handle greater packet rates. Cary - Optimizing sub-networks and data movers, diversification of that network to achieve higher rates. Jim - Packaging technologies to keep solutions efficient and as low power to keep power manageable. Software is important to make it easier for customers to move from different deployment models, on-prem to cloud, etc. Having worked at other companies on this panel, I’m not sure if some of the solutions of other panelists will scale as we go to 400 and beyond. Mario - All the devices are programmable, along the lines of software are tools, a compiler can make a huge difference. Also, the architect and the way the components are integrated can make a huge difference. Nick - Our 2D NoC is very capable of 400G, scalable to 1.6 Tbps. So I believe having a high-speed network on the chip allows you to move data between parallel processing blocks to achieve these high data rates. It's not all about the speed of the interface but traffic management and shaping.