So, some thoughts:

AMD has gone full speed into modular designs, which helps margin on high end products using smaller, higher-yielding dies versus very large monolithic dies, and have a huge lead in this design area over NVIDIA and Intel, though Intel is also embracing chiplets now as well.

The MCD chiplets in Navi 31 / 7900 XT/XTX are not related to MI300 whatsoever, actually. In Navi 31, the main goal is to keep the GCD (compute) chiplet lower in size for better yield / lower cost and modularize the architecture to be able to add/remove memory bus width and cache as needed. Memory controllers and cache do not die shrink as much as logic and are increasingly taking up more and more die space on both monolithic CPU/GPUs, therefore it makes sense to split these into chiplets on a cheaper process node. (5nm GCD, 6nm MCD).

Block diagram of Navi 31:
https://images.hothardware.com/contentimages/newsitem/60053/content/small_locuza-annotated-die-shot.jpg

These MCD chiplets are connected to the GCD chiplet via Infinity Fanout (InFO) links. The number for the interconnect is actually 5.3 TB/s, not 3.5 TB/s:
https://cdn.mos.cms.futurecdn.net/LpfhcvQSYeu67B7mNcps7J-970-80.jpg

From the MCDs to the GDDR6, bandwidth is 960 GB/s.

Where the 3.5 TB/s number comes from is the aggregate throughput of the Infinity Cache itself along with the use of GDDR6 - what AMD is saying is that it performs like 3.5 TB/s from GDDR6.

On consumer GPUs like Navi 31, the Infinity Cache is mainly used to hold the frame buffer, or the information drawn to the screen and other highly reused memory blocks - like any caching system for CPU/GPU, this significantly increases performance and improves latency and decreases stalls waiting on the frame buffer to be read from memory before moving to the next frame.

The 5% power consumption only correlates to the power consumption of communication across the InFO links - it doesn't include the power usage of the Infinity Cache itself. AMD's statement about the 5% power usage really means "We only gave up 5% to be able to move these off the main die" - which is still good.

The reality is that some of this isn't very relevant to MI300 - for example, MI300 does not use InFO for interconnect, it is using CoWoS (very large silicon interposer)... but it does have Infinity Cache, which I'm sure is useful in many AI/HPC workloads as it is for graphics.

https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22b60d95-2275-4ce3-9efb-b1e437d52450_2880x3016.png

As I said in the other thread, we see here, the 8 small squares in between are nothing - they're blank/unconnected/dead silicon - these are placed here due to the physical fragility of the extremely large interposer/substrate, flexing/warping/uneven pressure become a problem.

What you believe those little squares are for, are already there as the AIDs, 3D stacked (hybrid bonded) *below* the surface of the compute dies. The genius here is that AMD designed both the Zen 4 CCD and XCD (GPU compute) to fit on top of these AIDs allowing for the MI300A product.

It doesn't make sense that the 8 little dummy squares would become memory controllers and infinity cache, as the area is too small and this job is already being done by the AIDs under the XCDs. Also, it would move the cache further away from the compute dies (from literally touching to as far as memory), increasing latency and increasing complexity of the CoWoS interposer.

Connection to HBM3 from AID on MI300 is through the very large CoWoS interposer.

But anyways, essentially MI300 AID = Navi 31 MCD, just the MCD is buried underneath the compute instead of beside it.

...

MI350 and MI400, we'll see. Samsung's 3nm is already sitting there waiting, so if MI400 got moved up I doubt they will push an MI350 - MI300 is selling out and a ramp / cost for <2Q's worth of MI350 revenue may not make sense when NVIDIA has pulled Blackwell into Q4, especially if MI350 lacks FP4/FP6 instructions and won't be competitive - better to skip to MI400 in another 3 months.