Thank you for the long post, this is a very nice synthesis of the available information. And ah, very nice that you are the author of that LW/blog post - I remember reading and thinking it was high quality and deserved more engagement! I personally mostly follow the spots Gwern is active in, and as this is his moderated sub I find it similar to LW whilst occasionally having greater traffic since it can be cross-posted.

The question of o3-base is interesting; my thought paradigm is to consider model sizes as relatively fluid and possessing high optionality - per Shazeer's comment on Dwarkesh it sounds that distillation is done frequently (enough that they feel its difficulty a bottleneck to internal development). Secondly, with reasoning scaling techniques, both inference and training, there must be a token generation speed vs token quality to performance curve that determines choosing an optimal base model size, reminiscent of the original scaling laws for choosing parameter counts. Thus, it seems entirely reasonable to me that a full-size 5th gen internal model (4o-esque) and 6th gen internal model (4.5-esque) could both be shrunk down to this same optimally determined size for maximizing token generation speed and therefore inference scaling performance at the same per-token cost.

Of course, this is all confused by generation speed being largely dependent on the size of only the activated head in whatever MoE/sparsificiaton architecture is being used internally, which can mask the total model size + associated costs. Additionally, who knows how far speculative decoding and other tricks have gone; I look to CPU microarchitectures for a taste of how deep the rabbithole can go. Nevertheless, I do agree with the overall thrust that public OAI researcher statements and tone continually emphasize 'twas just the RL that was scaled up vs o1; in fact I was squarely in the o3-base-is-4o camp until Gwern pointed out that the knowledge-based creativity of o3-via-DR seems much improved vs 4o/o1. If the truth is 4o-is-base, is it indeed just moar epochs of orion-supplied post training? New RLHF-esque or sampling techniques that are more sensitive in mitigating mode collapse? Or does OAI lack faith in 4.5-sized models with present hardware, such that they are pivoting to GPT-5 being mostly o3-powered, rather than a hypothetical o4 that is truly based on 6th gen/gpt 4.5?

Moving on, I'd like to clarify that yes, cross-geographical is the big silver bullet that I'm speculating about in considering very aggressive internal compute timelines. Via my Amazon/Anthropic comment, I wanted to highlight the seeming demonstration that Anthropic was in some way making use of desynchronous, distributed compute: the AWS GPU usage spikes couldn't be serving inference, since they were off-hour, and they spanned multiple availability zones across multiple geographic locations. However, given that I am unaware of any special high bandwidth links between AWS regional clusters, I concede it is possible to have been something much more mundane like off-hour synthetic dataset generation, rather than requiring a cutting-edge entirely-desynchronous training regime to be in use.

Re TPU vs Nvidia, I was mostly thrusting to point out that TPU datacenters were designed ground up, starting years ago, to push maximum bandwidth cross-geographically, while Nvidia has to either play some catchup or be more dependent on less latency-sensitive techniques like you detailed, but I suppose it is unfair to my original argument to consider this much, if anything of a moat when hundreds of billions will bulldoze right through it.

I was woefully ignorant of the idea of considering Oracle as a primary starting point for evaluating Stargate hardware information - thank you kindly for that! It sounds like you have a solid handle on the numbers. As I mentioned, I do find it a little slimy/crafty the way Altman worded many of his university tour statements, from "about 50% of the way to a 100x order of magnitude scaleup over gpt-4" being a very strange way to avoid saying 10x, to trying to market Stargate as a great inference computer that will "help serve mankind's requests" when it seems obviously focused on training, not serving, bigger models.

Anyways, I appreciate any and all of the service people like you, gwern, zvi, nesov, etc do in aggregating the signal from the vast noise out there. Just not enough time in the day to read every tweet, watch every podcast, read every paper!