Interesting read.
my takehome is that google is now building cabinet-sized NUMA clusters with
hot-swappable 1U "cards".

It's not the Aquila replaces ethernet, it's that the compute cluster has
eaten the ToR switch, and moved the ethernet ports outward.