I don't think it's all that counter intuitive. You just need to consider that information transfer is limited by speed of light - it naturally follows that access times must grow with size of data unless you are able to pack the information infinitely dense.
It's a great series for reminding us of that, though, and illustrating it well and actually putting numbers on it.
If this were the only factor, access time would actually grow with the cubic root of N, because you could arrange memory in a sphere, which grows with r^3.
The ultimate theoretical limit is the berkenstein bound, which implies that the information content of a region is bounded by its area, not its volume. This is where r^2 comes from.
I get that there's additional constraints - my point was simply that intuitively even without thinking through or knowing about additional ways the communication is constrained, you'll arrive at the necessity of an increase in latency just with the knowledge of the limitation of speed of light alone
It's a great series for reminding us of that, though, and illustrating it well and actually putting numbers on it.