Well, that depends on the context. You're totally right in general, but for the purpose of order-of-magnitudes of maximal clock rates, the speed of light hardly varies. I can't find any normal, plausible processor material with a refractive index of 4, let alone 10. So while light in silicon may well travel [almost 3.5 times](http://refractiveindex.info/?shelf=main&book=Si&page=Chandle...) more slowly than in a vacuum - which is hardly nothing, at the granularity we're talking about I think it's not a huge issue. But that's just perspective - clearly the materials matter and will affect signal transmission speeds, and probably even today impose significant (if indirect) limitations.

It's possible weird dopings or quantum effect change this dramatically, I suppose. But if I had to guess, they don't.