Ah, let me explain the problem better.
The chromecast connects just fine via 2.4 ghz. It's unable to connect via 5ghz or ethernet. But to stream to the chromecast you have to be connected to the router via 2.4ghz as well. If you're connected via ethernet or 5ghz you can't see the chromecast to stream to it.
I've tried several devices (ipad, android phone, desktop) and they all come back with the same result. If you connect to the network via 2.4ghz the chromecast works fine. If you connect any other way, you can't talk to the chromecast.
I've verified that the networks I'm connecting to are all on the same vlan.