I see the linking code on the TV screen. It's overlaid on top of my regular Chromecast wallpaper in the top right corner. I open the stadia app on my Pixel 4 XL and tap the controller icon at the top to begin the linking process. I pick the only controller in the list on the first screen on the phone app. I enter the wifi password. It then asks me to enter the code seen on the TV screen. I hit the 4 buttons, x left x a, I think. The controller vibrates and then goes to a blinking orange light. According to the manual, blinking orange means it needs to be configured with the wifi. But I just did that 2 screens ago. I've tried factory resetting the controller and the Chromecast several times and it always does the same thing.
@LiquidFlames Did you make sure that both Chromecast and Stadia controller are on the same WiFi network? The same thing happened to me once and it turns out my Chromecast was on a different network than the controller.
Blinking Orange means it can't connect to WiFi for some reason.
If you can choose a different SSID, try that. If you can't, try changing the channels that your router/AP are configured to use to different ones and try again.