How are you playing Stadia? Is it using a Chromecast Ultra connected to your TV, with the Stadia Controller connected to Wi-Fi, or is it on a computer/tablet/phone with the Stadia Controller (or another) plugged directly into the device?
If you are using the first-described setup, is the Chromecast Ultra using a wired Ethernet connection? I would expect, if both the controller and it are using Wi-Fi, for them to both have issues simultaneously, which is why I am asking.
I can only say I play in both environments, using a Stadia Controller in both, and do not have disconnection issues. The only time my controller has disconnected is when I wanted to see how long it would last before charging, and let it die on purpose.