I haven't ever had this happen and have played an hour or two at least of Destiny on Stadia since launch (trying to get used to playing this game with a controller......)
Sounds like your controller has a poor wifi connection, or you have a lot of interference. I'd try changing the channel your access point is using to one that isn't overlapping with other networks. Use any of the hundreds of wifi analyzer apps out there to determine this.
Edit: Hour or two per day. ![]()