I had this in farming simulator:
https://community.stadia.com/t5/The-Stadia-Controller/Headset-audio-crashes-chromecast/m-p/8006
But never tried in other games as I had no chance to play silent at night.