You can change the chromecast volume while in-game by pressing the Stadia button, going to the audio section, and adjusting chromecast volume there.
I find the same with my Stadia. For me it's most apparent when switching between using the TV speakers and plugging my earbuds into the controller, since the volume I need Stadia set to (Chromecast Volume max) will hemorrhage my ears when I plug my headset in. At the least, it would be really nice if Stadia differentiated between speaker volume and headset volume, as most computers and mobile devices can do nowadays.