I suspect the plug on your headset isn't getting fully inserted, really push it in there until it clicks. Of course the best thing would be to try/borrow another headset or even just any old headphones you have and see if the problem persists.
I tried with 3 separate sets of headphones, two of which are brand new. The plug is fully inserted. Sometimes the sound will smooth once a game starts, but sometimes it doesn't.
I am on wifi and my speed test is running just shy of a gig, but the games also seem to have more latency when using a headset. Both the Chromecast and controller are on 5GHz.