So I found out that the controller will stop working if it is connected to Chromecast
But not the pixel 2 xl
So I'm guessing the connection issue is between the Chromecast and the servers.