From the help page below I can see there's some limitations for using Xbox controllers, but only Windows was stated. But I can see how it will be an issue on Android as well
Stadia-compatible controllers and screens
At this point, make sure to send feedback to Google through the Stadia app. The more people reporting it's an issue, the better chance it become a priority to fix.
Not defending Google, because this launch was kind of messy. But the reality is they don't even have all the functionality of the Stadia controller working yet, so issues with 3rd party controllers should probably be expected. I do agree with @Ivan you guys should make sure you report these issues in the app. Google, and particularly the developers, want to make money, and they know not everybody is going to buy the Stadia hardware. So if they know about these issues, they will try to fix them in order to keep the games and the platform selling.