@HudRed You're really going to have the best experience with the Chromecast wired to the network, if at all possible. Televisions seem to do a really great job of interfering with WiFi networks even with otherwise-ideal network connections.
What band is your wireless using (2.4/5GHz)? Can you use an app like WifiMan to make sure there aren't a lot of other networks on the same channels as yours?
What is the quality of the Wi-Fi at that spot in your home? If you live in a crowded area with many wireless networks on the same band and channels, or are far from the wireless access point, or simply have a poor wireless access point, the Wi-Fi signal to where the Chromecast Ultra is could be terrible.
Is that "45-50" Mbps number you mentioned what a speed test provides when on Wi-Fi, or is it what the link speed of the Wi-Fi connection is? Link speed is how fast the connection to your local network is, and it should be much, much higher. For example, my link speed over Wi-Fi is 866 Mbps, and the latency to the gateway over Wi-Fi is 2 ms. This is connected to the 5 GHz band on a Wireless-AC network, which is what Google recommends.
If you are able to test these things and see a link speed that is under 100 Mbps and a high latency to your own gateway, that is a problem. Worse still is if there is packet loss within your own local network on Wi-Fi, which is a serious issue. Packet loss should be near-zero on Wi-Fi, and should literally be zero on a wired connection.
I realize testing some of these things can be a bit difficult, but it is worth doing if you know how to or can get help. One easy place to start could be with your ISP if you have a combination modem/router/wireless access point from them, like most homes do. If it has not been replaced in years, you may be due for an upgrade and that could potentially fix things all at once. ISPs often will not be proactive with this, so calling them and asking whether newer hardware is available can be a good idea. It is often "free" with your Internet subscription, but that varies from provider to provider.
If it is straight up lag on Chromecast, are you absolutely sure it isn't just the input latency on your TV? Triple check that you have game mode enabled on the input for your Chromecast on the TV.