One thing to consider right now is with Covid19 on, more people will be using the Xbox services and that could increase latency a bit as well.
On the Xbox, what results do you see when running the built in speed test?
What does the test show for up/down speeds and latency on the Xbox?
Do you know where the game server is located? (The further they are away, the more likely you'll have higher latency)
Which game(s) are affected?