I finally got around to setting up the e-mails correctly for my instance (mander.xyz) and noticed that the e-mail deliver-ability is awful, as I get heavily penalized by spam filters for having an .xyz domain. I would not be surprised if search engines penalize this too.

It is not a simple task to change a domain name for an instance without breaking federation, so once you have a federating instance you are are stuck with that domain name.

I would not recommend picking an .xyz domain for your instances unless you intend it to be a private instance forever. If possible, try to go for a .com, or some other more standard TLD!