I don’t have much of a homeserver, its more of an experiment - but I have Fedora IoT as hypervisor OS running a Open Media Vault guest and another Fedora IoT VM for container services.
I’m a big fan of Fedora’s Ostree setup, and have used Silverblue on the desktop for a while now, so IoT makes a lot of sense for me.
JabRef with some kind of cloud file sharing or sync? You don’t get a reference manager with a server component here, but maybe it could work. You don’t edit the bibtex file that often, so conflicts should be minimal.