I don’t write books but I’ve helped written a couple textbooks which used LaTeX. I personally use TeXstudio, but there’s many clients out there. If you appreciate beautiful typesetting, you’ll likely enjoy TeX despite its learning curve.

I’ve been using Obsidian lately. Proprietary with an open plugin ecosystem. Works well, makes it easy for me to integrate with other notes and such, but I haven’t figured out a good workflow for exporting work for submission. That said, it’s all markdown and there are lots of plugins for stuff like that, so it’s probably mostly just that I haven’t tried very hard.

In the past I’ve used Google Docs (proprietary), Scrivener (proprietary), Manuskript (open), Zim (open), and probably a few I’m forgetting. Really it just comes down to what you’re looking for out of the software, there are lots of options.

The biggest thing to keep in mind from a self-hosting perspective is local storage and easy backups under your own control. I use syncthing to keep my whole Obsidian vault synced across a few devices; for some software that’s easier or harder due to file formats and accessibility.

I don’t write novels, but lately found apostrophe (gnome) and ghostwriter (KDE) which are intended to write using markdown, and have a UI intended to allow you to focus on writing. You can later use git to manage versions and backups (in a remote repository).

If you want something more focused on relationships, and regarding the answer from another user suggesting Obsidian, you might use also logseq, but I didn’t use it yet (but hear a lot of positive vibes around it).