I think the Rust code is holding people back, and the queuing for federation and the overhead of HTTP connects for each individual comment and posting is too much.
Major Bypass Surgery is an option here. “Lemmy to Lemmy” MTA.
Two Birds with One Stone
-
The Client API is supposed to be high performance for loading 50 postings at a time, 300 comments at a time, Exercising the Client API is a good thing, identify the performance problems and log how it performs on multiple servers during replication.
-
Avoiding the Federation path that is currently being used in the Rust code for outbound sending will likely have an immediate improvement in server performance and stability.
-
Third bird: this tool build an API for the foundation of bulk back-fill between servers.
Changes to Rust code
-
New Client API to query “raw0” instead of “hot” and other sorting options. “raw0” sorting would be the local servers latest INSERT into the posting table without concern for pinned/sticky posts, votes, etc. Can we ORDER BY post_id itself, does that make sense?
-
Same with comments, can we ORDER BY coment_id itself on a posting?
-
Can we get the RAW JSON of the POST and COMMENT tables as output. Even if we need a api/v4 or api/bulk0/ path for a different output format?
-
Do we want to use Rust? Are we confident in the code not binding up the monolithic Rust application and should we consider having an api/bulk0 path into another app that directly accesses PostgreSQL?
-
Comments and Posts are the bulk concern. Federation of user profiles, community announcement, etc leave alone. Have a switch in the code to turn off SEND/OUTBOUND Comments/posts/votes - but leave everything else flowing through federation for a specific peer instance.
api/bulk0
-
A Query for a community to return a list of all peer instances that needs to have comments and posts delivered. At lest one subscriber on that remote instance.
-
A Query for an instance and all Communities that need comment and post delivery, have at least 1 subscriber on that remote instance.
-
Edited posts and Comments, Query all newly edited post/message from a specific post_id and comment_id forward. Perhaps limit query to 300 of each, but allow higher number in a list return format (without the body of he posting/comment).
Incomplete…
The current comment table doesn’t have a local/remote flag in the database? I think for a high-performance bulk API for comment replication you would want to have that flag on each comment. Maybe instead of having the current local/remote flag, just have the instance ID of the remote server, and 1 always represents local (isn’t that how server installs are coming out anyway, that 1 is the local install?).
Admin of lemmy.world tried to upgrade from 0.17.4 to 0.18.1-rc1 today. I commented:
The project leaders seemingly do not care about federation 24x7 problems with both performance and data loss. That they are operating a living network and there are aspects of server operations and even routine Internet outages, that can cause the code to overload. I think it is entirely possibly that peer servers are bring down each other.
As a server operator, having no idea of your queue and replication load to peer servers is nightmarish. And not being able to stop/start the MTA independent of the website.
Drastic Changes
To deal with the performance crisis of 0.18.1 release, maybe have a tool that removes all comments and posts older than 45 days to another set of PostgreSQL tables. And run this on a daily basis.
For 34 days I was hoping that the Rust community would come on the scene and implement PostgreSQL caching in key areas… but it hasn’t happened, and I think drastic solutions that don’t require Rust code changes would be good.
Old sticky postings and announcements would be an issue, but and SQL query to exclude those in moving out from database tables would be good. Also votes/likes from tables?