I suggest Lemmy incoming federation inserts of votes, comments, and possibly postings - be queued so that concurrent INSERT operations into these very large database tables be kept linear so that local-instance interactive web and API (app) users are given performance priority.

This could also be a way to keep server operating costs more predictable with regard to using cloud-services for PostgreSQL.

There are several approaches that could be taken: Message Queue systems, queue to disk files, queue to an empty PostgreSQL table, queue to another database system such as SQLite, etc.

This would also start the basis for being able to accept federation incoming data while the PostgreSQL is down / website is offline for upgrades or whatever.

I would also suggest code for incoming federation data be moved to a different service and not run in-process of lemmy_server. This would be a step towards allowing replication integrity checks, backfill operations, firewall rules, CDN bypassing, etc

EDIT: And really much of this applies to outgoing, but that has gotten more attention in 0.17.4 time period - but ultimately I was speculating that the incoming backend transactions are a big part of why outbound queues are bunching up so much.

  • RoundSparrowOPM
    1·
    1 year ago

    So glad to see this and think this is super important conversation to be had if Reddit exodus makes their way to ActivityPub platforms such as Lemmy.

    The way many Lemmy instance operators are gong online, wanting to run on shoestring hardware budgets, I think a store-and-forward design like email or Usenet used to be (in the 1990’s) is a more proven design. And even it has to start as only “Lemmy to Lemmy”, some kind of bulk transfer session concept.

    I’ve even been tossing around the idea in my mind of some kind of (optional to have exposed) notification to end-users on delayed delivery, like beyond 3 minutes, so there is some awareness among the community that Lemmy isn’t run on the kind of budget Facebook is with hardware, operations teams, etc.

    Concepts like reliably deleting a message and editing a message I think are also going to need end-user education. On Reddit, edit wars aren’t unknown, especially in heated topics. Small time owner/operators are a different world from what users have known with the Big Guys.