Federation likes (votes) are wildly different from server to server as it stands now. And unless something is way off on my particular server, 0.4 seconds is what PostgreSQL is reporting as the mean (average) time per single comment vote INSERT, and post vote INSERT is similar. (NOTE: my server is classic hard drives, 100MB/sec bencharked, not a SSD)

Discussion of the SQL statement for a single comment vote insert: https://lemmy.ml/post/1446775

Every single VOTE is both a HTTP transaction from the remote server and a SQL transaction. I am looking into Postgress supporting batches of inserts to not check all the index constraints at each single insert: https://www.postgresql.org/docs/current/sql-set-constraints.html

Can the Rust code for inserts from federation be reasonably modified to BEGIN TRANSACTION only every 10th comment_like INSERT and then do a COMMIT of all of them at one time? and possibly a timer that if say 15 seconds passes with no new like entries from remote servers, do a COMMIT to flush based a timeout.

Storage I/O writing for votes alone is pretty large…

  • phiresky@lemmy.world
    1·
    1 year ago

    well tuned a single insert should take less than a tenth of a millisecond on postgresql. if you’re seeing slow times it might be due to very different things, especially the huge locking hot_rank updates that make all inserts / updates on comments,posts table pause until done (which will show up as slower times in the query times you’re looking at)

    • RoundSparrowOPM
      1·
      1 year ago

      I mixed up the units, but it is taking .4 milliseconds typically. I really want to get some of these stats out of the major servers (Beehaw, Lemmy.world, Lemmy.ml)

      • phiresky@lemmy.world
        2·
        1 year ago

        yes, real stats would be real helpful. what about those user queries you said take 10 seconds? is that still true? maybe you could publish a new overview over what you find?

        • RoundSparrowOPM
          1·
          1 year ago

          what about those user queries you said take 10 seconds? is that still true?

          I made a major mistake (discovered yesterday), it’s been over a decade since I ran a major Postgres website. The units were fractions of a millisecond, not seconds. So that was 10 milliseconds.

          At this point I don’t have reason to hide the data, here is my two significant server stats:
          https://lemmyadmin.bulletintree.com/query/pgcounts?output=table Record counts in tables https://lemmyadmin.bulletintree.com/query/pgstatements?output=table pg_stat_statement extension output, curated columns. Full columns is available if you need it: https://lemmyadmin.bulletintree.com/query/pgstatements1?output=table

          Feel free to refresh these pages as much as you like. The pg_stat_statement can be reset, I think I reset them 24 hours ago.

          Server is 4 ARM cores, 24 GB of RAM, 200GB IDE level storage performance (Oracle Cloud). There are no interactive users other than myself and perhaps some Google traffic. Federation is the main activity, and I’ve been subscribing to as many big communities as I can for weeks.