https://github.com/LemmyNet/lemmy/issues/3395

This GitHub issue is getting ignored, just as data-size related crashes in general aren’t being reported by the major site operators to GitHub. The admin of lemmy.world posted yesterday that he “talked to the devs”, but all this talk seems to be behind the scenes and no Github issue was opened about 0.18.1 performance problems or any server logs as details. This is really holding bakc Lemmy as a platform, the lack of server logs from high-activity more-data servers being shared.

Is upvote doing backend-federation activity process spawning into the queue, is that is what is slow? Are database inserts into the comment_like table on that server taking so long given the amount of rows in the table from accumulated data?

The federation performance is also showing signs of serious backend delays that the server logs would show. And adding emergency logging to the code to establish what exactly is going on with timeouts, retries, etc.

  • RoundSparrowOP
    1·
    1 year ago

    Lemmy.ml via lemmy-ui was going really slow for likes for me at the time of this posting and comment. Creating post and comment seems fast, but like is incredibly slow. That’s also the largest database table (comment_like) I’ve seen in my testing (outside of activity table).

    The project is in crisis, but attention to this problem isn’t being requested. I just don’t get why nobody cares about actually how obvious the problem is and doesn’t want to describe what the code is doing and how we can meter the internals better and share logs.

    Lemmy.world admin Rudd says he backed out 0.18.1 because it was slower, but a mystery what logs said on the server.

    These tests are from lemmy.ml like of a comment, 15 second pause between each test, run right now at time of comment:

    0 timeConnect 10.502 timeParse 0.005 errorCount 0 https://lemmy.ml/  vote 1
1 timeConnect 3.193 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
2 timeConnect 4.248 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
3 timeConnect 3.835 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
4 timeConnect 4.913 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
5 timeConnect 9.233 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
6 timeConnect 8.625 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
7 timeConnect 3.625 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
8 timeConnect 12.638 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
9 timeConnect 11.224 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
10 timeConnect 5.861 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
11 timeConnect 7.311 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
12 timeConnect 24.308 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
13 timeConnect 3.421 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
14 timeConnect 4.823 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
15 timeConnect 5.223 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
16 timeConnect 6.4 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
17 timeConnect 5.853 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
18 timeConnect 6.907 timeParse 0.001 errorCount 0 https://lemmy.ml/  vote 1
19 timeConnect 4.239 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
20 timeConnect 5.266 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
21 timeConnect 14.185 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
22 timeConnect 4.951 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
23 timeConnect 20.759 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
24 timeConnect 4.519 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
25 timeConnect 5.701 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
26 timeConnect 8.513 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
27 timeConnect 6.29 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
28 timeConnect 3.622 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
29 timeConnect 3.302 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
30 timeConnect 12.157 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
31 timeConnect 13.847 timeParse 0 errorCount 0 https://lemmy.ml/  vote 1
  • RoundSparrowOP
    1·
    1 year ago

    If it is indeed the SQL INSERT that is the bottleneck, Emergency measures, code changes, could be put in place.

    1. Turn off the comment-return on like, make it a hotfix. Or fake it, just do a comment read to a like request and echo-back the vote parameter in the JSON.

    2. Insert into a new table in the database, comment_like_queue that has none of the index, record locking problems of the massive primary table.

    3. Batch insert all likes fro the comment_like_queue table into the live table every 3 minutes with a outside shell script or other quick and dirty application that can be easily tweaked and monitored.