ok, I had done all the simulations of network delays on 0.17.4 and setting it all up with 0.18.0rc6 was a pain with the hacked-in code I put to try and not flood the actual Lemmy peers with outbound queue.

So much time would have been saved by the whole project if 4 weeks ago lemmy.ml just started posting their logs.

There is a major problem, still in 0.18.0, if peer servers are offline. If we had lemmy.ml’s logs, the whole chain of resource problems get bunched up starting with this logging pattern:

https://github.com/LemmyNet/activitypub-federation-rust/blob/325f66ba324037a4f1d330a0dbea6e062ba34f50/src/activity_queue.rs#L117

let stats_fmt = format!(
            "Activity queue stats: pending: {}, running: {}, retries: {}, dead: {}, complete: {}",

Try to keep my 4-server simulation running is a pain in the ass (brain overload), but I keep coming to the same conclusion. Damn, it is so frustrating why so much valuable production data is sitting on lemmy.ml servrer logs that reveal all this and the labor it has taken me to reproduce it on independent network.

I’m dumbfounded that nobody else sees that outbound federation is a ticking time-bomb and has already brought down several of the big servers. It’s right here, among others:

            warn!(
                "Sending activity {} to {} to the retry queue to be tried again later",
                message.activity_id, message.inbox
            );

Why keep the logs a secret when you are sending so many replication messages out? This is causing server crashes on all the big servers. Why hide these logs?

This isn’t just a programming project, this is a living network with dynamics based on user data and peer server connections and peer server outages. Nobody seems to give a crap about the dynamics of message delivery.

Why? Why would you care so little about the data.

  • RoundSparrowOP
    1·
    1 year ago
    commentMissing 0 unequal 0 server0 11 server1 11 https://lemmy.ml/post/1468095 https://sh.itjust.works/post/372144
commentMissing 33 unequal 1 server0 46 server1 46 https://lemmy.ml/post/1565858 https://sh.itjust.works/post/506988
commentMissing 4 unequal 0 server0 9 server1 9 https://lemmy.ml/post/1564565 https://sh.itjust.works/post/505075
commentMissing 6 unequal 0 server0 9 server1 9 https://lemmy.ml/post/1565001 https://sh.itjust.works/post/506191
commentMissing 1 unequal 0 server0 1 server1 1 https://lemmy.ml/post/1561990 https://sh.itjust.works/post/505023
commentMissing 6 unequal 0 server0 12 server1 12 https://lemmy.ml/post/1561692 https://sh.itjust.works/post/502885
commentMissing 3 unequal 0 server0 8 server1 8 https://lemmy.ml/post/1561365 https://sh.itjust.works/post/502281
commentMissing 3 unequal 0 server0 6 server1 6 https://lemmy.ml/post/1561312 https://sh.itjust.works/post/502177
commentMissing 22 unequal 0 server0 42 server1 42 https://lemmy.ml/post/1559762 https://sh.itjust.works/post/500134
commentMissing 5 unequal 0 server0 15 server1 15 https://lemmy.ml/post/1559585 https://sh.itjust.works/post/499867
commentMissing 8 unequal 2 server0 17 server1 17 https://lemmy.ml/post/1559186 https://sh.itjust.works/post/499010
commentMissing 3 unequal 0 server0 3 server1 3 https://lemmy.ml/post/1559057 https://sh.itjust.works/post/506881
commentMissing 0 unequal 0 server0 10 server1 10 https://lemmy.ml/post/1557161 https://sh.itjust.works/post/495445
commentMissing 11 unequal 0 server0 24 server1 24 https://lemmy.ml/post/1557147 https://sh.itjust.works/post/495175
commentMissing 1 unequal 0 server0 2 server1 2 https://lemmy.ml/post/1556987 https://sh.itjust.works/post/494761
commentMissing 10 unequal 0 server0 34 server1 34 https://lemmy.ml/post/1554510 https://sh.itjust.works/post/491568
commentMissing 3 unequal 0 server0 7 server1 7 https://lemmy.ml/post/1553293 https://sh.itjust.works/post/489777
commentMissing 0 unequal 0 server0 10 server1 10 https://lemmy.ml/post/1552624 https://sh.itjust.works/post/489194
commentMissing 58 unequal 2 server0 184 server1 184 https://lemmy.ml/post/1551349 https://sh.itjust.works/post/487134
commentMissing 10 unequal 2 server0 50 server1 50 https://lemmy.ml/post/1550630 https://sh.itjust.works/post/486266
commentMissing 16 unequal 0 server0 51 server1 51 https://lemmy.ml/post/1549912 https://sh.itjust.works/post/485383
commentMissing 29 unequal 3 server0 126 server1 126 https://lemmy.ml/post/1549004 https://sh.itjust.works/post/483930
commentMissing 0 unequal 0 server0 4 server1 4 https://lemmy.ml/post/1548995 https://sh.itjust.works/post/483916
commentMissing 1 unequal 0 server0 4 server1 4 https://lemmy.ml/post/1548500 https://sh.itjust.works/post/483348
commentMissing 14 unequal 1 server0 66 server1 66 https://lemmy.ml/post/1548501 https://sh.itjust.works/post/483349
commentMissing 4 unequal 0 server0 53 server1 53 https://lemmy.ml/post/1548340 https://sh.itjust.works/post/483227
commentMissing 0 unequal 0 server0 5 server1 5 https://lemmy.ml/post/1546274 https://sh.itjust.works/post/480849
commentMissing 0 unequal 0 server0 0 server1 0 https://lemmy.ml/post/1545196 https://sh.itjust.works/post/479173
commentMissing 1 unequal 0 server0 19 server1 19 https://lemmy.ml/post/1543869 https://sh.itjust.works/post/477252
commentMissing 0 unequal 0 server0 2 server1 2 https://lemmy.ml/post/1543661 https://sh.itjust.works/post/476974
commentMissing 0 unequal 0 server0 22 server1 22 https://lemmy.ml/post/1543542 https://sh.itjust.works/post/476914
commentMissing 0 unequal 0 server0 16 server1 16 https://lemmy.ml/post/1543306 https://sh.itjust.works/post/476771
commentMissing 0 unequal 0 server0 7 server1 7 https://lemmy.ml/post/1543053 https://sh.itjust.works/post/476767
commentMissing 3 unequal 0 server0 27 server1 27 https://lemmy.ml/post/1542279 https://sh.itjust.works/post/475482
commentMissing 2 unequal 0 server0 32 server1 32 https://lemmy.ml/post/1542907 https://sh.itjust.works/post/476659
commentMissing 3 unequal 0 server0 21 server1 21 https://lemmy.ml/post/1542224 https://sh.itjust.works/post/475374
commentMissing 3 unequal 1 server0 16 server1 16 https://lemmy.ml/post/1540881 https://sh.itjust.works/post/473290
commentMissing 3 unequal 1 server0 47 server1 47 https://lemmy.ml/post/1540571 https://sh.itjust.works/post/472776
commentMissing 1 unequal 0 server0 24 server1 24 https://lemmy.ml/post/1541171 https://sh.itjust.works/post/473545