ok, I had done all the simulations of network delays on 0.17.4 and setting it all up with 0.18.0rc6 was a pain with the hacked-in code I put to try and not flood the actual Lemmy peers with outbound queue.
So much time would have been saved by the whole project if 4 weeks ago lemmy.ml just started posting their logs.
There is a major problem, still in 0.18.0, if peer servers are offline. If we had lemmy.ml’s logs, the whole chain of resource problems get bunched up starting with this logging pattern:
let stats_fmt = format!(
"Activity queue stats: pending: {}, running: {}, retries: {}, dead: {}, complete: {}",
Try to keep my 4-server simulation running is a pain in the ass (brain overload), but I keep coming to the same conclusion. Damn, it is so frustrating why so much valuable production data is sitting on lemmy.ml servrer logs that reveal all this and the labor it has taken me to reproduce it on independent network.
I’m dumbfounded that nobody else sees that outbound federation is a ticking time-bomb and has already brought down several of the big servers. It’s right here, among others:
warn!(
"Sending activity {} to {} to the retry queue to be tried again later",
message.activity_id, message.inbox
);
Why keep the logs a secret when you are sending so many replication messages out? This is causing server crashes on all the big servers. Why hide these logs?
This isn’t just a programming project, this is a living network with dynamics based on user data and peer server connections and peer server outages. Nobody seems to give a crap about the dynamics of message delivery.
Why? Why would you care so little about the data.
Some not-yet analyzed output of comparing postings for comment count.
compare posts. Community name asklemmy@lemmy.ml server0 https://lemmy.ml/ server1 https://sh.itjust.works/ {"communityname":"asklemmy@lemmy.ml","server0":"https://lemmy.ml/","server1":"https://sh.itjust.works/"} posts timeConnect 0.943 timeParse 0.655 server https://lemmy.ml/ timeConnect 0.838 timeParse 0.491 server https://sh.itjust.works/ [ '1unfound', '2unfound', '3unfound', '4unfound', '5unfound', '6unfound', '7unfound', '10unfound', '23:17:SKIP?14?', '24:19:SKIP?17?', '26:22:SKIP?20?', '48:45:SKIP?43?', 'extrasOnJ:3' ] ------------ comments of posts ============== --------------------- POSTING 0 [ 1468095, 372144 ] Now taking submissions for a new banner image! timeConnect 0.255 timeParse 0.201 server https://lemmy.ml/ timeConnect 0.208 timeParse 0.001 server https://sh.itjust.works/ tree0 Comment count 11 tree1 Comment count 11 commenttMissing 0 unequal 0 server0 11 server1 11 --------------------- POSTING 1 [ 1565858, 506988 ] What podcasts are you listening to? timeConnect 0.272 timeParse 0.283 server https://lemmy.ml/ timeConnect 0.207 timeParse 0.187 server https://sh.itjust.works/ tree0 Comment count 39 tree1 Comment count 14 commenttMissing 26 unequal 0 server0 39 server1 14 --------------------- POSTING 2 [ 1564565, 505075 ] What if a Reddit admin is stalking Lemmy? timeConnect 0.184 timeParse 0.006 server https://lemmy.ml/ timeConnect 0.137 timeParse 0.001 server https://sh.itjust.works/ tree0 Comment count 9 tree1 Comment count 5 commenttMissing 4 unequal 0 server0 9 server1 5 --------------------- POSTING 3 [ 1565001, 506191 ] What argument are you still holding onto? timeConnect 0.255 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.146 timeParse 0.001 server https://sh.itjust.works/ tree0 Comment count 8 tree1 Comment count 3 commenttMissing 5 unequal 0 server0 8 server1 3 --------------------- POSTING 4 [ 1561990, 505023 ] What interesting Lemmy posts have you saved that everyone should save as well? timeConnect 0.168 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.13 timeParse 0.001 server https://sh.itjust.works/ tree0 Comment count 1 tree1 Comment count 0 commenttMissing 1 unequal 0 server0 1 server1 0 --------------------- POSTING 5 [ 1561692, 502885 ] [Discussion] What's your thoughts on AI generated search engines compared to other ways to find information? timeConnect 0.271 timeParse 0.385 server https://lemmy.ml/ timeConnect 0.236 timeParse 0.005 server https://sh.itjust.works/ tree0 Comment count 11 tree1 Comment count 5 commenttMissing 6 unequal 0 server0 11 server1 5 --------------------- POSTING 6 [ 1561365, 502281 ] Is there a fantasy art community? timeConnect 0.316 timeParse 0.005 server https://lemmy.ml/ timeConnect 0.204 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 7 tree1 Comment count 5 commenttMissing 2 unequal 0 server0 7 server1 5 --------------------- POSTING 7 [ 1561312, 502177 ] When I see the number of upvotes and comments, are they from the whole federation or just my instance? timeConnect 0.271 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.106 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 6 tree1 Comment count 4 commenttMissing 3 unequal 0 server0 6 server1 4 --------------------- POSTING 8 [ 1559762, 500134 ] If you hade a superpower what would it be? timeConnect 0.29 timeParse 0.289 server https://lemmy.ml/ timeConnect 0.187 timeParse 0.127 server https://sh.itjust.works/ tree0 Comment count 41 tree1 Comment count 20 commenttMissing 21 unequal 0 server0 41 server1 20 --------------------- POSTING 9 [ 1559585, 499867 ] How do we contact a mastodon instance from lemmy? timeConnect 0.232 timeParse 0.014 server https://lemmy.ml/ timeConnect 0.12 timeParse 0.018 server https://sh.itjust.works/ tree0 Comment count 15 tree1 Comment count 10 commenttMissing 5 unequal 0 server0 15 server1 10 --------------------- POSTING 10 [ 1559186, 499010 ] Tips for using Lemmy? timeConnect 0.241 timeParse 0.005 server https://lemmy.ml/ timeConnect 0.14 timeParse 0.011 server https://sh.itjust.works/ tree0 Comment count 17 tree1 Comment count 9 commenttMissing 8 unequal 2 server0 17 server1 9 --------------------- POSTING 11 [ 1559057, 506881 ] People who have their phone on vibrate and have video doorbells, how many rings do you miss? timeConnect 0.182 timeParse 0 server https://lemmy.ml/ timeConnect 0.1 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 3 tree1 Comment count 0 commenttMissing 3 unequal 0 server0 3 server1 0 --------------------- POSTING 12 [ 1557161, 495445 ] If I actually wanted to drink the tears of my enemies, what cocktail would they work best in? timeConnect 0.276 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.226 timeParse 0.003 server https://sh.itjust.works/ tree0 Comment count 9 tree1 Comment count 9 commenttMissing 0 unequal 0 server0 9 server1 9 --------------------- POSTING 13 [ 1557147, 495175 ] How do you rewrite or delete all your old reddit comments? timeConnect 0.199 timeParse 0.164 server https://lemmy.ml/ timeConnect 0.12 timeParse 0.085 server https://sh.itjust.works/ tree0 Comment count 24 tree1 Comment count 13 commenttMissing 11 unequal 0 server0 24 server1 13 --------------------- POSTING 14 [ 1556987, 494761 ] I erroneously set my personal lemmy instance to private! how do i access it now?! timeConnect 0.224 timeParse 0 server https://lemmy.ml/ timeConnect 0.093 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 2 tree1 Comment count 1 commenttMissing 1 unequal 0 server0 2 server1 1 --------------------- POSTING 15 [ 1554510, 491568 ] What Wi-Fi networking gear you guys use? timeConnect 0.327 timeParse 0.168 server https://lemmy.ml/ timeConnect 0.194 timeParse 0.113 server https://sh.itjust.works/ tree0 Comment count 34 tree1 Comment count 24 commenttMissing 10 unequal 0 server0 34 server1 24 --------------------- POSTING 16 [ 1553293, 489777 ] Did anyone else get an email about ATM Surcharge Class Action Settlement? timeConnect 0.193 timeParse 0.023 server https://lemmy.ml/ timeConnect 0.134 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 7 tree1 Comment count 4 commenttMissing 3 unequal 0 server0 7 server1 4 --------------------- POSTING 17 [ 1552624, 489194 ] Moderators, what rule violations infuriate you the most? timeConnect 1.89 timeParse 0.112 server https://lemmy.ml/ timeConnect 0.418 timeParse 0.012 server https://sh.itjust.works/ tree0 Comment count 10 tree1 Comment count 10 commenttMissing 0 unequal 0 server0 10 server1 10 --------------------- POSTING 18 [ 1551349, 487134 ] Do you use adblock? Why? Why not? timeConnect 0.346 timeParse 1.985 server https://lemmy.ml/ timeConnect 0.227 timeParse 0.585 server https://sh.itjust.works/ tree0 Comment count 183 tree1 Comment count 126 commenttMissing 57 unequal 2 server0 183 server1 126 --------------------- POSTING 19 [ 1550630, 486266 ] What are your top 5 favorite BAD movies of all time? timeConnect 0.246 timeParse 0.405 server https://lemmy.ml/ timeConnect 0.609 timeParse 0.211 server https://sh.itjust.works/ tree0 Comment count 49 tree1 Comment count 40 commenttMissing 9 unequal 1 server0 49 server1 40 --------------------- POSTING 20 [ 1549912, 485383 ] Fast food go-ers of Lemmy, what are your "usual" orders? timeConnect 0.21 timeParse 0.396 server https://lemmy.ml/ timeConnect 0.195 timeParse 0.155 server https://sh.itjust.works/ tree0 Comment count 51 tree1 Comment count 36 commenttMissing 16 unequal 0 server0 51 server1 36 --------------------- POSTING 21 [ 1549004, 483930 ] Is anyone else enjoying the slower pace of content? timeConnect 0.214 timeParse 0.567 server https://lemmy.ml/ timeConnect 0.23 timeParse 0.447 server https://sh.itjust.works/ tree0 Comment count 126 tree1 Comment count 97 commenttMissing 29 unequal 3 server0 126 server1 97 --------------------- POSTING 22 [ 1548995, 483916 ] Are there any communities for lemmy beginners? timeConnect 0.233 timeParse 0 server https://lemmy.ml/ timeConnect 0.176 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 4 tree1 Comment count 4 commenttMissing 0 unequal 0 server0 4 server1 4 --------------------- POSTING 23 [ 1548500, 483348 ] My phone's WiFi adapter keeps turning itself on but nothing that I've done on previous phones seems to be working. Any ideas as to what I can try? timeConnect 0.209 timeParse 0 server https://lemmy.ml/ timeConnect 0.245 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 4 tree1 Comment count 3 commenttMissing 1 unequal 0 server0 4 server1 3 --------------------- POSTING 24 [ 1548501, 483349 ] are we going to have a Lemmy Party July 1st to welcome all the new users? timeConnect 0.347 timeParse 0.936 server https://lemmy.ml/ timeConnect 0.562 timeParse 0.611 server https://sh.itjust.works/ tree0 Comment count 64 tree1 Comment count 52 commenttMissing 12 unequal 1 server0 64 server1 52