ok, I had done all the simulations of network delays on 0.17.4 and setting it all up with 0.18.0rc6 was a pain with the hacked-in code I put to try and not flood the actual Lemmy peers with outbound queue.
So much time would have been saved by the whole project if 4 weeks ago lemmy.ml just started posting their logs.
There is a major problem, still in 0.18.0, if peer servers are offline. If we had lemmy.ml’s logs, the whole chain of resource problems get bunched up starting with this logging pattern:
let stats_fmt = format!(
"Activity queue stats: pending: {}, running: {}, retries: {}, dead: {}, complete: {}",
Try to keep my 4-server simulation running is a pain in the ass (brain overload), but I keep coming to the same conclusion. Damn, it is so frustrating why so much valuable production data is sitting on lemmy.ml servrer logs that reveal all this and the labor it has taken me to reproduce it on independent network.
I’m dumbfounded that nobody else sees that outbound federation is a ticking time-bomb and has already brought down several of the big servers. It’s right here, among others:
warn!(
"Sending activity {} to {} to the retry queue to be tried again later",
message.activity_id, message.inbox
);
Why keep the logs a secret when you are sending so many replication messages out? This is causing server crashes on all the big servers. Why hide these logs?
This isn’t just a programming project, this is a living network with dynamics based on user data and peer server connections and peer server outages. Nobody seems to give a crap about the dynamics of message delivery.
Why? Why would you care so little about the data.
compare posts. Community name asklemmy@lemmy.ml server0 https://lemmy.ml/ server1 https://sh.itjust.works/ {"communityname":"asklemmy@lemmy.ml","server0":"https://lemmy.ml/","server1":"https://sh.itjust.works/"} posts timeConnect 4.347 timeParse 0.852 server https://lemmy.ml/ timeConnect 0.787 timeParse 0.258 server https://sh.itjust.works/ [ '1unfound', '2unfound', '3unfound', '4unfound', '5unfound', '6unfound', '7unfound', '8unfound', '9unfound', '12unfound', '25:17:SKIP?14?', '26:19:SKIP?17?', '28:22:SKIP?20?', 'extrasOnJ:6' ] ------------ comments of posts ============== ---- POSTS 0 [ 1468095, 372144 ] Now taking submissions for a new banner image! timeConnect 0.813 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.162 timeParse 0.071 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 11 server1 11 ---- POSTS 1 [ 1565858, 506988 ] What podcasts are you listening to? timeConnect 0.248 timeParse 0.628 server https://lemmy.ml/ timeConnect 0.257 timeParse 0.019 server https://sh.itjust.works/ commentMissing 26 unequal 0 server0 39 server1 14 ---- POSTS 2 [ 1564565, 505075 ] What if a Reddit admin is stalking Lemmy? timeConnect 0.238 timeParse 0.153 server https://lemmy.ml/ timeConnect 0.161 timeParse 0.001 server https://sh.itjust.works/ commentMissing 4 unequal 0 server0 9 server1 5 ---- POSTS 3 [ 1565001, 506191 ] What argument are you still holding onto? timeConnect 0.339 timeParse 0.002 server https://lemmy.ml/ timeConnect 0.417 timeParse 0.001 server https://sh.itjust.works/ commentMissing 6 unequal 0 server0 9 server1 3 ---- POSTS 4 [ 1561990, 505023 ] What interesting Lemmy posts have you saved that everyone should save as well? timeConnect 0.236 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.205 timeParse 0.001 server https://sh.itjust.works/ commentMissing 1 unequal 0 server0 1 server1 0 ---- POSTS 5 [ 1561692, 502885 ] [Discussion] What's your thoughts on AI generated search engines compared to other ways to find information? timeConnect 0.206 timeParse 0.722 server https://lemmy.ml/ timeConnect 0.192 timeParse 0.017 server https://sh.itjust.works/ commentMissing 6 unequal 0 server0 12 server1 6 ---- POSTS 6 [ 1561365, 502281 ] Is there a fantasy art community? timeConnect 0.294 timeParse 0.043 server https://lemmy.ml/ timeConnect 0.116 timeParse 0.001 server https://sh.itjust.works/ commentMissing 3 unequal 0 server0 8 server1 5 ---- POSTS 7 [ 1561312, 502177 ] When I see the number of upvotes and comments, are they from the whole federation or just my instance? timeConnect 0.261 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.104 timeParse 0.001 server https://sh.itjust.works/ commentMissing 3 unequal 0 server0 6 server1 4 ---- POSTS 8 [ 1559762, 500134 ] If you hade a superpower what would it be? timeConnect 0.497 timeParse 0.352 server https://lemmy.ml/ timeConnect 0.498 timeParse 0.113 server https://sh.itjust.works/ commentMissing 21 unequal 0 server0 41 server1 20 ---- POSTS 9 [ 1559585, 499867 ] How do we contact a mastodon instance from lemmy? timeConnect 0.266 timeParse 0.208 server https://lemmy.ml/ timeConnect 0.233 timeParse 0.011 server https://sh.itjust.works/ commentMissing 5 unequal 0 server0 15 server1 10 ---- POSTS 10 [ 1559186, 499010 ] Tips for using Lemmy? timeConnect 0.203 timeParse 0.025 server https://lemmy.ml/ timeConnect 0.127 timeParse 0.06 server https://sh.itjust.works/ commentMissing 8 unequal 2 server0 17 server1 9 ---- POSTS 11 [ 1559057, 506881 ] People who have their phone on vibrate and have video doorbells, how many rings do you miss? timeConnect 0.251 timeParse 0 server https://lemmy.ml/ timeConnect 0.095 timeParse 0 server https://sh.itjust.works/ commentMissing 3 unequal 0 server0 3 server1 0 ---- POSTS 12 [ 1557161, 495445 ] If I actually wanted to drink the tears of my enemies, what cocktail would they work best in? timeConnect 0.197 timeParse 0.006 server https://lemmy.ml/ timeConnect 0.207 timeParse 0.001 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 9 server1 9 ---- POSTS 13 [ 1557147, 495175 ] How do you rewrite or delete all your old reddit comments? timeConnect 0.181 timeParse 0.161 server https://lemmy.ml/ timeConnect 0.186 timeParse 0.108 server https://sh.itjust.works/ commentMissing 11 unequal 0 server0 24 server1 13 ---- POSTS 14 [ 1556987, 494761 ] I erroneously set my personal lemmy instance to private! how do i access it now?! timeConnect 0.361 timeParse 0 server https://lemmy.ml/ timeConnect 0.119 timeParse 0 server https://sh.itjust.works/ commentMissing 1 unequal 0 server0 2 server1 1 ---- POSTS 15 [ 1554510, 491568 ] What Wi-Fi networking gear you guys use? timeConnect 0.289 timeParse 0.554 server https://lemmy.ml/ timeConnect 0.421 timeParse 0.244 server https://sh.itjust.works/ commentMissing 10 unequal 0 server0 34 server1 24 ---- POSTS 16 [ 1553293, 489777 ] Did anyone else get an email about ATM Surcharge Class Action Settlement? timeConnect 0.303 timeParse 0.037 server https://lemmy.ml/ timeConnect 0.28 timeParse 0.001 server https://sh.itjust.works/ commentMissing 3 unequal 0 server0 7 server1 4 ---- POSTS 17 [ 1552624, 489194 ] Moderators, what rule violations infuriate you the most? timeConnect 0.307 timeParse 0.208 server https://lemmy.ml/ timeConnect 0.234 timeParse 0.171 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 10 server1 10 ---- POSTS 18 [ 1551349, 487134 ] Do you use adblock? Why? Why not? timeConnect 0.322 timeParse 1.347 server https://lemmy.ml/ timeConnect 0.279 timeParse 0.571 server https://sh.itjust.works/ commentMissing 58 unequal 2 server0 184 server1 126 ---- POSTS 19 [ 1550630, 486266 ] What are your top 5 favorite BAD movies of all time? timeConnect 0.265 timeParse 0.409 server https://lemmy.ml/ timeConnect 0.138 timeParse 0.208 server https://sh.itjust.works/ commentMissing 9 unequal 1 server0 49 server1 40 ---- POSTS 20 [ 1549912, 485383 ] Fast food go-ers of Lemmy, what are your "usual" orders? timeConnect 0.472 timeParse 0.263 server https://lemmy.ml/ timeConnect 0.124 timeParse 0.198 server https://sh.itjust.works/ commentMissing 16 unequal 0 server0 51 server1 36 ---- POSTS 21 [ 1549004, 483930 ] Is anyone else enjoying the slower pace of content? timeConnect 0.438 timeParse 0.599 server https://lemmy.ml/ timeConnect 0.294 timeParse 0.455 server https://sh.itjust.works/ commentMissing 29 unequal 3 server0 126 server1 97 ---- POSTS 22 [ 1548995, 483916 ] Are there any communities for lemmy beginners? timeConnect 0.228 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.161 timeParse 0 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 4 server1 4 ---- POSTS 23 [ 1548500, 483348 ] My phone's WiFi adapter keeps turning itself on but nothing that I've done on previous phones seems to be working. Any ideas as to what I can try? timeConnect 0.181 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.11 timeParse 0 server https://sh.itjust.works/ commentMissing 1 unequal 0 server0 4 server1 3 ---- POSTS 24 [ 1548501, 483349 ] are we going to have a Lemmy Party July 1st to welcome all the new users? timeConnect 0.235 timeParse 0.381 server https://lemmy.ml/ timeConnect 0.372 timeParse 0.472 server https://sh.itjust.works/ commentMissing 13 unequal 1 server0 65 server1 52 ---- POSTS 25 [ 1548340, 483227 ] Help: I'm dealing with hundreds of ripe plums timeConnect 0.273 timeParse 0.586 server https://lemmy.ml/ timeConnect 0.216 timeParse 0.474 server https://sh.itjust.works/ commentMissing 4 unequal 0 server0 53 server1 49 ---- POSTS 26 [ 1546274, 480849 ] Anyone have a good suggestion on how to block generated blog spam from google search results? timeConnect 0.244 timeParse 0 server https://lemmy.ml/ timeConnect 0.201 timeParse 0.001 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 5 server1 5 ---- POSTS 27 [ 1545196, 479173 ] Is there a blogging service that gives you hits/stats information similar to blogger.com? timeConnect 0.159 timeParse 0 server https://lemmy.ml/ timeConnect 0.096 timeParse 0 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 0 server1 0 ---- POSTS 28 [ 1543869, 477252 ] What animal or insect going extinct would have the greatest impact on the ecosystem? timeConnect 0.198 timeParse 0.057 server https://lemmy.ml/ timeConnect 0.207 timeParse 0.048 server https://sh.itjust.works/ commentMissing 1 unequal 0 server0 19 server1 18 ---- POSTS 29 [ 1543661, 476974 ] search in lemmy timeConnect 0.324 timeParse 0 server https://lemmy.ml/ timeConnect 0.228 timeParse 0 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 2 server1 2 ---- POSTS 30 [ 1543542, 476914 ] How do you negotiate with your apartment rental landlord when they are a huge corporation with 500+ properties? timeConnect 0.201 timeParse 0.391 server https://lemmy.ml/ timeConnect 0.121 timeParse 0.127 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 22 server1 22