ok, I had done all the simulations of network delays on 0.17.4 and setting it all up with 0.18.0rc6 was a pain with the hacked-in code I put to try and not flood the actual Lemmy peers with outbound queue.
So much time would have been saved by the whole project if 4 weeks ago lemmy.ml just started posting their logs.
There is a major problem, still in 0.18.0, if peer servers are offline. If we had lemmy.ml’s logs, the whole chain of resource problems get bunched up starting with this logging pattern:
let stats_fmt = format!(
"Activity queue stats: pending: {}, running: {}, retries: {}, dead: {}, complete: {}",
Try to keep my 4-server simulation running is a pain in the ass (brain overload), but I keep coming to the same conclusion. Damn, it is so frustrating why so much valuable production data is sitting on lemmy.ml servrer logs that reveal all this and the labor it has taken me to reproduce it on independent network.
I’m dumbfounded that nobody else sees that outbound federation is a ticking time-bomb and has already brought down several of the big servers. It’s right here, among others:
warn!(
"Sending activity {} to {} to the retry queue to be tried again later",
message.activity_id, message.inbox
);
Why keep the logs a secret when you are sending so many replication messages out? This is causing server crashes on all the big servers. Why hide these logs?
This isn’t just a programming project, this is a living network with dynamics based on user data and peer server connections and peer server outages. Nobody seems to give a crap about the dynamics of message delivery.
Why? Why would you care so little about the data.
You must log in or register to comment.
I guess some people care more about coding than the data. I’ve seen it before, but it’s hard to face. It’s like people who care so much about their car machines they don’t care about drunk driving, the navigation and safety “data” of using the car. Painful to watch this play out in slow motion and recreate it in a simulator and people just ignore the server crashes from the design.
Lemmy 0.18.1 is in release candidates on GItHub. I encourage the project to put in hacks to throttle performance features. Such as some buttons on the admin page to limit all queries to > than a certain record number on postings, as a date cut-off. Or turn off vote inserts, just no-op them in the API to the client, returning a success and put them in a text logfile if you want.
Some not-yet analyzed output of comparing postings for comment count.
compare posts. Community name asklemmy@lemmy.ml server0 https://lemmy.ml/ server1 https://sh.itjust.works/ {"communityname":"asklemmy@lemmy.ml","server0":"https://lemmy.ml/","server1":"https://sh.itjust.works/"} posts timeConnect 0.943 timeParse 0.655 server https://lemmy.ml/ timeConnect 0.838 timeParse 0.491 server https://sh.itjust.works/ [ '1unfound', '2unfound', '3unfound', '4unfound', '5unfound', '6unfound', '7unfound', '10unfound', '23:17:SKIP?14?', '24:19:SKIP?17?', '26:22:SKIP?20?', '48:45:SKIP?43?', 'extrasOnJ:3' ] ------------ comments of posts ============== --------------------- POSTING 0 [ 1468095, 372144 ] Now taking submissions for a new banner image! timeConnect 0.255 timeParse 0.201 server https://lemmy.ml/ timeConnect 0.208 timeParse 0.001 server https://sh.itjust.works/ tree0 Comment count 11 tree1 Comment count 11 commenttMissing 0 unequal 0 server0 11 server1 11 --------------------- POSTING 1 [ 1565858, 506988 ] What podcasts are you listening to? timeConnect 0.272 timeParse 0.283 server https://lemmy.ml/ timeConnect 0.207 timeParse 0.187 server https://sh.itjust.works/ tree0 Comment count 39 tree1 Comment count 14 commenttMissing 26 unequal 0 server0 39 server1 14 --------------------- POSTING 2 [ 1564565, 505075 ] What if a Reddit admin is stalking Lemmy? timeConnect 0.184 timeParse 0.006 server https://lemmy.ml/ timeConnect 0.137 timeParse 0.001 server https://sh.itjust.works/ tree0 Comment count 9 tree1 Comment count 5 commenttMissing 4 unequal 0 server0 9 server1 5 --------------------- POSTING 3 [ 1565001, 506191 ] What argument are you still holding onto? timeConnect 0.255 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.146 timeParse 0.001 server https://sh.itjust.works/ tree0 Comment count 8 tree1 Comment count 3 commenttMissing 5 unequal 0 server0 8 server1 3 --------------------- POSTING 4 [ 1561990, 505023 ] What interesting Lemmy posts have you saved that everyone should save as well? timeConnect 0.168 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.13 timeParse 0.001 server https://sh.itjust.works/ tree0 Comment count 1 tree1 Comment count 0 commenttMissing 1 unequal 0 server0 1 server1 0 --------------------- POSTING 5 [ 1561692, 502885 ] [Discussion] What's your thoughts on AI generated search engines compared to other ways to find information? timeConnect 0.271 timeParse 0.385 server https://lemmy.ml/ timeConnect 0.236 timeParse 0.005 server https://sh.itjust.works/ tree0 Comment count 11 tree1 Comment count 5 commenttMissing 6 unequal 0 server0 11 server1 5 --------------------- POSTING 6 [ 1561365, 502281 ] Is there a fantasy art community? timeConnect 0.316 timeParse 0.005 server https://lemmy.ml/ timeConnect 0.204 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 7 tree1 Comment count 5 commenttMissing 2 unequal 0 server0 7 server1 5 --------------------- POSTING 7 [ 1561312, 502177 ] When I see the number of upvotes and comments, are they from the whole federation or just my instance? timeConnect 0.271 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.106 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 6 tree1 Comment count 4 commenttMissing 3 unequal 0 server0 6 server1 4 --------------------- POSTING 8 [ 1559762, 500134 ] If you hade a superpower what would it be? timeConnect 0.29 timeParse 0.289 server https://lemmy.ml/ timeConnect 0.187 timeParse 0.127 server https://sh.itjust.works/ tree0 Comment count 41 tree1 Comment count 20 commenttMissing 21 unequal 0 server0 41 server1 20 --------------------- POSTING 9 [ 1559585, 499867 ] How do we contact a mastodon instance from lemmy? timeConnect 0.232 timeParse 0.014 server https://lemmy.ml/ timeConnect 0.12 timeParse 0.018 server https://sh.itjust.works/ tree0 Comment count 15 tree1 Comment count 10 commenttMissing 5 unequal 0 server0 15 server1 10 --------------------- POSTING 10 [ 1559186, 499010 ] Tips for using Lemmy? timeConnect 0.241 timeParse 0.005 server https://lemmy.ml/ timeConnect 0.14 timeParse 0.011 server https://sh.itjust.works/ tree0 Comment count 17 tree1 Comment count 9 commenttMissing 8 unequal 2 server0 17 server1 9 --------------------- POSTING 11 [ 1559057, 506881 ] People who have their phone on vibrate and have video doorbells, how many rings do you miss? timeConnect 0.182 timeParse 0 server https://lemmy.ml/ timeConnect 0.1 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 3 tree1 Comment count 0 commenttMissing 3 unequal 0 server0 3 server1 0 --------------------- POSTING 12 [ 1557161, 495445 ] If I actually wanted to drink the tears of my enemies, what cocktail would they work best in? timeConnect 0.276 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.226 timeParse 0.003 server https://sh.itjust.works/ tree0 Comment count 9 tree1 Comment count 9 commenttMissing 0 unequal 0 server0 9 server1 9 --------------------- POSTING 13 [ 1557147, 495175 ] How do you rewrite or delete all your old reddit comments? timeConnect 0.199 timeParse 0.164 server https://lemmy.ml/ timeConnect 0.12 timeParse 0.085 server https://sh.itjust.works/ tree0 Comment count 24 tree1 Comment count 13 commenttMissing 11 unequal 0 server0 24 server1 13 --------------------- POSTING 14 [ 1556987, 494761 ] I erroneously set my personal lemmy instance to private! how do i access it now?! timeConnect 0.224 timeParse 0 server https://lemmy.ml/ timeConnect 0.093 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 2 tree1 Comment count 1 commenttMissing 1 unequal 0 server0 2 server1 1 --------------------- POSTING 15 [ 1554510, 491568 ] What Wi-Fi networking gear you guys use? timeConnect 0.327 timeParse 0.168 server https://lemmy.ml/ timeConnect 0.194 timeParse 0.113 server https://sh.itjust.works/ tree0 Comment count 34 tree1 Comment count 24 commenttMissing 10 unequal 0 server0 34 server1 24 --------------------- POSTING 16 [ 1553293, 489777 ] Did anyone else get an email about ATM Surcharge Class Action Settlement? timeConnect 0.193 timeParse 0.023 server https://lemmy.ml/ timeConnect 0.134 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 7 tree1 Comment count 4 commenttMissing 3 unequal 0 server0 7 server1 4 --------------------- POSTING 17 [ 1552624, 489194 ] Moderators, what rule violations infuriate you the most? timeConnect 1.89 timeParse 0.112 server https://lemmy.ml/ timeConnect 0.418 timeParse 0.012 server https://sh.itjust.works/ tree0 Comment count 10 tree1 Comment count 10 commenttMissing 0 unequal 0 server0 10 server1 10 --------------------- POSTING 18 [ 1551349, 487134 ] Do you use adblock? Why? Why not? timeConnect 0.346 timeParse 1.985 server https://lemmy.ml/ timeConnect 0.227 timeParse 0.585 server https://sh.itjust.works/ tree0 Comment count 183 tree1 Comment count 126 commenttMissing 57 unequal 2 server0 183 server1 126 --------------------- POSTING 19 [ 1550630, 486266 ] What are your top 5 favorite BAD movies of all time? timeConnect 0.246 timeParse 0.405 server https://lemmy.ml/ timeConnect 0.609 timeParse 0.211 server https://sh.itjust.works/ tree0 Comment count 49 tree1 Comment count 40 commenttMissing 9 unequal 1 server0 49 server1 40 --------------------- POSTING 20 [ 1549912, 485383 ] Fast food go-ers of Lemmy, what are your "usual" orders? timeConnect 0.21 timeParse 0.396 server https://lemmy.ml/ timeConnect 0.195 timeParse 0.155 server https://sh.itjust.works/ tree0 Comment count 51 tree1 Comment count 36 commenttMissing 16 unequal 0 server0 51 server1 36 --------------------- POSTING 21 [ 1549004, 483930 ] Is anyone else enjoying the slower pace of content? timeConnect 0.214 timeParse 0.567 server https://lemmy.ml/ timeConnect 0.23 timeParse 0.447 server https://sh.itjust.works/ tree0 Comment count 126 tree1 Comment count 97 commenttMissing 29 unequal 3 server0 126 server1 97 --------------------- POSTING 22 [ 1548995, 483916 ] Are there any communities for lemmy beginners? timeConnect 0.233 timeParse 0 server https://lemmy.ml/ timeConnect 0.176 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 4 tree1 Comment count 4 commenttMissing 0 unequal 0 server0 4 server1 4 --------------------- POSTING 23 [ 1548500, 483348 ] My phone's WiFi adapter keeps turning itself on but nothing that I've done on previous phones seems to be working. Any ideas as to what I can try? timeConnect 0.209 timeParse 0 server https://lemmy.ml/ timeConnect 0.245 timeParse 0 server https://sh.itjust.works/ tree0 Comment count 4 tree1 Comment count 3 commenttMissing 1 unequal 0 server0 4 server1 3 --------------------- POSTING 24 [ 1548501, 483349 ] are we going to have a Lemmy Party July 1st to welcome all the new users? timeConnect 0.347 timeParse 0.936 server https://lemmy.ml/ timeConnect 0.562 timeParse 0.611 server https://sh.itjust.works/ tree0 Comment count 64 tree1 Comment count 52 commenttMissing 12 unequal 1 server0 64 server1 52compare posts. Community name asklemmy@lemmy.ml server0 https://lemmy.ml/ server1 https://sh.itjust.works/ {"communityname":"asklemmy@lemmy.ml","server0":"https://lemmy.ml/","server1":"https://sh.itjust.works/"} posts timeConnect 4.347 timeParse 0.852 server https://lemmy.ml/ timeConnect 0.787 timeParse 0.258 server https://sh.itjust.works/ [ '1unfound', '2unfound', '3unfound', '4unfound', '5unfound', '6unfound', '7unfound', '8unfound', '9unfound', '12unfound', '25:17:SKIP?14?', '26:19:SKIP?17?', '28:22:SKIP?20?', 'extrasOnJ:6' ] ------------ comments of posts ============== ---- POSTS 0 [ 1468095, 372144 ] Now taking submissions for a new banner image! timeConnect 0.813 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.162 timeParse 0.071 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 11 server1 11 ---- POSTS 1 [ 1565858, 506988 ] What podcasts are you listening to? timeConnect 0.248 timeParse 0.628 server https://lemmy.ml/ timeConnect 0.257 timeParse 0.019 server https://sh.itjust.works/ commentMissing 26 unequal 0 server0 39 server1 14 ---- POSTS 2 [ 1564565, 505075 ] What if a Reddit admin is stalking Lemmy? timeConnect 0.238 timeParse 0.153 server https://lemmy.ml/ timeConnect 0.161 timeParse 0.001 server https://sh.itjust.works/ commentMissing 4 unequal 0 server0 9 server1 5 ---- POSTS 3 [ 1565001, 506191 ] What argument are you still holding onto? timeConnect 0.339 timeParse 0.002 server https://lemmy.ml/ timeConnect 0.417 timeParse 0.001 server https://sh.itjust.works/ commentMissing 6 unequal 0 server0 9 server1 3 ---- POSTS 4 [ 1561990, 505023 ] What interesting Lemmy posts have you saved that everyone should save as well? timeConnect 0.236 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.205 timeParse 0.001 server https://sh.itjust.works/ commentMissing 1 unequal 0 server0 1 server1 0 ---- POSTS 5 [ 1561692, 502885 ] [Discussion] What's your thoughts on AI generated search engines compared to other ways to find information? timeConnect 0.206 timeParse 0.722 server https://lemmy.ml/ timeConnect 0.192 timeParse 0.017 server https://sh.itjust.works/ commentMissing 6 unequal 0 server0 12 server1 6 ---- POSTS 6 [ 1561365, 502281 ] Is there a fantasy art community? timeConnect 0.294 timeParse 0.043 server https://lemmy.ml/ timeConnect 0.116 timeParse 0.001 server https://sh.itjust.works/ commentMissing 3 unequal 0 server0 8 server1 5 ---- POSTS 7 [ 1561312, 502177 ] When I see the number of upvotes and comments, are they from the whole federation or just my instance? timeConnect 0.261 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.104 timeParse 0.001 server https://sh.itjust.works/ commentMissing 3 unequal 0 server0 6 server1 4 ---- POSTS 8 [ 1559762, 500134 ] If you hade a superpower what would it be? timeConnect 0.497 timeParse 0.352 server https://lemmy.ml/ timeConnect 0.498 timeParse 0.113 server https://sh.itjust.works/ commentMissing 21 unequal 0 server0 41 server1 20 ---- POSTS 9 [ 1559585, 499867 ] How do we contact a mastodon instance from lemmy? timeConnect 0.266 timeParse 0.208 server https://lemmy.ml/ timeConnect 0.233 timeParse 0.011 server https://sh.itjust.works/ commentMissing 5 unequal 0 server0 15 server1 10 ---- POSTS 10 [ 1559186, 499010 ] Tips for using Lemmy? timeConnect 0.203 timeParse 0.025 server https://lemmy.ml/ timeConnect 0.127 timeParse 0.06 server https://sh.itjust.works/ commentMissing 8 unequal 2 server0 17 server1 9 ---- POSTS 11 [ 1559057, 506881 ] People who have their phone on vibrate and have video doorbells, how many rings do you miss? timeConnect 0.251 timeParse 0 server https://lemmy.ml/ timeConnect 0.095 timeParse 0 server https://sh.itjust.works/ commentMissing 3 unequal 0 server0 3 server1 0 ---- POSTS 12 [ 1557161, 495445 ] If I actually wanted to drink the tears of my enemies, what cocktail would they work best in? timeConnect 0.197 timeParse 0.006 server https://lemmy.ml/ timeConnect 0.207 timeParse 0.001 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 9 server1 9 ---- POSTS 13 [ 1557147, 495175 ] How do you rewrite or delete all your old reddit comments? timeConnect 0.181 timeParse 0.161 server https://lemmy.ml/ timeConnect 0.186 timeParse 0.108 server https://sh.itjust.works/ commentMissing 11 unequal 0 server0 24 server1 13 ---- POSTS 14 [ 1556987, 494761 ] I erroneously set my personal lemmy instance to private! how do i access it now?! timeConnect 0.361 timeParse 0 server https://lemmy.ml/ timeConnect 0.119 timeParse 0 server https://sh.itjust.works/ commentMissing 1 unequal 0 server0 2 server1 1 ---- POSTS 15 [ 1554510, 491568 ] What Wi-Fi networking gear you guys use? timeConnect 0.289 timeParse 0.554 server https://lemmy.ml/ timeConnect 0.421 timeParse 0.244 server https://sh.itjust.works/ commentMissing 10 unequal 0 server0 34 server1 24 ---- POSTS 16 [ 1553293, 489777 ] Did anyone else get an email about ATM Surcharge Class Action Settlement? timeConnect 0.303 timeParse 0.037 server https://lemmy.ml/ timeConnect 0.28 timeParse 0.001 server https://sh.itjust.works/ commentMissing 3 unequal 0 server0 7 server1 4 ---- POSTS 17 [ 1552624, 489194 ] Moderators, what rule violations infuriate you the most? timeConnect 0.307 timeParse 0.208 server https://lemmy.ml/ timeConnect 0.234 timeParse 0.171 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 10 server1 10 ---- POSTS 18 [ 1551349, 487134 ] Do you use adblock? Why? Why not? timeConnect 0.322 timeParse 1.347 server https://lemmy.ml/ timeConnect 0.279 timeParse 0.571 server https://sh.itjust.works/ commentMissing 58 unequal 2 server0 184 server1 126 ---- POSTS 19 [ 1550630, 486266 ] What are your top 5 favorite BAD movies of all time? timeConnect 0.265 timeParse 0.409 server https://lemmy.ml/ timeConnect 0.138 timeParse 0.208 server https://sh.itjust.works/ commentMissing 9 unequal 1 server0 49 server1 40 ---- POSTS 20 [ 1549912, 485383 ] Fast food go-ers of Lemmy, what are your "usual" orders? timeConnect 0.472 timeParse 0.263 server https://lemmy.ml/ timeConnect 0.124 timeParse 0.198 server https://sh.itjust.works/ commentMissing 16 unequal 0 server0 51 server1 36 ---- POSTS 21 [ 1549004, 483930 ] Is anyone else enjoying the slower pace of content? timeConnect 0.438 timeParse 0.599 server https://lemmy.ml/ timeConnect 0.294 timeParse 0.455 server https://sh.itjust.works/ commentMissing 29 unequal 3 server0 126 server1 97 ---- POSTS 22 [ 1548995, 483916 ] Are there any communities for lemmy beginners? timeConnect 0.228 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.161 timeParse 0 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 4 server1 4 ---- POSTS 23 [ 1548500, 483348 ] My phone's WiFi adapter keeps turning itself on but nothing that I've done on previous phones seems to be working. Any ideas as to what I can try? timeConnect 0.181 timeParse 0.001 server https://lemmy.ml/ timeConnect 0.11 timeParse 0 server https://sh.itjust.works/ commentMissing 1 unequal 0 server0 4 server1 3 ---- POSTS 24 [ 1548501, 483349 ] are we going to have a Lemmy Party July 1st to welcome all the new users? timeConnect 0.235 timeParse 0.381 server https://lemmy.ml/ timeConnect 0.372 timeParse 0.472 server https://sh.itjust.works/ commentMissing 13 unequal 1 server0 65 server1 52 ---- POSTS 25 [ 1548340, 483227 ] Help: I'm dealing with hundreds of ripe plums timeConnect 0.273 timeParse 0.586 server https://lemmy.ml/ timeConnect 0.216 timeParse 0.474 server https://sh.itjust.works/ commentMissing 4 unequal 0 server0 53 server1 49 ---- POSTS 26 [ 1546274, 480849 ] Anyone have a good suggestion on how to block generated blog spam from google search results? timeConnect 0.244 timeParse 0 server https://lemmy.ml/ timeConnect 0.201 timeParse 0.001 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 5 server1 5 ---- POSTS 27 [ 1545196, 479173 ] Is there a blogging service that gives you hits/stats information similar to blogger.com? timeConnect 0.159 timeParse 0 server https://lemmy.ml/ timeConnect 0.096 timeParse 0 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 0 server1 0 ---- POSTS 28 [ 1543869, 477252 ] What animal or insect going extinct would have the greatest impact on the ecosystem? timeConnect 0.198 timeParse 0.057 server https://lemmy.ml/ timeConnect 0.207 timeParse 0.048 server https://sh.itjust.works/ commentMissing 1 unequal 0 server0 19 server1 18 ---- POSTS 29 [ 1543661, 476974 ] search in lemmy timeConnect 0.324 timeParse 0 server https://lemmy.ml/ timeConnect 0.228 timeParse 0 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 2 server1 2 ---- POSTS 30 [ 1543542, 476914 ] How do you negotiate with your apartment rental landlord when they are a huge corporation with 500+ properties? timeConnect 0.201 timeParse 0.391 server https://lemmy.ml/ timeConnect 0.121 timeParse 0.127 server https://sh.itjust.works/ commentMissing 0 unequal 0 server0 22 server1 22commentMissing 0 unequal 0 server0 11 server1 11 https://lemmy.ml/post/1468095 https://sh.itjust.works/post/372144 commentMissing 33 unequal 1 server0 46 server1 46 https://lemmy.ml/post/1565858 https://sh.itjust.works/post/506988 commentMissing 4 unequal 0 server0 9 server1 9 https://lemmy.ml/post/1564565 https://sh.itjust.works/post/505075 commentMissing 6 unequal 0 server0 9 server1 9 https://lemmy.ml/post/1565001 https://sh.itjust.works/post/506191 commentMissing 1 unequal 0 server0 1 server1 1 https://lemmy.ml/post/1561990 https://sh.itjust.works/post/505023 commentMissing 6 unequal 0 server0 12 server1 12 https://lemmy.ml/post/1561692 https://sh.itjust.works/post/502885 commentMissing 3 unequal 0 server0 8 server1 8 https://lemmy.ml/post/1561365 https://sh.itjust.works/post/502281 commentMissing 3 unequal 0 server0 6 server1 6 https://lemmy.ml/post/1561312 https://sh.itjust.works/post/502177 commentMissing 22 unequal 0 server0 42 server1 42 https://lemmy.ml/post/1559762 https://sh.itjust.works/post/500134 commentMissing 5 unequal 0 server0 15 server1 15 https://lemmy.ml/post/1559585 https://sh.itjust.works/post/499867 commentMissing 8 unequal 2 server0 17 server1 17 https://lemmy.ml/post/1559186 https://sh.itjust.works/post/499010 commentMissing 3 unequal 0 server0 3 server1 3 https://lemmy.ml/post/1559057 https://sh.itjust.works/post/506881 commentMissing 0 unequal 0 server0 10 server1 10 https://lemmy.ml/post/1557161 https://sh.itjust.works/post/495445 commentMissing 11 unequal 0 server0 24 server1 24 https://lemmy.ml/post/1557147 https://sh.itjust.works/post/495175 commentMissing 1 unequal 0 server0 2 server1 2 https://lemmy.ml/post/1556987 https://sh.itjust.works/post/494761 commentMissing 10 unequal 0 server0 34 server1 34 https://lemmy.ml/post/1554510 https://sh.itjust.works/post/491568 commentMissing 3 unequal 0 server0 7 server1 7 https://lemmy.ml/post/1553293 https://sh.itjust.works/post/489777 commentMissing 0 unequal 0 server0 10 server1 10 https://lemmy.ml/post/1552624 https://sh.itjust.works/post/489194 commentMissing 58 unequal 2 server0 184 server1 184 https://lemmy.ml/post/1551349 https://sh.itjust.works/post/487134 commentMissing 10 unequal 2 server0 50 server1 50 https://lemmy.ml/post/1550630 https://sh.itjust.works/post/486266 commentMissing 16 unequal 0 server0 51 server1 51 https://lemmy.ml/post/1549912 https://sh.itjust.works/post/485383 commentMissing 29 unequal 3 server0 126 server1 126 https://lemmy.ml/post/1549004 https://sh.itjust.works/post/483930 commentMissing 0 unequal 0 server0 4 server1 4 https://lemmy.ml/post/1548995 https://sh.itjust.works/post/483916 commentMissing 1 unequal 0 server0 4 server1 4 https://lemmy.ml/post/1548500 https://sh.itjust.works/post/483348 commentMissing 14 unequal 1 server0 66 server1 66 https://lemmy.ml/post/1548501 https://sh.itjust.works/post/483349 commentMissing 4 unequal 0 server0 53 server1 53 https://lemmy.ml/post/1548340 https://sh.itjust.works/post/483227 commentMissing 0 unequal 0 server0 5 server1 5 https://lemmy.ml/post/1546274 https://sh.itjust.works/post/480849 commentMissing 0 unequal 0 server0 0 server1 0 https://lemmy.ml/post/1545196 https://sh.itjust.works/post/479173 commentMissing 1 unequal 0 server0 19 server1 19 https://lemmy.ml/post/1543869 https://sh.itjust.works/post/477252 commentMissing 0 unequal 0 server0 2 server1 2 https://lemmy.ml/post/1543661 https://sh.itjust.works/post/476974 commentMissing 0 unequal 0 server0 22 server1 22 https://lemmy.ml/post/1543542 https://sh.itjust.works/post/476914 commentMissing 0 unequal 0 server0 16 server1 16 https://lemmy.ml/post/1543306 https://sh.itjust.works/post/476771 commentMissing 0 unequal 0 server0 7 server1 7 https://lemmy.ml/post/1543053 https://sh.itjust.works/post/476767 commentMissing 3 unequal 0 server0 27 server1 27 https://lemmy.ml/post/1542279 https://sh.itjust.works/post/475482 commentMissing 2 unequal 0 server0 32 server1 32 https://lemmy.ml/post/1542907 https://sh.itjust.works/post/476659 commentMissing 3 unequal 0 server0 21 server1 21 https://lemmy.ml/post/1542224 https://sh.itjust.works/post/475374 commentMissing 3 unequal 1 server0 16 server1 16 https://lemmy.ml/post/1540881 https://sh.itjust.works/post/473290 commentMissing 3 unequal 1 server0 47 server1 47 https://lemmy.ml/post/1540571 https://sh.itjust.works/post/472776 commentMissing 1 unequal 0 server0 24 server1 24 https://lemmy.ml/post/1541171 https://sh.itjust.works/post/473545deleted by creator
missing 0 unequal 0 11 on https://lemmy.ml/ vs. 11 on https://sh.itjust.works/
missing 35 unequal 1 48 on https://lemmy.ml/ vs. 14 on https://sh.itjust.works/
missing 4 unequal 0 9 on https://lemmy.ml/ vs. 5 on https://sh.itjust.works/
missing 6 unequal 0 9 on https://lemmy.ml/ vs. 3 on https://sh.itjust.works/
missing 1 unequal 0 1 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 6 unequal 0 12 on https://lemmy.ml/ vs. 6 on https://sh.itjust.works/
missing 3 unequal 0 8 on https://lemmy.ml/ vs. 5 on https://sh.itjust.works/
missing 3 unequal 0 6 on https://lemmy.ml/ vs. 4 on https://sh.itjust.works/
missing 22 unequal 0 42 on https://lemmy.ml/ vs. 20 on https://sh.itjust.works/
missing 5 unequal 0 15 on https://lemmy.ml/ vs. 10 on https://sh.itjust.works/
missing 8 unequal 2 17 on https://lemmy.ml/ vs. 9 on https://sh.itjust.works/
missing 3 unequal 0 3 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 0 unequal 0 10 on https://lemmy.ml/ vs. 10 on https://sh.itjust.works/
missing 11 unequal 0 24 on https://lemmy.ml/ vs. 13 on https://sh.itjust.works/
missing 1 unequal 0 2 on https://lemmy.ml/ vs. 1 on https://sh.itjust.works/
missing 13 unequal 0 37 on https://lemmy.ml/ vs. 24 on https://sh.itjust.works/
missing 3 unequal 0 7 on https://lemmy.ml/ vs. 4 on https://sh.itjust.works/
missing 0 unequal 0 10 on https://lemmy.ml/ vs. 10 on https://sh.itjust.works/
missing 60 unequal 2 186 on https://lemmy.ml/ vs. 126 on https://sh.itjust.works/
missing 10 unequal 2 51 on https://lemmy.ml/ vs. 41 on https://sh.itjust.works/
missing 16 unequal 0 51 on https://lemmy.ml/ vs. 36 on https://sh.itjust.works/
missing 31 unequal 3 128 on https://lemmy.ml/ vs. 97 on https://sh.itjust.works/
missing 0 unequal 0 4 on https://lemmy.ml/ vs. 4 on https://sh.itjust.works/
missing 2 unequal 0 5 on https://lemmy.ml/ vs. 3 on https://sh.itjust.works/
missing 15 unequal 1 67 on https://lemmy.ml/ vs. 52 on https://sh.itjust.works/
missing 4 unequal 0 53 on https://lemmy.ml/ vs. 49 on https://sh.itjust.works/
missing 0 unequal 0 5 on https://lemmy.ml/ vs. 5 on https://sh.itjust.works/
missing 0 unequal 0 0 on https://lemmy.ml/ vs. 0 on https://sh.itjust.works/
missing 1 unequal 0 19 on https://lemmy.ml/ vs. 18 on https://sh.itjust.works/
missing 0 unequal 0 2 on https://lemmy.ml/ vs. 2 on https://sh.itjust.works/
missing 0 unequal 0 22 on https://lemmy.ml/ vs. 22 on https://sh.itjust.works/
missing 0 unequal 0 16 on https://lemmy.ml/ vs. 18 on https://sh.itjust.works/
missing 0 unequal 0 7 on https://lemmy.ml/ vs. 7 on https://sh.itjust.works/
missing 3 unequal 0 27 on https://lemmy.ml/ vs. 24 on https://sh.itjust.works/
missing 2 unequal 0 32 on https://lemmy.ml/ vs. 30 on https://sh.itjust.works/
missing 3 unequal 0 21 on https://lemmy.ml/ vs. 18 on https://sh.itjust.works/
missing 3 unequal 1 16 on https://lemmy.ml/ vs. 13 on https://sh.itjust.works/
missing 3 unequal 1 47 on https://lemmy.ml/ vs. 44 on https://sh.itjust.works/
missing 1 unequal 0 24 on https://lemmy.ml/ vs. 23 on https://sh.itjust.works/