FYI: even Lemmy servers upgraded to 0.18.0 are having problems replicating comments to each other. Missing comments on posts examples

RoundSparrow · 1 year ago

FYI: even Lemmy servers upgraded to 0.18.0 are having problems replicating comments to each other. Missing comments on posts examples

DaEagle · 1 year ago

I wonder if some instances are more prone to drop comments that others? have you looked for example at lemmy.world (very large but performant) and another relatively small instance?

RoundSparrow · 1 year ago

have you looked for example at lemmy.world (very large but performant) and another relatively small instance?

Yes, and I run my own instance that is on a high performance server and it has no users other than myself for testing Lemmy.

Lemmy.ml is having the most problems, but really all of the big servers are dropping comment and post replication. The biggest problem is that server operators have no way to know this is happening other than to read raw Linux error logs. The post ID numbers are unique to each server, so it is not easy to identify which link is for the same postings.

DaEagle · 1 year ago

I think I saw one of your earlier posts and I really appreciate you chasing this down and raising awareness. As a relatively savvy user this is definitely something I’ve noticed and I share your concern that it will slowly erode user’s trust in the concept of federation.

Technically, can you trace where the comments are dropped? does the target receive the response but fails to process it, or does it break somewhere at the network layer? if so, is there no receiver “ack” built into the protocol? sorry for asking a bunch of questions and feel free to ignore (I’m an engineer but I don’t know much about the federation protocol…)

RoundSparrow · 1 year ago

Technically, can you trace where the comments are dropped?

There are multiple timing and resource issues with the way content is sent. Every single vote and comment has a lot of overhead, and the Lemmy servers are causing each other to slow down with the overhead of it all. There are even very tight security timings that have been hit causing rejection. And there system has no automated way to repair missing content, it just tires to keep up with each new posting, comment, vote.

maegul (he/they) · 1 year ago

Could you do a little bit more analysis to determine the typical failure mode?

For instance, my first guess would be that information is more likely to get lost when a comment/vote needs to go through multiple servers.

That is, a thread is on a community on server A, with people commenting from servers A, B and C. AFAIU, all data synchronisation goes through server A, as that’s where the community lives. So a comment from server B, even it’s in response to a comment/post from server C needs to go through server A and then to server C.

If these multi-server trips get dropped the most, you’d expect views of the thread to be most inconsistent between servers B and C and that the most dropped content to be from servers B or C when viewed from the other of these two.

RoundSparrow · 1 year ago

The failures have more to do with server performance related to the quantity of new comments and activities than anything. There are periods of time that it fails worse than others and even when web browsers visiting the servers show errors.

maegul (he/they) · 1 year ago

How much of this is just the nature of activity pub and federation over it?

Putting aside whether lemmy is doing a good or bad job, it seems like an issue at the protocol level? For instance, if lemmy were to implement some additional procedures as you hint at, would they work for federation outside of lemmy or would they even cause problems or bugs?

I’m pretty sure I’ve seen things get dropped in similar ways on other micro-blog platforms. It could be that the community/group based structure just surfaces the issues more because you can compare whole communities instead of individual reply threads in microblogging. Maybe activity pub isn’t appropriate for federated group activity?

Have you looked at similar issues with kbin?

Also, thanks for this work. High level end-to-end testing like this is probably invaluable!!

RoundSparrow · 1 year ago

I haven’t had time to look at kbin.

The Lemmy servers are logging errors, but the messages need organization and are rather difficult to access by server operators. Growing pains.

haganbmj · 1 year ago

Just to pose these in a similar thread, I have a few questions as a casual observer, some of which I’m unclear if they’re handled at the protocol or Lemmy level.

As I understand it servers subscribe to other servers and everything is then push based?
I assume ordering is not a guarantee. So there’s probably no concept of offset tracking on subscriptions or replaying a time range?
If ordering is not a requirement how do likes/comments handle out of order receipt? Everything seems to have a local ID, so can content get pre-liked before the root message arrives? Unclear if ID generation is based on any identifiers you’d have to work with or not - or whether remote content retains its origin IDs?
Lemmy at least appears to have some retry mechanism, but I’m unclear the behavior on that - seems annoying with 1000+ subscribing servers.
I seem to recall reading ActivityPub has some pattern for batching, but reading the spec again I’m not seeing it. Is that a thing?

maegul (he/they) · 1 year ago

Sorry I don’t know the answers to these questions. Also, I don’t think the OP will get a notification for your content (?) unless you reply to them directly, just in case you want to ask them.

RoundSparrow · 1 year ago

is there no receiver “ack” built into the protocol?

That’s one of the things that bothers me the most, even email systems in 1993 that I worked on - which far slower hardware and matching the volume Lemmy is doing today - would at least send a delivery failure report back to the sender. Here on Lemmy, you can reply to someone on another server, but you do not get notified if the delivery fails.

Gompje@lemmy.world · 1 year ago

Agreed. Today I realized that not only do I not see the same posts/comments in the same community with different accounts (world & sh.it.works) but even in my own posts the comments/votes are vastly different. In one instance even a reply comment from someone was linked to another comment! I know things are new and all but… this kinda worries me… a lot in fact. Yes today world had major issues but still… I mean the “silo” community on there looks completely different with a world account than with the other one.

RoundSparrow · edit-2 1 year ago

Here is a fresh report on Lemmy.world vs. feddit.nl: