ok, I had done all the simulations of network delays on 0.17.4 and setting it all up with 0.18.0rc6 was a pain with the hacked-in code I put to try and not flood the actual Lemmy peers with outbound queue.

So much time would have been saved by the whole project if 4 weeks ago lemmy.ml just started posting their logs.

There is a major problem, still in 0.18.0, if peer servers are offline. If we had lemmy.ml’s logs, the whole chain of resource problems get bunched up starting with this logging pattern:

https://github.com/LemmyNet/activitypub-federation-rust/blob/325f66ba324037a4f1d330a0dbea6e062ba34f50/src/activity_queue.rs#L117

let stats_fmt = format!(
            "Activity queue stats: pending: {}, running: {}, retries: {}, dead: {}, complete: {}",

Try to keep my 4-server simulation running is a pain in the ass (brain overload), but I keep coming to the same conclusion. Damn, it is so frustrating why so much valuable production data is sitting on lemmy.ml servrer logs that reveal all this and the labor it has taken me to reproduce it on independent network.

I’m dumbfounded that nobody else sees that outbound federation is a ticking time-bomb and has already brought down several of the big servers. It’s right here, among others:

            warn!(
                "Sending activity {} to {} to the retry queue to be tried again later",
                message.activity_id, message.inbox
            );

Why keep the logs a secret when you are sending so many replication messages out? This is causing server crashes on all the big servers. Why hide these logs?

This isn’t just a programming project, this is a living network with dynamics based on user data and peer server connections and peer server outages. Nobody seems to give a crap about the dynamics of message delivery.

Why? Why would you care so little about the data.