I have a domain name that I own but am not making use of and was thinking of setting up my own personal Lemmy instance, partly so I can have a Lemmy id and instance that I can completely control, partly so that I can contribute directly to my hosting cost, and partly because it might be fun to tinker with (or it might just end up being a pain; I’m still trying to figure out which is the case).
However, from this comment it sounds like, rather than contributing to horizontal scaling and easing the load on other servers, I might actually end up increasing the load on other servers by adding yet another server that the other servers have to talk to in order to keep my server updated on the latest comments and posts to which I am subscribed.
So given this, would self-hosting a personal instance actually make things worse for everyone else and thus be an irresponsible action at this time and/or for the foreseeable future? Because the last thing that I want to do is to inadvertently add a burden to the fediverse!
It’s been said here that accessing content from the large servers via a federation connection is less taxing on the servers than accessing them directly, so there’s that.
I’m thinking this would be the case if your instance had more than 1 user subscribed to that community in the large instance - but if it is just you, wouldn’t it be similar to accessing each instance and viewing the communities there?
No. Operations like removing blocked user comments, tracking your subscriptions themselves (showing only yours when you’re on teh subscription tab), and tracking your sessions are all database heavy operations. You would be trading database heavy operations with “origin needs to send activity pub messages” to you. Activitypub messages were already being generated and is simply just also sent to you… it’s very little network traffic and that’s it.
As long as your instance views most content it receives at least once… you’re well above par for doing your work to lighten the load on the origin server.
As I understand it, there are two kinds of costs that need to be considered: the cost of viewing content, and the cost of receiving content. The first is incurred every time you access your instance and is limited to your instance, whereas the latter is incurred every time something you’ve subscribed to has received an update, and is incurred not only by your instance but also by the server hosting the community. My concern is that, while hosting my own instance would reduce the load on other servers by absorbing the first kind of cost, it would also increase the load on other servers by increasing the second kind of cost.
The devs have stated that browsing load is where the main performance problems are right now. That said, it’s mainly a problem on the biggest instances like lemmy.ml. I don’t believe that sdf.org where your account is hosted is having any difficulty serving the browse load for it’s user. But if it did grow (or if I’m already wrong), then moving to a smaller instance would help them.
The devs had stated as recently as last-weekish that load from federated replication was not an issue. That said, this week I’ve seen reports from non-devs of lagging replication… so that could indicate that as the network has grown, and has the rate of writable interactions has grown, maybe replication load is becoming interesting. Another possibility is that people have poorly tuned settings and Lemmy has poorly tuned defaults. There’s a setting that controls the number of workers that perform federation replication and large instances at least must tune this to keep up with the rate of federation events they must send. it may be as simple as updating this default to fix the current wave of federation lag.
So… theoretically… the optimal balance is a medium number of medium sized instances. You’re already contributing to that balance by being on sdf.org (a medium sized instance), and I don’t think it’s worth moving your account for performance reasons. That said, if you really wanted to move your account for personal reasons, doing so would be fine. If 10k or 20k people moved to single user instances overnight, could there be problems caused by the increase in federated replication load? Quite probably yes. Lemmy will have to grow it’s capacity both in terms of handling browse load on bigger nodes… and also federation load on a bigger network. There are still unexplored approaches for scaling in both these directions and I’m not sure anyone knows which will be more serious yet except to observe that mastodon is already a much bigger ActivityPub network that’s doing alright in terms of federation load.
My advice is don’t sweat it and jistmuse your current account.
Oh interesting, I’m curious on the answer to this as well.
Hey that’s me!
If you’re excited, hop on, as have I! The protocol related scaling issue will resolve itself over time; a few of us are throwing ideas out and hoping some will stick with the developers.
But just bear in mind that you’re not adding to the network scaling because federation on the big servers will not be alleviated by you (or me) having an extra instance.
For my personally, I’m hosting an instance so that I can play with it, but also so that I can test new functionality, and help fix issues.
So I think if you are concerned about being a burden, then actively use your instance for the betterment of lemmy!
Whether the comment is true or not, I feel like part of federation is the exercise of this freedom. If Lemmy can’t support this paradigm, then the answer is that it needs to adapt and overcome these limitations, not that we need to adapt to its limitations.
Making new instances into the federation will bring this problem further into the attention of the developers and probably pressure them to improve that specific weakness in Lemmy.
At least that’s my theory.
The impact either way is minimal, so if it is something you enjoy doing go for it. Maybe it will turn into a slightly larger instance for your friends over time and then it can have a positive impact.
ActivityPub traffics are not that heavy. The activities are queued in the background and processed by a pool of dedicated workers, so no matter how long the queue, they should not bog the webserver down. Each entry in the queue does occupy some memory, but should be negligible.
I would say that an active users subscribing to a community using their own instance is actually generating less server load to the subscribed community’s instance compared to a user in that instance actively interacting with the community directly. But, if you subscribe using a personal instance but never actually use it (never logging in, never interacting with the community, never lurking in any discussion, basically forgot it exists), then yeah, you’re probably a dead weight, albeit a tiny one.
I did just that, set up for me and some friends. Working nicely so far although takes a bit of time to get enough stuff federated!
As what I understood, actually it makes things worse. This shouldn’t be the case because the concept of fediverse itself is to have a lot of many instances communicating with each other.
The problem is the Lemmy is still a young project and we weren’t expecting all this explosion in users. activityPub implementation doesn’t scale well for now and so adding a new instance theoretically makes things worse.
But this is something that devs have as a high priority, in my opinion, because is very important to have instances correctly communicating with each other, otherwise the concept of federation falls.
When I first heard of the fediverse without understanding the architecture I envisioned something like torrent networks, where the larger the network the stronger the network. After learning more I’m not sure that’s the case yet. Hopefully that is the endgame.
I would want to be in a place where I could enrich an existing community by self-hosting and synching content of that community and offering my small chunk of bandwidth to that community. I realize there is no community synching between instances, but I feel that’s where it should be to prevent corporate control of communities in the future.
You can read the ActivityPub protocol… I don’t think it will become like bittorrent, as there peer-to-peer concept doesn’t seem to exist in the spec.
Having said that, while we are running into implementation limitations on the larger instances, the problems are being tackled. A couple of us are chatting on
!lemmyperformance@lemmy.mland!lemmyfederation@lemmy.mlto see if we can come up with good ideas to present to the devs to help Lemmy scale. You’re most certainly welcome to join in on the fun!
It’s something I’ve considered as personal self-hoster. Technically, keeping my instance updated is incurring more traffic on the end server than I ever would individually.
It’s got me thinking about opening up my instance to maybe a few hundred people to make the extra load on the other servers worth their while.
If it is a single-user instance and always going to be, then yes what you have heard might be true. But again that is not a lot of load to begin with, and if your instance ends up with more users I don’t see how it can put more load on other servers than if all those users are on “other servers” directly.
That’s a good question. Every federated action (post, comment, votes) is transmitted to every subscribed server in real time. Computers are really fast, and should be able to handle thousands of federated actions every second. I don’t think we have a good idea on what number of servers are too many.
Tbh to me it just feels like we are in the early stages of a huge boom of users. I’d assume in a few months most of these growing pain issues will be gone. IF anything, maybe wait until the load balancing is settled and make your instance (but no issues if you want to help stress test the system).
Having all the servers you want content from pushing duplicate data out just for you would increase load. You as a single user on an instance that already receives that data is a significantly smaller footprint. Only people truly interested in running a public instance and all the associated work that comes with it should really spin up instances. Contributing monetarily to your favorite instance would be much more sustainable.
