I want Lemmy to succeed, but I’m highly skeptical of the ability of the instance operators to be able to do so. There’s a great deal of technical sophistication that is required to support a large number of users, and from what I’ve seen, they don’t have it. This isn’t a slight against them in any way, but they freely admit that they lack SQL expertise, and I think I’ve seen some significant gaps in their knowledge on how to horizontally scale. This instance, for example, is all hosted on a single virtual server. There are no load balancers, no database sharding, no fanning out of services onto different servers…security is as well also likely in a shoddy state.
Again, no hate from me, nothing but praise so far. But there are some significant technological gaps here, and I worry their team isn’t large or technically deep enough to fill them. What’s in place at the moment is just waiting to tip over when any amount of traffic starts coming over. For what it’s worth, I have offered my expertise to the admins around networking, security, scale, and automation.
For what it’s worth, I have offered my expertise to the admins around networking, security, scale, and automation.
It’s open source. That’s what’s great about it, the pro that beats out all those cons. You don’t have to offer anything to anyone, you can just start contributing.
That’s not how this works. Lemmy itself may be open source, but the instance it runs on is not. All the work in work in the world on the Lemmy codebase won’t mean anything if its actual deployment is not built for scale, and that’s not anything anyone but the admins can do anything about.
That’s not how this works. Lemmy itself may be open source, but the instance it runs on is not. All the work in work in the world on the Lemmy codebase won’t mean anything if its actual deployment is not built for scale, and that’s not anything anyone but the admins can do anything about.
That’s not how this works.
Lemmy doesn’t run on an instance. It runs on everyone’s instances. If lemmy should be deployed differently, then the first thing that would be needed is documentation and automated tools that make it easier for everyone to deploy their instances that way. You might be using lemmy.ml, but I’m not!
I’m referring specifically to Lemmy.ml, which is what the admins (of that instance) have been discussing and posting links to GitHub issues for. You can’t just take ‘everyone’s’ instance and spread it out into one giant working install of Lemmy. Every single instance that wants to handle scale is going to have to be built, managed, and maintained for it. If Lemmy.ml isn’t built to handle scale, then it’s going to go down when traffic spikes. They’re already having problems with their SQL database and traffic levels are basically nothing. You’ll end up with a bunch of users attempting to access any of the communities on Lemmy.ml and being unable to. They will need to go to a different Lemmy instance, which will have all of the same issues that Lemmy.ml will have regarding traffic load, and interact with threads there. The good thing about federation is that they’ll be able to keep using Lemmy on other instances, even if they don’t have access to Lemmy.ml specifically.
I promise I understand what I’m talking about, building for scale on a global level is what I do for a living. I also know something about open source projects, having co-founded Rocky Linux and the Rocky Enterprise Software Foundation and serving as its Director of Operations.
I promise I understand what I’m talking about, building for scale on a global level is what I do for a living. I also know something about open source projects, having co-founded Rocky Linux and the Rocky Enterprise Software Foundation and serving as its Director of Operations.
I’m not calling into question your qualifications. I do think you have misunderstood how lemmy works.
The lemmy.ml website could go dark tomorrow, completely offline, and lemmy would continue to exist and the software would continue to need maintenance and optimization. Those GitHub issues are for improvements that will help everyone, not only people using lemmy.ml specifically.
If you persuade lemmy.ml’s admins to deploy a load balancer and whatever else, that doesn’t help me. It doesn’t help anyone who’s hosting an instance that isn’t lemmy.ml, which is most of them. It’s arguable whether it even helps the admins and users of lemmy.ml itself, since half the point of federation is to not funnel users into one massive canonical instance that everyone is using. But if you write documentation or share automation tools that guide anyone on making their other federated instances more scalable, or if you contribute to lemmy’s source code to make improvements there, then that helps everyone. It improves lemmy the federated network as opposed to improving only the single inconsequential instance that is lemmy.ml.
Yes, I understand all of that. I know that it helps all the various instance owners. But that’s a problem that has already been solved. Building for scale is not specific or special to Lemmy. There are already entire automation toolsets—things like K8s or Docker Swarm, Terraform and Ansible, and endless documentation and examples on how to use and implement all of this. You’re talking about the greater whole, and what I’m trying to talk about is Lemmy.ml.
I do agree we’re probably talking past each other, though, and that’s alright, that’s how it goes on the Internet sometimes.
I am planning on taking advantage of the inherent scaling of federated services and will be playing around with various self hosted fedi instances. friends and family get access and, if I am, comfortable, I will offer it more widely in my community in micro doses. I suspect that this will be replicated again and again by others. certainly not the solution to scaling, but an important, native part of it.
There’s also a financial problem here. Money is needed to scale vertically or horizontally. Are the admins of lemmy.ml paying out of pocket currently? With current donations would they be able to scale?
I still don’t fully get how the federated thing works. Would it help with scaling and keeping costs low for lemmy.ml? I know you offload the user creation and frontend stuff to another server, but say if I use anotherlemmy.xyz to mainly interact with communities of lemmy.ml would that take a significant load off of lemmy.ml?
I want Lemmy to succeed, but I’m highly skeptical of the ability of the instance operators to be able to do so. There’s a great deal of technical sophistication that is required to support a large number of users, and from what I’ve seen, they don’t have it. This isn’t a slight against them in any way, but they freely admit that they lack SQL expertise, and I think I’ve seen some significant gaps in their knowledge on how to horizontally scale. This instance, for example, is all hosted on a single virtual server. There are no load balancers, no database sharding, no fanning out of services onto different servers…security is as well also likely in a shoddy state.
Again, no hate from me, nothing but praise so far. But there are some significant technological gaps here, and I worry their team isn’t large or technically deep enough to fill them. What’s in place at the moment is just waiting to tip over when any amount of traffic starts coming over. For what it’s worth, I have offered my expertise to the admins around networking, security, scale, and automation.
It’s open source. That’s what’s great about it, the pro that beats out all those cons. You don’t have to offer anything to anyone, you can just start contributing.
That’s not how this works. Lemmy itself may be open source, but the instance it runs on is not. All the work in work in the world on the Lemmy codebase won’t mean anything if its actual deployment is not built for scale, and that’s not anything anyone but the admins can do anything about.
That’s not how this works.
Lemmy doesn’t run on an instance. It runs on everyone’s instances. If lemmy should be deployed differently, then the first thing that would be needed is documentation and automated tools that make it easier for everyone to deploy their instances that way. You might be using lemmy.ml, but I’m not!
I’m referring specifically to Lemmy.ml, which is what the admins (of that instance) have been discussing and posting links to GitHub issues for. You can’t just take ‘everyone’s’ instance and spread it out into one giant working install of Lemmy. Every single instance that wants to handle scale is going to have to be built, managed, and maintained for it. If Lemmy.ml isn’t built to handle scale, then it’s going to go down when traffic spikes. They’re already having problems with their SQL database and traffic levels are basically nothing. You’ll end up with a bunch of users attempting to access any of the communities on Lemmy.ml and being unable to. They will need to go to a different Lemmy instance, which will have all of the same issues that Lemmy.ml will have regarding traffic load, and interact with threads there. The good thing about federation is that they’ll be able to keep using Lemmy on other instances, even if they don’t have access to Lemmy.ml specifically.
I promise I understand what I’m talking about, building for scale on a global level is what I do for a living. I also know something about open source projects, having co-founded Rocky Linux and the Rocky Enterprise Software Foundation and serving as its Director of Operations.
I’m not calling into question your qualifications. I do think you have misunderstood how lemmy works.
The lemmy.ml website could go dark tomorrow, completely offline, and lemmy would continue to exist and the software would continue to need maintenance and optimization. Those GitHub issues are for improvements that will help everyone, not only people using lemmy.ml specifically.
If you persuade lemmy.ml’s admins to deploy a load balancer and whatever else, that doesn’t help me. It doesn’t help anyone who’s hosting an instance that isn’t lemmy.ml, which is most of them. It’s arguable whether it even helps the admins and users of lemmy.ml itself, since half the point of federation is to not funnel users into one massive canonical instance that everyone is using. But if you write documentation or share automation tools that guide anyone on making their other federated instances more scalable, or if you contribute to lemmy’s source code to make improvements there, then that helps everyone. It improves lemmy the federated network as opposed to improving only the single inconsequential instance that is lemmy.ml.
Yes, I understand all of that. I know that it helps all the various instance owners. But that’s a problem that has already been solved. Building for scale is not specific or special to Lemmy. There are already entire automation toolsets—things like K8s or Docker Swarm, Terraform and Ansible, and endless documentation and examples on how to use and implement all of this. You’re talking about the greater whole, and what I’m trying to talk about is Lemmy.ml.
I do agree we’re probably talking past each other, though, and that’s alright, that’s how it goes on the Internet sometimes.
I am planning on taking advantage of the inherent scaling of federated services and will be playing around with various self hosted fedi instances. friends and family get access and, if I am, comfortable, I will offer it more widely in my community in micro doses. I suspect that this will be replicated again and again by others. certainly not the solution to scaling, but an important, native part of it.
There’s also a financial problem here. Money is needed to scale vertically or horizontally. Are the admins of lemmy.ml paying out of pocket currently? With current donations would they be able to scale?
I still don’t fully get how the federated thing works. Would it help with scaling and keeping costs low for lemmy.ml? I know you offload the user creation and frontend stuff to another server, but say if I use anotherlemmy.xyz to mainly interact with communities of lemmy.ml would that take a significant load off of lemmy.ml?