It’s dumb that someone needs to interact with community in order for it to federate with your instance. Like, how are you supposed to find it in the first place? It makes it difficult for communities to grow on instances that are not mainstream which makes decentralization useless.
In order to avoid this restriction you would need a global instance discovery mechanism, which is extremely hard to implement without a central server that keeps a list of all instances in the network. And if you do implement instance discovery through a central server you really are losing the whole point of decentralization.
Additionally, it’s good that each instance does not federate with everyone else by default. If it did, it would have to process all activity and keep a local copy of all the content in the entire network. This would be insanely inefficient, and make it prohibitively expensive to run even a tiny instance with 1 user and no communities.
Decentralization isn’t useless if you can’t immediately see everything in the network, come on… We’re just spoiled by centralized services.
Ummmm… All you need is some bots on each instance that automatically will interact once with communities known on lemmyverse.net and boom, you have unlocked full federation for every user on those instances. You are not losing any privacy through that, it just skips the steps where user has to manually index a sublemmy before it federates and makes platform more usable.
Sure, but now this system has a dependency on the “centralized” lemmyverse.net service. And also your instance now has to receive and store a copy of almost the entire network’s content. Lots of instances are already struggling to sustain the load, this would make the problem even worse.
If a single instance decides that it can sustain the increased load and doesn’t mind depending on lemmyverse.net sure, nothing’s stopping them. But it shouldn’t be the default behavior for all instances.
If load is that big of a problem then pretty much all instances are in a big trouble because all you need is ONE person that decides to run some bots and it’s done anyway.
Well yeah, this is a fair point. It’s on the instance admins to vet signups and monitor usage. In general the fediverse has a lot of potential for easy DOS attacks, it’s one of the main weak points.
Yet bittorrent dht is 20 years old. How can this supposedly decentralized service be unable to self organize. Is Lemmy some kitchen napkin high school fair project ?
Social media and torrents are pretty damn different. There’s a reason no federated platform has implemented automatic discovery, even ones with much more resources than Lemmy, like Mastodon.
I don’t know why you folks keep pointing at missing features and saying “Lemmy doesn’t have this pretty advanced network feature, so it’s not really decentralized”, or “it cannot organize”, or “it’s useless”… It’s basically two people’s passion project that only blew up in the past month because reddit fucked up. You’re not paying for it, are you? So I really don’t see how this attitude is warranted.
The tech is right there, it’s 20 years old. I’m pointing at it in response to people saying “this is too hard, we can’t have 700 instances sharing a few kilobytes of text !, You’re asking too much”
Look, if you understand what a DHT is I also expect you to understand the amount of effort it would take to implement such a feature. The fact that the tech exists does not mean you can just plug it in and go. It took the devs weeks to move from websockets to HTTP, a feature like this would take months and take time away from a lot of other important work.
The pseudocode for this feature is as follows
Cron job to be run at regular intervals (say top of the hour)
Query database for all content (posts, comments) between last packet and now, place in packet.
Query database for all moderation actions ( votes, censure, blocking, pinning, etc ) between last packet and now, place in packet.
Take packet, name with name on instance plus date range.
Export as dht enable magnet link torrent
Forward magnet url to all known federated instances
That’s certainly an option. Probably pretty useful too. Though I think it would put quite a load on server resources to have every community in the threadiverse cached though.
I don’t know enough about how the instances interact on the activitypub protocol to comment on that. I see that Mastodon has over 11,000 instances running, and haven’t heard of instabilities, but they also are fundamentally different applications so idk how comparable they are.
How is that not part of base code. Lemmy is completely unusable until this is fixed. The clock is ticking on Reddit’s implosion. If this isn’t fixed, the Reddit userbase will go back to Reddit for another 20 years.
Please don’t let Lemmy as useless as Mastodon, this is clearly design sabotage by silicon Valley big tech.
Calm down bro. Your negative energy is harshing the vibe here. Let’s try that again
I see, thanks for the info. The devs should prioritize adding a feature like that into the base code. It would really help reddit migrants get set up here.
See? I liked a couple of your comments in the other thread about downvotes being public information, but you’re sounding pretty defeatist and frankly, miserable in the past few hours. If you don’t want the reddit userbase to go back, the best thing you can do is be a user that other users enjoy interacting with.
Unless you wanna step in and start coding this thing 🤣
you would need a global instance discovery mechanism
I don’t think you do. Instances should merely reach out to other instances it’s federated with periodically to get a list of communities and some of their metadata. Ideally, they could ask all of those other instances to notify it when a community is added, modified, or deleted, and then store that metadata.
That should be pretty easy to implement, and maybe it already has, idk.
Sure, but this isn’t finding new instances, just new communities on known instances. Indeed, this is not difficult to implement. The reason it’s not done already is for resource economy. A lot of instances are already struggling to scale, making them process and store a lot more content with little value for most users of the instance isn’t feasible for a lot of servers right now.
A list of communities isn’t “a lot more content.” Just run it once daily and the problem is solved. Instances don’t need to store posts, just community names available on that instance.
If the issue is finding new instances, I think it’s fine for it to take some time the first time someone tries to find a community on that instance. But after that, it should immediately have a list of communities from that instance.
I just did some math and assumed there are 700 (some instances are blocking other instances) instances and 12 000 communities. 700*12 000 = 8 400 000, users across the platform need to copy url of community and paste it into search this many times to make the platform fully federate with everything. Numbers were taken from here: https://lemmyverse.net/
Sorry! My bad, numbers are incorrrect but I’m not sure how to calculate correct numbers lol. Wouldn’t it be 700x699 or 699x699 because instance wouldn’t have to federate with itself?
a continue inside an if statement checking if the bot domain is the same as the subscribing one would suffice
edit: or use a search algorithm to pop the domain from the stack before starting the loop, that would be more efficient
edit2: even better: assuming all bot names are the same, you just iterate over the stack (constant), and pop each domain (new stack minus popped domain) and feed it to the 2nd loop.
edit3: SCRATCH EVERYTHING. The bot names don’t even have to be the same, you just iterate over a constant stack of botname@domain entries and pop the iteration from it… feed the new stack to the second loop (nested) and done. Why did I take so long to reach this?
This would increase server load and traffic by a huge amount though, and would the benefit really be that big?
There are already quite good places to find communities for stuff you’re interested in, and less niche stuff will be federated “automatically” by these rules unless you’re on a small instance.
And I’m not even taking into account platforms that are federating with lemmy, this is amount of actions needed for lemmy to simply federate with itself.
I don’t think full federation is required. I wouldn’t want Roblox and Minecraft communities to have porn visible to the users that sign up on those instances.
Dude… If I make an account on instance that has minecraft community on it and go into the fediverse and subscribe to porn sublemmy then it will show up anyway… If you don’t want to see porn then turn off NSFW setting and if some instance doesn’t tag porn as NSFW report it to your admins and defederate from them if they refuse to resolve the issue.
this number is only completely relevant for someone on an instance all by themselves or with no communities at all. And discounts instances that are or will de-federate either partially or fully. It also assumes some need to be a part of all 12,000 communities. I think tools like you linked solve this issue anyway. I personally believe to a certain extent every community being federated to every instance kind of defeats the purpose of federalization.
But that’s not my point though? The point is to federate with all sublemmies of FEDERATED instances. The problem is that when you federate other instance it doesn’t federate sublemmies of that instance automatically which limits interactions between instances by a HUGE amount
It’s a risk reward question then. That would 100% slowdown the initial federation if it needed to pull in every community and even if that was accepted, should every instance constantly poll any instance it knows about for new communities? Also you aren’t guaranteed to need all those instances anyway and then that’s just a waste of space and processing power. Correct, it limits interactions but only to what’s necessary which allows instances to be ran on lower powered hardware, allowing more people to join in. With the possibility of third party tools I don’t see much of an upside of building that into lemmy.
It would be cool if you could see stuff from other servers/instances without someone from your server/instance interacting with it first, yes.
At the moment I am personally not complaining though. I am sure the developers have tons of things they’re trying to work on at the moment. And I can still see stuff from other instances/servers.
I guess the only way to make things work will be to setup bot accounts that will exist on each existing instance and automatically interact with every community from other instances. With bot account it will be easier to do 10 000 000 actions when it’s impossible to do manually.
It’s dumb that someone needs to interact with community in order for it to federate with your instance. Like, how are you supposed to find it in the first place? It makes it difficult for communities to grow on instances that are not mainstream which makes decentralization useless.
In order to avoid this restriction you would need a global instance discovery mechanism, which is extremely hard to implement without a central server that keeps a list of all instances in the network. And if you do implement instance discovery through a central server you really are losing the whole point of decentralization.
Additionally, it’s good that each instance does not federate with everyone else by default. If it did, it would have to process all activity and keep a local copy of all the content in the entire network. This would be insanely inefficient, and make it prohibitively expensive to run even a tiny instance with 1 user and no communities.
Decentralization isn’t useless if you can’t immediately see everything in the network, come on… We’re just spoiled by centralized services.
Ummmm… All you need is some bots on each instance that automatically will interact once with communities known on lemmyverse.net and boom, you have unlocked full federation for every user on those instances. You are not losing any privacy through that, it just skips the steps where user has to manually index a sublemmy before it federates and makes platform more usable.
Sure, but now this system has a dependency on the “centralized” lemmyverse.net service. And also your instance now has to receive and store a copy of almost the entire network’s content. Lots of instances are already struggling to sustain the load, this would make the problem even worse.
If a single instance decides that it can sustain the increased load and doesn’t mind depending on lemmyverse.net sure, nothing’s stopping them. But it shouldn’t be the default behavior for all instances.
If load is that big of a problem then pretty much all instances are in a big trouble because all you need is ONE person that decides to run some bots and it’s done anyway.
Well yeah, this is a fair point. It’s on the instance admins to vet signups and monitor usage. In general the fediverse has a lot of potential for easy DOS attacks, it’s one of the main weak points.
I will send you my post on how we got 550 000 bot accounts in 3 days lol
https://lemmy.fmhy.ml/post/176547
Oh right, forgot to mention it, lemmyverse.net is open source so there’s no problem with centralisation because you can run it yourself 👍
Yet bittorrent dht is 20 years old. How can this supposedly decentralized service be unable to self organize. Is Lemmy some kitchen napkin high school fair project ?
Social media and torrents are pretty damn different. There’s a reason no federated platform has implemented automatic discovery, even ones with much more resources than Lemmy, like Mastodon.
I don’t know why you folks keep pointing at missing features and saying “Lemmy doesn’t have this pretty advanced network feature, so it’s not really decentralized”, or “it cannot organize”, or “it’s useless”… It’s basically two people’s passion project that only blew up in the past month because reddit fucked up. You’re not paying for it, are you? So I really don’t see how this attitude is warranted.
The tech is right there, it’s 20 years old. I’m pointing at it in response to people saying “this is too hard, we can’t have 700 instances sharing a few kilobytes of text !, You’re asking too much”
Look, if you understand what a DHT is I also expect you to understand the amount of effort it would take to implement such a feature. The fact that the tech exists does not mean you can just plug it in and go. It took the devs weeks to move from websockets to HTTP, a feature like this would take months and take time away from a lot of other important work.
The pseudocode for this feature is as follows Cron job to be run at regular intervals (say top of the hour) Query database for all content (posts, comments) between last packet and now, place in packet. Query database for all moderation actions ( votes, censure, blocking, pinning, etc ) between last packet and now, place in packet. Take packet, name with name on instance plus date range. Export as dht enable magnet link torrent Forward magnet url to all known federated instances
That’s certainly an option. Probably pretty useful too. Though I think it would put quite a load on server resources to have every community in the threadiverse cached though.
Yeah, from what I see the whole thing is not scalable and will need some changes in how this all works… :/
I don’t know enough about how the instances interact on the activitypub protocol to comment on that. I see that Mastodon has over 11,000 instances running, and haven’t heard of instabilities, but they also are fundamentally different applications so idk how comparable they are.
How is that not part of base code. Lemmy is completely unusable until this is fixed. The clock is ticking on Reddit’s implosion. If this isn’t fixed, the Reddit userbase will go back to Reddit for another 20 years. Please don’t let Lemmy as useless as Mastodon, this is clearly design sabotage by silicon Valley big tech.
Calm down bro. Your negative energy is harshing the vibe here. Let’s try that again
See? I liked a couple of your comments in the other thread about downvotes being public information, but you’re sounding pretty defeatist and frankly, miserable in the past few hours. If you don’t want the reddit userbase to go back, the best thing you can do is be a user that other users enjoy interacting with.
Unless you wanna step in and start coding this thing 🤣
deleted by creator
I don’t think you do. Instances should merely reach out to other instances it’s federated with periodically to get a list of communities and some of their metadata. Ideally, they could ask all of those other instances to notify it when a community is added, modified, or deleted, and then store that metadata.
That should be pretty easy to implement, and maybe it already has, idk.
Sure, but this isn’t finding new instances, just new communities on known instances. Indeed, this is not difficult to implement. The reason it’s not done already is for resource economy. A lot of instances are already struggling to scale, making them process and store a lot more content with little value for most users of the instance isn’t feasible for a lot of servers right now.
A list of communities isn’t “a lot more content.” Just run it once daily and the problem is solved. Instances don’t need to store posts, just community names available on that instance.
If the issue is finding new instances, I think it’s fine for it to take some time the first time someone tries to find a community on that instance. But after that, it should immediately have a list of communities from that instance.
I just did some math and assumed there are 700 (some instances are blocking other instances) instances and 12 000 communities. 700*12 000 = 8 400 000, users across the platform need to copy url of community and paste it into search this many times to make the platform fully federate with everything. Numbers were taken from here: https://lemmyverse.net/
wait, so if you federate with another instance through one community, you won’t get to see the rest of the feeds from that instance?
would subscribing to the domain directly (like kbin allows, maybe lemmy could in the future) reduce the number of actions to 700^2?
Sorry! My bad, numbers are incorrrect but I’m not sure how to calculate correct numbers lol. Wouldn’t it be 700x699 or 699x699 because instance wouldn’t have to federate with itself?
oh yeah, it is
n * (n - 1)bc ~700 bots need to do ~700 minus 1 actions
a
continueinside anifstatement checking if the bot domain is the same as the subscribing one would sufficeedit: or use a search algorithm to pop the domain from the stack before starting the loop, that would be more efficient
edit2: even better: assuming all bot names are the same, you just iterate over the stack (constant), and pop each domain (new stack minus popped domain) and feed it to the 2nd loop.
edit3: SCRATCH EVERYTHING. The bot names don’t even have to be the same, you just iterate over a constant stack of
botname@domainentries and pop the iteration from it… feed the new stack to the second loop (nested) and done. Why did I take so long to reach this?This would increase server load and traffic by a huge amount though, and would the benefit really be that big?
There are already quite good places to find communities for stuff you’re interested in, and less niche stuff will be federated “automatically” by these rules unless you’re on a small instance.
Good community lists:
https://lemmyverse.net/communities
https://browse.feddit.de/
Reddit communities now on Lemmy:
https://sub.rehab
https://redditmigration.com/
This is dancing around the issue and doesn’t really solve the problem. Also, using 3rd party sites to be able to use the platform? Kinda sucks.
Yes and yes. I have no idea how kbin works but numbers are correct.
Edit: numbers are actually not correct
And I’m not even taking into account platforms that are federating with lemmy, this is amount of actions needed for lemmy to simply federate with itself.
I don’t think full federation is required. I wouldn’t want Roblox and Minecraft communities to have porn visible to the users that sign up on those instances.
Dude… If I make an account on instance that has minecraft community on it and go into the fediverse and subscribe to porn sublemmy then it will show up anyway… If you don’t want to see porn then turn off NSFW setting and if some instance doesn’t tag porn as NSFW report it to your admins and defederate from them if they refuse to resolve the issue.
this number is only completely relevant for someone on an instance all by themselves or with no communities at all. And discounts instances that are or will de-federate either partially or fully. It also assumes some need to be a part of all 12,000 communities. I think tools like you linked solve this issue anyway. I personally believe to a certain extent every community being federated to every instance kind of defeats the purpose of federalization.
But that’s not my point though? The point is to federate with all sublemmies of FEDERATED instances. The problem is that when you federate other instance it doesn’t federate sublemmies of that instance automatically which limits interactions between instances by a HUGE amount
It’s a risk reward question then. That would 100% slowdown the initial federation if it needed to pull in every community and even if that was accepted, should every instance constantly poll any instance it knows about for new communities? Also you aren’t guaranteed to need all those instances anyway and then that’s just a waste of space and processing power. Correct, it limits interactions but only to what’s necessary which allows instances to be ran on lower powered hardware, allowing more people to join in. With the possibility of third party tools I don’t see much of an upside of building that into lemmy.
All you need is one person that decides to run a bot for their instance and it will be done anyway.
It would be cool if you could see stuff from other servers/instances without someone from your server/instance interacting with it first, yes.
At the moment I am personally not complaining though. I am sure the developers have tons of things they’re trying to work on at the moment. And I can still see stuff from other instances/servers.
Check my math (which I hope is correct) and see the insane amount of actions needed for full federation with our current system
I agree. My instance feels very dead at times because of this. A lot of communities aren’t added and adding all on my own is tiresome. :c
This is one of the key issues with the entire Fediverse setup and it’s not one there are any good answers to.
Not least since it makes being on the biggest instance objectively the best experience, this defeating the point of decentralisation.
I guess the only way to make things work will be to setup bot accounts that will exist on each existing instance and automatically interact with every community from other instances. With bot account it will be easier to do 10 000 000 actions when it’s impossible to do manually.