Any post and community could be accessed through a theoretically limitless amount of instances, which also means a theoretically limitless amount of URLs.
Will this hinder Lemmy from ever coming into the mainstream? If I type any topic in Google, I will get a reddit thread that deals with that. Can something like that ever happen for Lemmy?
Please make sure that you’re only indexing Lemmy communities and Kbin magazines (i.e. not microblogs)
In the wider fediverse, there is an actual expectation of privacy beyond “well it’s technically possible to scrape everything so we may as well give up”. Several people (with reasons of innocent naivete & explicit and blatant malice alike) have tried making fediverse search engines, but all of them are either dead or blocked.
Lemmy/Kbin is in a unique position where global search does make some sense to have, due to it being a public forum focused on topics (and not people), but there is a very real chance that assholes could use an “unbounded” fediverse search engine to find vulnerable people (quite a few of them specifically fleeing to the fediverse to avoid that kind of problem) and harass them.
The concept of privacy within today’s Fediverse is asinine and everyone should be pointing that out at every opportunity. Doing otherwise, making believe that some sort of code of conduct or public shame cycle is somehow going to keep people safe, is ridiculous and even more dangerous than a public search engine. By not talking about, very loudly, just how trivial it is to gather this data and how impossible it is to remove it we’re sticking our heads in the sand and there will be people who suffer as a result.
Feel free to go do that then