I did a couple of searches with duckduckgo and google, finding that unlike reddit it doesnt really suggest posts on lemmy, just whole communities. Im curious if this is unavoidable with this number of users or if there is a way to make search engines suggest better lemmy results and make the resources accumulating here useful for a broader audience.
For results to appear in search engines they need to be indexed by their crawlers (bots that check webpages and analyse their content, following up any links they find in that page) so it takes time. As a Lemmy instance becomes more popular, more links will show up for it in other websites and by consequence more crawlers will index its pages.
Isn’t another issue that it requires Javascript for the content to be visible?
Not that i’m aware, I think sites without Javascript should be indexed even better since they are easier to parse by the crawlers.
Yeah that’s my point - Lemmy requires JavaScript whereas Reddit will still show the comments without it (at least on old Reddit IIRC).
You can also do a site search on google, I guess the problem remain that you have do this for every server.
“site:beehaw.org”
I have been thinking about this. For other sites it is super easy to google e.g. “bifl socks reddit” but with federated services the information is scattered across many domains/services that the search engines do not know are federated. There either needs to be a site that archives everything in real time that search engines can crawl, or a robust search engine specifically designed for searching posts on all known instances.
Full disclosure I am not an expert on the topic these are just my thoughts.
it is super easy to google e.g. “bifl socks reddit” but with federated services
I also guess that’s the issue. Many lemmy instances do not have ‘lemmy’ in their URL. So if someone searches for “bifl socks lemmy”, they might not see results from these lemmy instances with non-lemmy names.
90% of the content has been made in like, the last 4 days. Search engines probably weren’t even aware of Lemmy before. Give it time.
That said, the federated nature may cause results to always be deprioritized, as the exact same thing may be available on dozens to hundreds of servers with different domains and policies. Search engines don’t like that.
Is there a way to get a live feed of all submissions to all of Lemmy? Cause I’ll make a searchable index of everything.
You can rss instances, but I don’t think that is possible across all instances.
