• @0x1C3B00DA
    link
    51 year ago

    just because the (corporate) internet works this way now, doesn’t mean it should

    The web worked this way before there was a large corporate presence. Scraping was common during the blogosphere period and robots.txt was the solution everyone at the time agreed on and that’s been the standard ever since.

    I’m happy that you’re comfortable with this model, but I don’t want people who operate like this to intrude on the spaces we’re building to get away from it

    We’re not intruding on this space. We’ve been in the fediverse for just as long or longer; the fediverse has been scrapable since 2008.

    • Arthur BesseA
      link
      3
      edit-2
      1 year ago

      We’re not intruding on this space. We’ve been in the fediverse for just as long or longer; the fediverse has been scrapable since 2008.

      Totally. And while it was scrapable, and scraped a lot, I wish there had been a lot more systematic public scraping of the “federated social web” (as it was called before the terrible name “fediverse” was adopted) back then - I had a lot of public conversations on identi.ca and StatusNet which I wish I could still see, but they now exist only in a bunch of private databases I don’t have access to. 😢