The rise and fall of robots.txt
www.theverge.com
As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.
  • BogasseEnglish
    281·
    10 months ago

    Detecting crawlers can be easier said than done 🙁

    • KillingTimeItself@lemmy.dbzer0.comEnglish
      11·
      10 months ago

      i mean yeah, but at a certain point you just have to accept that it’s going to be crawled. The obviously negligent ones are easy to block.