TikTok’s parent launched a web scraper that's gobbling up the world’s online data 25-times faster than OpenAI
fortune.com
The crawler, dubbed Bytespider, is scraping the internet at 3,000 times the rate of other genAI tools like Anthropic.
  • zod000English
    3·
    1 month ago

    I don’t hate all bots, I hate this bot specifically because:

    • they intentionally hide that they are a bot to evade our, and everyone else’s, methods of restricting which bots we allow and how much activity we allow.
    • they do not respect the robots.txt
    • the already mentioned lack of rate limiting