Appoxo@lemmy.dbzer0.com to Technology@lemmy.worldEnglish · 2 months agoCloudflare plans marketplace to sell permission to scrape websitestechcrunch.comexternal-linkmessage-square3fedilinkarrow-up1152arrow-down11cross-posted to: technology@lemmy.ziptechnology
arrow-up1151arrow-down1external-linkCloudflare plans marketplace to sell permission to scrape websitestechcrunch.comAppoxo@lemmy.dbzer0.com to Technology@lemmy.worldEnglish · 2 months agomessage-square3fedilinkcross-posted to: technology@lemmy.ziptechnology
minus-squaregaylord_fartmaster@lemmy.worldlinkfedilinkEnglisharrow-up46·2 months agoThey’re already ignoring robots.txt, so I’m not sure why anyone would think they won’t just ignore this too. All they have to do is get a new IP and change their useragent.
minus-squareredditReallySucks@lemmy.dbzer0.comlinkfedilinkEnglisharrow-up4·2 months agoCloudflare is protecting a lot of sites from scraping with their POW captchas. They could allow people who pay
They’re already ignoring robots.txt, so I’m not sure why anyone would think they won’t just ignore this too. All they have to do is get a new IP and change their useragent.
Cloudflare is protecting a lot of sites from scraping with their POW captchas. They could allow people who pay