lemmyreader to SecurityEnglish · 9 months agoAmazon storing classified US government documents improperlyimagemessage-square77fedilinkarrow-up1556arrow-down124file-text
arrow-up1532arrow-down1imageAmazon storing classified US government documents improperlylemmyreader to SecurityEnglish · 9 months agomessage-square77fedilinkfile-text
minus-squareAmbiguousProps@lemmy.todaylinkfedilinkEnglisharrow-up7·9 months agorobots.txt doesn’t have to be followed. It doesn’t block crawling.
robots.txt doesn’t have to be followed. It doesn’t block crawling.