We have paused all crawling as of Feb 6th, 2025 until we implement robots.txt support. Stats will not update during this period.
We have paused all crawling as of Feb 6th, 2025 until we implement robots.txt support. Stats will not update during this period.
Then I’m not sure what point you were trying to make in the above conversation lol.
The point was “don’t add another bot into the pile of millions of bots that hit people’s servers every day unless you’re gonna be polite about it”