Absolutely, but the API offers a really smooth and convenient way of doing it without a lot of extra processing overhead. Scraping HTML is a little bit more involved.
But using an API requires integration with every individual site they want to consume. Crawlers do not. For the same reason, LLMs aren’t using the API.
Reddit could also enforce existing limits or change their TOS to explicitly ban this activity of it was indeed leading to millions of dollars in additional operating expenses. They have done neither.
Huffman is just lying about OpenAI and others being the problem.
Absolutely, but the API offers a really smooth and convenient way of doing it without a lot of extra processing overhead. Scraping HTML is a little bit more involved.
But using an API requires integration with every individual site they want to consume. Crawlers do not. For the same reason, LLMs aren’t using the API.
Reddit could also enforce existing limits or change their TOS to explicitly ban this activity of it was indeed leading to millions of dollars in additional operating expenses. They have done neither.
Huffman is just lying about OpenAI and others being the problem.