From the README:
Mastodon 4.x radically changed the frontend, to much dismay from the actual community. It’s now a (slow) webapp, which requires access to lots of API routes that were previously unavailable to unauthenticated parties. It gives the public a much deeper view into your (private) community, both non-techincal (instance home pages now show an ‘explore’ page nobody asked for, that shows public content from instances you federate with. [you have to fully disable trending]), and on a technical level (toots and search API are publicly available allowing for much easier programmatic scraping).
I do not know the exact nature of the changes in 4.x but imho it’s all about preferences. If someone wants this shield, they should use it. And there’s a whole lot of fedizens who do not benefit if someone scrapes the fedi and makes it deeply searchable.
As I see it there’s two extremes in microblogging: Public-square microblogging a la Birdsite, and personal social networking microblogging in your friends network. A Hometown server where people only use local-only toots is an example of the latter. Both are perfectly valid use cases.
Until someone gets burned by posting things they think are private, but due to how the protocol is designed are not. I think these kind of failure modes should be best avoided.