The authentication could be another service, split from Mastodon, Lemmy, Pixelfed, … that only gave that service. The instance asks the auth server about “user@instance: password” and the server just says “OK/fail”. That or sending the user to the auth server to get a session cookie.