My bad, figured it out :) They simply appear when you downvote a comment. If you upvote, the counters don’t appear.