I think it depends a lot on the nature of your world. In most of our myths, monsters are a product of some breach of the ‘natural order’ for instance, I don’t believe a centaur is ever described as a monster, though they are often depicted as a menace. The Minotaur on the other hand was a punishment devised by the gods, and I think could be considered a monster.

If you don’t want this system you could simply state the monsters are an especially dangerous type of animal, and could include, say, a specific dangerous wolf as a warg while generally excluding wolves that keep to themselves.

What is and what isn’t a monster can be (probably should be) culturally defined unless you really need something to only interact with monsters, like Pokémon, or D&D’s distinction between charm person and charm monster.

It depends. A monster can be any kind of being which is perceived as dangerous and malicious. They might even be a culture-bearing species. I’d consider the distinction between “monster” and “not monster” to be one of perception. Whether something is considered a monster within a world might depend on many factors like location and even social circle.

Let’s say we have a pseudo-medieval fantasy world. Humans have settled all over the place but there’s also a forest-dwelling race that’s extremely protective of its territory. Let’s say they look like overgrown bodybuilder gorillas; very imposing. Unlike humans those gorilla people don’t care about spreading out; they just have one specific woodland area that they claim as their own. While they do have a culture, they don’t care much for things like permanent structures or settlements. They also don’t like rhetorics and neither send nor receive diplomats. They don’t care much for the outside world, especially for things like cultural exchange or trade goods. They especially don’t like unannounced large groups of people.

Traders all over the continent consider them inherently malicious beings who will slaughter anyone who passes anywhere near their territory. (This typically happens when a caravan is behind schedule and decides to cut through gorilla territory.) You can’t make deals with them; they just want to kill you. Any trader will probably consider them to be monsters and most places that aren’t near their territory will do so even more because they only hear about them from the traders.

The areas adjoining their territory might have a different view. There are villages near the border and the people in those villages know that the gorilla people are perfectly friendly as long as you respect their territory. They’ll even let you go in there as long as you behave yourself and respect their authority. Perhaps they even barter goods and simple services. The villagers are convinced that the gorilla people are decent people who just happen to have their own way of life.

The nobles of the adjoining areas, however, consider them thoroughly savage because they don’t have a head of state or any kind of discernible court, have no temples and don’t follow any officially recognized religion, don’t tolerate diplomatic missions, won’t enter into trade deals or free passage treaties… The list goes on. These animals won’t even let traders pass through their territory unannounced as if they didn’t care about the long-distance trade economy at all.

Now, are the gorilla people monsters? It depends on who you ask.

• The local villagers don’t think so; they don’t care that much about whether traders have to take a detour around the forest (and might even like it because now those traders visit a lot of villages they otherwise wouldn’t). They treat the gorilla people as neighbors and know that you can get along with them just fine if you respect their laws.
• The local nobles may not consider them monsters (due to a general lack of unprovoked trouble) but also don’t really consider them people. They expect civilized people to behave roughly as they do, which the gorilla people don’t.
• The traders generally do see them as monsters. Whenever a trade caravan cuts through their territory just a little they get attacked. The fact that the gorillas target caravans because they’re groups of people with armed escorts who cross the border whenever they feel like it is irrelevant to the traders.
• People from distant countries only hear the traders’ embellished stories and are convinced that the gorilla people are evil monsters and probably also demons.
• The gorilla people themselves just want people to respect their damn borders and stop with the bullshit smalltalk, thankyouverymuch.

I believe there is no strict rule for how to distinguish between monsters and animals, beyond those you already suggested. While you can pick freely which characteristics distinguish them, I’d suggest tailoring them to your world. They should mean something. Perhaps monsters are magical to show off the magical system/ideas of your world. Perhaps they are tormented souls of humans to show how grim your setting is. You might make monsters more intelligent to help people emphasizing with them, or blur the line completely by just making monsters a fictional kind of animal to support a scientific narrative.