Stories like these are often about protecting the status quo, or at least protecting it from change involving harm and destruction. Refugees generally want to see the status quo changed because things suck for them. The more radical among the refugees will not be above using a bit of harm and destruction to get what they want. The ends justify the means and all that. I find it better to use the term “antagonist”. They are people that the protagonists have conflict with, but they aren’t necessarily bad people.