“Loyalist” would probably hurt more. Especially once you remind them that the loyalists were fighting against the patriots, thereby excluding them from being patriotic Americans.