I think it's a variety of factors, including the other characters having their own sets of biases unrelated to Anders' backstory, and probably that they haven't met all the templars at skyhold, etc.

I think any lack of wanting to acknowledge it OOCly might just be that our sampling of templars is really small and not super reflective of what canon templars have gotten up to. The more peaceful templars are seen as exceptions to the rule, which to be fair I think they basically are. They're dealing with a really violent and corrupt history, and most characters are going to respond to that first and get to know them after.

I don't think you should give up hope though, since honestly some of the most satisfying CR can come from characters who used to mistrust or dislike each other. If you're uncomfortable with how something is going, I bet you could bring it up to the other player OOCly and try to figure something out that won't ruin your day.