Not wanting to answer a question is fine, but doing that by just being pretentious isn’t a good way of going about it. Being forthcoming about not really wanting to talk about your music preferences is fine, changing the subject politely is also fine so you don’t leave the onus of carrying conversation entirely on the other person.

Acting like your music choices are too out there for anyone else to understand is a good way to not have them relate to you in any way.