Advice please?

Yeah, so it may be something stupid but can any of you lovely bastards help me out with something?
So I was talking to my crush, and told him that I bought yaoi manga. He said "ugh fetishist" and I was kinda hurt it was like he found me disgusting.
Is he right tho? Idk what should I say guys (￣ε(#￣)Σ