Are fetishes bad?

Yaoi is known for fetishizing gay men/relationships. I might sound ignorant for asking this but why is that bad? Why is having this fetish bad? I'm assuming it's thoughts people have in their heads, but as long as others aren't harmed, it should be fine?