To be fair the Duke is purposefully playing into his father's misconception that he's gay and in love with his male servant.
Plus it's been stated that Chloe's perceived as androgenous, and a lot of people are gossiping about her being the duke's loverboy, so among the general confusion plus her close friend's recent coming out, I'd say that it's a understandable assumption to make

Ok but I seriously hate how fls always assume “ :o he doesn’t like girls, he must be gay” or “ :o he doesn’t like girls, he must have erectile dysfunction” like wtf is wrong with them. Idk it always pisses me off