Nope I dont blame him. The magic that maid and that red hair uses are not known to the public. Only Fl's uncle knows it. So how can he know something is wrong? From the idea that he needs to be careful, he already doing as much as he can to put distance, without alerting the enemies. For hiw protective his father is to shield him from all this evil, he still went out to the world and faced it head on. So no, I dont blame the ml. He did good and is doing good.
It only seems frustrating to us cuz we are seeing everything from a 3rd POV. We know stuff from both sides.

That crown prince is infuriating. He vows to become an outstanding crown prince, knows magic exists, has gut feelings that something is wrong, there was a coup d'etat attempt when he was a child, knows there are traitors in the empire.... but then goes about people being too harsh about the maid that goes against protocol in the court even though he shows signs of resisting the mind control. Like. Come on author, there are other ways of writing drama with that maid