Same. His naivety to his son's living conditions was irking. But at the same time, he trusted his father-in-law because he already helped him in his early years as crown prince and emperor. Plus the empress and other big political figures also don't like that he has a son from the faction opposing his current biggest ally (his wife's family). So the "Wow. They hate me that much." is such a sad comment from Asha himself; literally [almost] nobody wants him near the other members of the royal family. So it was really satisfying to see him progressively deflating pride as a father. I can see that his mini self-depracation was almost at the same weight of Harty's anger when she found out Hina was being fed lies about Asha's group.
...in other stories that I don't hate the emperor...he is lacking but he is not bad. He entrusted his first child to an adult and that adult is the one who's at fault. He did what he believes was best and he genuinely love his children... that's why I can't blame him at all. And I'm glad he finally know rhat truth that was kept hidden from him.