Judging by the latest chapters it seems because of his past of killing all the royals in order to ascend the throne and then going on a 8 year war, he has been stereotyped as gruff and violent man.
And rumors spiral everytime making it worst for him hence how his reputation is the way it it.
For example when during the meeting the nobles started nagging him for marriage, he got angry and stabbed the table to stop them from yapping. And that was it, they all left unscathed yet the rumors that spread to the protagonist was that he tried to kill them and that someone even had to be carried out or smthg like that.
So maybe it's one of those half truths which are something in the novel but misunderstandings in the actual behind the scenes

He doesn’t seem bad enough to be called a tyrant so every time she acts like he is one I feel such a sense of dissonance and kind of annoyed at her.