I don't know how you came to this conclusion, like did we read the same story?? If you read the last chapter, the Duke even admits that he never actually loved his original wife. He's confronted with all her acts of love for him through her diary, and recognizes that he, at any point before she died, could've had everything he needed by just accepting her love and loving her back. He was obsessed with what he owned and acquiring more, and put his original wife into that category of possessions, when she's an actual person that needs reciprocal affection.
That's why he goes off the rails when the fl starts doing things for herself and divorces him. He sees people as possessions, pawns, or enemies, and that type of thinking does not make him capable of loving someone else. But even in the end, I can't even say if he learned his lesson or not, because he doesn't even regret what he did to the kingdom just the fact that he should have loved his wife instead of ignoring her and being uncomfortable by her presence and kindness.

I do agree that what he did was truly wrong but he's just a man who truly love his ex wife. The fl didn't know the promise in freesia since she did not read the novel until the end and assumed that the duke doesn't love his wife and just a cheating bastard. He doesn't want to divorce and did menial task just to get back his wife and when it failed he went to the worng path but yeah it doesn't still justified what he did but the author should have just killed him right there so he can be with original Tareya and not suffer like that.