I think this is just readers POV bias. He's facially deadpan, and we don't really get to see his inner thoughts or how he processes information, we only see FL's. They start with a business relationship, but he does seem to increasingly want to care about her, and he's slowly understanding her circumstances even when she never tells him anything. I think his characterization is interesting, and he treats her fairly but sternly. IMO, to me he seems to be slightly on the spectrum, with low emotional EQ. But he's definitely not dumb or uncaring.

I really don't like the duke, he's too stupid for me, it's very refreshing, but he's too arrogant and rude to understand the importance of certain things, and he doesn't care at all, he ain't trying to understand nothing.
He only want things in his way, and fuck her opinion or privacy.