This is probably a rhetorical question but I want to answer it anyway, sorry! I haven't read the novel, but I think there are really just a few reasons:
1) for the kids, it's Yutia. I don't think it's an exaggeration to say almost everything has been filtered through her view which was formed through the "friendship spell" (can't remember how they described it) in the very beginning.
2) for everyone else, in the beginning, it was his family's reputation. They were a crime family, so no one's thinking he did good things just cuz (except maybe Evan) lol
3) and now it's really just due to his own reputation. He did some pretty unbelievable things in front of people. Thanks to point 2, his kids doing things that benefit him without his knowledge just seems like he's playing coy or that he has power over them (and since they're now in pretty high positions, just leads the outside character to wonder how powerful Alan is)
TL;DR: everyone's lacking context lol
I haven't read the novel too but based on the manhwa I think you got it right. Also his nonchalant face gives some misunderstanding too.
Why is everyone just interpreting everything however they want it to be