I actually get what you meant. It's not very obvious and clear but it is implied. Like how they wouldn't do anything because Hyesung avoiding to talk about his trauma and past was pictured as "we don't want to give him hard time", like we afraid of his reaction and response rather than it is just bad. Whether or not his reaction is positive, they have no rights to decide it for him.

Why the author always try to make Hye sung the bad one