**Spoilers Ahead**
I read it as Soongap was reminded of his past by seeing his brother. Like it reminded Soongap of his crappy childhood (his dad's cold treatment, his mom's death, etc.) and eventually the traumatic way he was forced to leave his home to become a slave.
IDK it kind of felt like his brother was a reminder of what he could have had and what was taken from him. So Soongap didn't want to be reminded I guess?

I don't understand why Soongap hates his brother so much at the beginning of the story. Like, I know the brother turned horrible as the story progressed, but even before that, Soongap just constantly rejects him and barely acknowledges him and basically treats him like the worst of all evils, while the brother initially truly just wanted to reconnect with and care for the only person who ever showed him affection and who's still alive. I just don't understand.