MC never have a Family in his previous life and he was a slave for god's sake! He never knew what's kindness feels like and all people around him and that bastard who hurt him were selfish or monsters.
His Mother left her family suddenly and He doesn't like abandon and unanswered questions so in his view, It's not nice but doesn't mean he hates his mother.
It was just his thoughts that moment so don't worry.
I undesirable that MC is reincarnated so he doesn't really have an familial love for his mother and is jaded by his past life, but calling a woman who left her uncaring, power pursuing family for true love and happiness and lost 1 of her kids and her husband, SELFISH!? she literally was powerless and pregnant so of course she would go back to the safety of her family home and MC only caring that she "raised him with kindness and love" after telling him what happened is wacko. Like she has sacrificed everything and is now being cursed at and humiliated to keep you, her only remaining child, safe and you just care about how she's perceived by your lame ass family.... hate that so much I'm gonna stop reading ヽ(`Д´)ノ