Yeah I think he prioritize his wife and kid alot, you can't actually always pick a fight any chance you can get, it always attracts unwanted attention, and it was the woman's choice to react aggressively to the mocks, sometime a father figure has to balance out the situation whether it will be safe and beneficial for the family or does he have to stand on what's morally right but could risk their situation. I would rather pick the safe choice especially when you don't know much about the culture and norms of the otherworld.

Idk but i get the iffy way of how they put main lead in this white knight stand and all the women around are just pitiful and… idk it doesnt sit right with me i guess