That's not even his issue. It's Shizuru smiling. If I remember right, they discussed Shizuru should only smile the way he did with Minato. Pair that smile with 1) Director being female, and 2) the underwear, Minato's walls got activated to save both of them getting hurt from Minato lashing out.
i'm at minato side of course, like if i see other person giving my bf his underwear i would be feral
.
thanks for the update