But why does Kiriya think Youmei betrayed his brother by falling in love with Hizuru???

If I remember correctly, Youmei made a promise to his brother after he found out that youmei had sexual relations with his mother even though she was the one who started it and since kiriya seemed quite close to his brother it hit him even harder. And since falling in love with another man is considered immoral than that would break the promise of not having immoral relationships. I can't remember the exact promise between them but it is something to that effect.