I think the point of that scene is that Yuji is shouting right in Kirato's face—"You don't know what it's like to be cast aside"—while simultaneously wanting to abandon Kirato just like that. He says, "You don't know what it's like to be hated," yet Yuji was harsh with Kirato before—behavior that made Kirato wonder if Yuji hated him or was upset with him. He says, "You don't know what it's like to be told you aren't needed," but Yuji did exactly that more than once—telling Kirato to leave him alone, or swatting his hand away when Kirato tried to wipe away his tears. Yuji is so consumed by his own pain and his distorted self-image that he fails to realize he is doing the very same thing to Kirato—and that Kirato *does* know exactly what that feels like; he was simply trying to be a source of stability for Yuji.
It makes sense that he’s avoidant, I think I would’ve liked to see that present itself a bit differently.
Instead of explaining the heart of why you’re avoidant “you don’t know what it’s like to feel rejected and not needed,” I feel like it’s more relatable to find excuses and rationalize the breakup without admitting it’s because you’ve been hurt before.
Not that what was represented in the manga was unrealistic, I just think the excuses are more common since it’s essentially a defense mechanism against vulnerability.