So he loved his little bro????like that of a lover love?

Yes, and the little bro didn't return his feelings. Things went awry between them and that's why he left.
http://www.mangago.me/read-manga/yorokobi_wa_hiza_no_ue/ is the prequel to this and I think there are bits with the brothers relationship in there (not sure tho, I read it ages ago, gotta re-read it again someday).

Xc sad