From what I remember, it's because FL was being "too much" and ML had to bottle up his feelings to avoid fights with her but their relationship just got worse after that. Eventually when FL confronted him and asked for a break up because obviously their relationship was falling apart because ML "changed". ML just agreed with ending things since he was also fed up with having to accommodate FLs feelings.,, you can refer to ch19 :'33

And they went back at it again... I was hoping to see more of ML's backstory that show how he processed everything after he realized what he had lost (^~^;)ゞ but this is ok too...