The mother's got to know each other in the hospital and became good friends. I think this implies that HER dad was a one night stand.
But I feel like since ML and his two brothers look alike, the dad was there until at least his birth... But most likely passed.
That would at least explain why the mothers' leaned on each other so much, and why the eldest brother is so mature. First born had to grow up quickly, and get a job to support the other two boys, buuuuut that also ended up ruining the eldest's marriage... since he put his brothers before his wife.

I’m rereading all these chapter and I haven’t seen a glimpse of either father figure did they die or what?