The baby sitting the friend's little brother thing depends on the age gap. You can be only a few years older in that situation and in a similar age group.
This story is more akin to a teacher dating one of their kindergarten students down the line. The adult watched him grow up....so yes it's wrong.
Can you, as a teacher and adult, imagine dating one of the kiddos you used to teach/watch? I personally could not.

Pedophilic disorder is characterized by recurring, intense sexually arousing fantasies, urges, or behavior involving children (usually 13 years old or younger).
Sooo, I don't see the problem. Yeah he does take care of him when he was younger but does he attracted to him when he was a child? No.
It like this, you once babysit your friends younger brother. Just normal when you are teenager for 2-3 years. He always proposed to you. Then that younger brother always hangs out with you like eating out.
When he was 20, he confess to you again but you cannot help felt something with his persistence. You fall in love. Does is it wrong?
Sooo yeah... Plus in this world they literally half animal. They have animals instinct.