Can't remember Title

I cannot remember the Title of it but I know it was yaoi.

It was about I think a guy who was going out with an older man. The older man is rich and the guy had met him before when he was a child however the man refused to let the guy, as a child, come into his house because he was of a lower social status, or poor, to play with his kid or might have been grand kid. Later on in the story the older man finds out the guy is the child he turned away.

That is all I remember. I would be grateful to any help finding this.
Thank you.