I'm in a similar situation.
Honestly it depends, personally and legally its not that bad. But some people may think its strange due to the potentially maturity difference.
As long as you and him are happy and there are no complaints from his parents then i think it is justified.
And if people you know judge you for it then maybe they are people ......
