Responses
By all accounts he did get sexually assaulted but no he never got penetrated by either the gardener or the master’s older brother.
He’s never quite understood what the Gardner was trying to do, but he knew it was disgusting and made him feel bad. It isn’t until he starts becoming more aware of the ML and the possibilities of continuing that he begins to unpack that trauma.

Spoilers below concerning MC’s trauma for those worried about how bad it is.
WARNINGS FOR: grooming, SA
I was really concerned about how they’d handle the trauma of the MC - luckily it was all taken seriously.
They don’t show him being abused if you worry about that even when he does fully remember. It’s uncomfortable as it’s meant to be and it’s not taken lightly to how it affected the MC
If yall need more information:
the gardener(?) that was grooming him kept being kind to him enough to get him into the shed, when he’s about to go through with it (fully abuse him) he gets killed by the abusive older brother of the younger lord. Older brother gets mad cuz he’s obsessed with the MC, he keeps him with his back to his chest and jacks it on top of him. Once he’s done, the MC kicks him off and he’s transformed.
He runs away and his sees the younger lord who was insinuated to have just watched and did nothing.