I think he is implying that others perceive him as impotent because he never showed interest in girls/guys like that in spite of his daily Subtance X. Because even if he a little bit of sexual desire to amplify it would be crazy high, and since its nothing its basically saying he has 0 potentcy to start with (coz anything multipling a 0 is still 0). That is why he wants to fall in love fast and prove them wrong lolol
The anime explains it better but what happens with substance x is that it was originally designed to enhance all three "hungers" eating, sleeping, and sex. the problem was that substance x was so disgusting that the priests wouldn't touch it (substance x was created by a member of the church) so he gave it to the adventurers and asked them to tell him of any side effects (cause he couldn't get anyone else to take it., They found that substance x randomly takes away one of the three hungers so because he still loves food and sleep the hunger that was suppressed was sex. hope that helps explain it better.
To expand on ice queen’s comment, specifically, one of the 3 hungers is “sacrficed” in order to increase the person’s abilities/strength/growth.
So it says it amplifies the three primal desire like why he eats and sleeps a lot but doesn't that mean it also amps him sexual drive why is he so sad like it made him impotent.