Its also been a while since I read it but if I remember correctly its just a part of their countries genetics, but the King of this country I believe, spread rumors that they are bad and use magic to manipulate and curse others.
Next part is kinda a spoiler??
The King was jealous and wanted to have the MLs mother, who also had green eyes and blond hair but she married the Duke so he took revenge by spreading those rumors and making sure everyone who looks like her suffers. The FLs mother was just a normal women who got forced into doinf what she did, but she knew magic too and thought the FL (but the FL of course didn't know it was magic spells).
The ML mother on the other hand was a princess and could use magic only her people with the green eyes can use, so she hid all the treasures in that white crystal mine and only someone with her same magic can break the spell (the FL).

Hes a good guy at his core, he was so hostile because he also has green eyes and since people with green eyes are looked down on and discriminated against he is angry because he thought the FL had an easy life and was not discriminated for her eyes because she is a noble lady.
Another thing he disliked was that she didn’t use her noble status to help the other green eyed people because she was too arrogant to do so, but she obviously wasn't, because she didn't know there were so many others(the whole city has people from our FL mothers country).
After all the misunderstanding and rumors about her get cleared, he apologized and becomes her true knight and like a real brother to our FL.