Don't worry, it's not. Most of the girls give up on the MC romantically at some point.
The cycle is:
Fall in love -> realize marriage with him is impossible -> give up on him romantically and focus completely on improving the territory. The Elf girl is just the latest one in the cycle.
Kinda getting boring with the politics shit and the sudden harem :/. Harems is such a big turn off in action manwha/manga