Doesn't look like it'll become a harem no. He'll probably at most he'll have a bunch of followers who admire him as a master, not as a love interest.
Also, funny how you went from "I'm bored at chapter 4" to "It was a good read" just like that. xD I honestly think it's a pretty good manhwa. Even side-characters are often a bit more fleshed out than the usual wuxia stuff, and there's still a lot of mystery around the main character. As in, we only know a tiny bit more compared to the other characters. "He lived there 500 years ago, as a sword god." But that one sentence is all we know about it.
Meh 2 ch in already got a thot, also guessing the so called immortal realm they talk about is probably only the beggining