Need Help Finding a Manhwa

So a while back I started reading this manhwa and I can only remember bits and pieces, if anyone knows the title I would very much appreciate it!

So I remember the mc was from a poorer family and ends up being sent away to help sort the royal library or something. He befriends the prince(?) and this causes controversy when discovered. The prince and him end up parting ways to get stronger I think, and that's all I can remember.
I think the mc was the son of a woodcutter, but had a strong affiliation for swords or magic or both idk. That might be a different manhwa.