Responses
https://www.mangago.me/read-manga/the_fallen_duke_the_knight_who_hated_him/
Not this one but your summary reminded me of it.. I love this story
Maybe - BECAME A SLAVE TO THE MAN I ABANDONED (Yaoi)
The uke had a servant, but due to circumstances, the servant was kicked out. That created a huge misunderstanding and the servant now hates the uke. Years after that, the rebellion happened and the servant aided it greatly. The uke also was helping behind the scenes, but someone else took credit and uke was enslaved. The servant now owns the uke and does a loooooot of horrible things.