Responses
Backlash from nobility. Especially since sheβs from a powerful family. You have to remember this a historical fantasy setting. He still needs to apprise the people, even if it means he has to keep her as his Empress. He has no valid reason to abdicate her from her position. Without a valid reason, the Duke family can easily oppose the imperial family with the support of the nobles.
If emperor have power to control his castle/kingdom, why didn't he punish or kick out the empress or that minister? Why he stay silent?