I think that it's the worry that they'll turn on each other one day like his younger brother did to him.
You saw how the thief tried to pin everything on the red prince and how the other brothers momentarily believed it. If it wasn't for the blue prince, then they would've been at each other's throats.
Their whole entire relationship would be based off of trust, but trust can only go so far when another party starts whispering in one of their ears. It'll be alot of hardship and an overall gamble. I would've love for your exact thing to happen, but what a risk it all would be.
The second best choice was to have the purple prince take the throne, and with his unwavering trust and love for the blue prince, have him naturally go to the blue prince for advice when he's unsure about something. In this way, it'll force him to learn and grow under the indirect tutelage of the blue prince.
...I hope that my way of perceiving this made sense!
I don't understand why he would do that. If he thinks that blind one is the most qualified but is physically weak but can be backed up by the other 3, why couldn't he just assigned the 3 as advisers?? (If that's how ancient governments work ig)