Pretty sure that was not in middle of the town...
It is kind of like a hall or lobby attached to the garden. And that garden was kinda magical?
It would be one of the best place to gather to meet other people to make connections... and as the prince, he would sometimes need to do his duty of visiting important places...
Imagine how sometimes a city’s councillors would visit the traditional market to give greetings and such to garner supporters...

why the heck was the prince standing in the middle f the town like a sore thumb???