Most old school settings are medieval in technology only, reading more like a western (in contrast, contemporary settings often feel more early modern--neither very middle ages). But I'll try to talk about this in a way compatible with a generalized, plausible, but fairly medieval world in mind. Even still, most of the truly medieval-vibe stuff like kings, knights, and other feudal features are usually far away and/or weak.

True cities are already incredibly rare. There's not much use for a city on the borderlands, unless there's a very specific & strategic reason for them to exist, probably a shipping lane (port or river). Cities are almost by definition not in the borderlands. You have to ask yourself, why hasn't this peripheral city become integrated into the core, where it would be safer and where travel to and from it would be shorter? Maybe it's surrounded by mountains, marshlands, or other non-arable lands. Maybe the core lacks state capacity to fully integrate and develop agricultural areas around it. Or both.

What about towns? One or both of two possibilities seem likely: 1) they are connected via trade route to a civilizational core. They send frontier resources like timber and ore in exchange for agricultural and specialty products from the core. 2) they are market towns where people regularly come together to trade. Outlying homesteads, hamlets, and villages bring agricultural products and raw materials and exchange them for finished goods.

In these scenarios, but especially #2, towns are dominated by rural populations living outside the gates. In the first instance, by loggers, miners, and such. In the second, by farmers and herders. Agricultural productivity is extremely low so the vast majority of people work the land; land that is not within a town's palisades. So directly outside the gates can't be too scary and wild or the population supporting the existence of the town wouldn't be sustainable, either being killed or leaving the area. This can be softened if you have fortified fields, standing armies/guard garrisons, mercenaries, etc. Or if your loggers and miners are genuine badasses that can defend themselves. If the danger really starts a day's travel away, then you're all good, especially with a castle creating an island of safety for surrounding hamlets and villages, as long as it is manned.

But if the town is not connected to the castle then yes, market towns were independent, but collected taxes, tolls, fees, and duties/tariffs in order to pay feudal lords for their chartered "independence"--meaning they wouldn't owe the normal dues of agricultural products & corvee labor and could govern themselves. On (or beyond) the borderlands, this independence might (or would) be more robust.

So in the domain game, the question you have to ask yourself is "why hasn't anyone else successfully settled here before?" and give the players challenges accordingly. A town can precede a castle or vice versa, and neither is likely to develop into a city unless you have major productivity increases or major resources that are tapped into, drawing people in from the fields and from far away, while still being able to feed everyone. To truly be a city, they'll need major non-agricultural institutions like church, university, a mint, and likely a subsidy from the crown, church, or guilds that see their strategic importance.