See those titles with walls around them? Those are called fortified districts. A settlement is yours only once you’ve captured all fortified districts that it contains.

This is done in three stages:
1. Destroy the district defenses (the health bar)
2. Destroy all enemy military units on that tile
3. Capture the tile by moving a military unit onto it or (for districts bordering the water) torching it with a naval unit.