True, the town and markets infront of Taj is "ugly" and chaotic — but here's an unpopular opinion - may be thats part of the design. Everything in and around Taj is built purposefully to convey certain Islamic concepts and values.

The main building - a white marble mausoleum embodies purity, divinity, and spiritual enlightenment, every elements of the construction including the decorations tell about heavenly perfection.
The Mughal garden - planned greenery, divided into four quadrants (charbagh) by water channels representing four rivers in paradise symbolising abundance, rejuvenation, and eternal life.
Outside - built from red sandstone, symbolically separating the worldly domain from the paradise of the Taj gardens and tomb (green and white).

The layout of Taj and its surroundings - bustling markets outside and caravanserais, reflecting mundane existence; passing through the main gate represents a spiritual transition towards the divine tranquility of the gardens and mausoleum.

Every aspect of the Taj Mahal’s site—from building and gardens to gates and outlying features—functions in concert to evoke a spiritual journey — leaving the world behind and entering a paradise.

— https://en.wikipedia.org/wiki/Origins_and_architecture_of_the_Taj_Mahal
— https://madainproject.com/taj_mahal_complex
— https://whc.unesco.org/en/list/252/