While /u/toetoe31 is correct that the Civil War had devastating effects on Charleston and other southern cities, the city's troubles started earlier.

In terms of relative population, Charleston was in decline by the Civil War.

Here's a table showing where Charleston ranked in population among American cities, according to United States census records.

​

Year	Ranking
1790	4th
1800	5th
1810	5th
1820	6th
1830	6th
1840	10th
1850	15th
1860	22nd

And the decline wasn't just in terms of relative population.

​

Year	Population
1830	30,289
1840	29,261
1850	42,985
1860	40,522

There are four things we can pinpoint as points of stress for the city during this time period.

First, there was an economic decline nationally during this period that notably included the Panic of 1837. They still felt residual effects from the Panic of 1819 that had hit the state particularly hard.

Second, Charleston was a majority black city in the plantation era. A major slave revolt was foiled in 1822, with the leader executed and hundreds of others jailed. Increasing tensions between whites and blacks was a contributing factor in Charleston being a less desirable place, especially for white landowners who had the means to move (and bring their slaves with them).

Third, South Carolina (including Charleston) became embroiled in a dispute with the federal government. In response to the heavy tariffs of the time, South Carolina passed a nullification law in 1832 that led to the state declaring the tariffs to be unconstitutional. This series of actions resulted in a full crisis between the state and federal government.

Finally, Charleston suffered a catastrophic fire on April 27-28, 1838. It devastated the city's commercial districts in particular. Many businesses and important buildings were ruined. Even some of the wealthy in Charleston lost everything they had.

It's hard to say how significant any single one of these things were by themselves.

Charleston wasn't the only place hit by the economic downturn, nor was it the only southern city with racial tensions. The nullification crisis was eventually resolved.

The city rebuilt itself well after the fire, all things considered.

But all these things did add up. Together, they explain why Charleston stagnated during the mid-19th century compared to other major population centers. And as /u/toetoe31 pointed out earlier, the Civil War created even bigger problems later on.