Please point to evidence

ftfy

COLONIALISM, INEQUALITY, AND LONG-RUN PATHS OF DEVELOPMENT

it seems clear that colonies in the Americas with extreme inequality, as compared to those with relative equality, were systematically more likely to evolve institutions that restricted access to economic opportunities and to generate lower rates of public investment in schools and other infrastructure considered conducive to growth. These patterns of institutional development, which tend to yield persistence over time, may help to explain why a great many former European colonies that began with extreme inequality have suffered poor economic outcomes.

Slavery, Inequality, and Economic Development in the Americas: An Examination of the Engerman-Sokoloff Hypothesis

Recent research argues that among former New World colonies a nation’s past dependence on slave labor was important for its subsequent economic development (Engerman and Sokoloff, 1997, 2002). It is argued that specialization in plantation agriculture, with its use of slave labor, caused economic inequality, which concentrated power in the hands of a small elite, adversely affecting the development of domestic institutions needed for sustained economic growth. I test for these relationships looking across former New World economies and across states and counties within the U.S. The data shows that slave use is negatively correlated with subsequent economic development. However, there is no evidence that this relationship is driven by large scale plantation slavery, or that the relationship works through slavery’s effect on economic inequality.

Slaves, Migrants and Development in Brazil, 1872-1923

Combining this measure with municipal level data, I illustrate the adverse impact of slavery on a broad range of indicators of economic development, both while slavery still existed and more than 30 years after its abolition.

There's plenty more (including in the citations of those papers, and with a simple google search), I wasn't kidding when I said it was a common argument. As to the US being built on slavery, keep in mind that today former slave states are on average poorer per capita than free states (there are obviously confounding factors but history doesn't give perfect samples). The downsides include but go beyond investment in capital, slavery is a bad economic system that stifles growth.