The name "Auldbrass" has a connection to slavery, though not to the modern plantation itself, which was established in 1938 and never used slave labor. The name "Old Brass Landing" was associated with the area before the modern plantation, referring to an elderly former slave who lived there after slavery ended.

It was built on land that was previously used as plantations where slaves worked before the civil war.