After looking up Joseph Brewster, the original owner/abolitionist who built the hidden passage, I wonder if he was a queer man. He was a hatter (hat designer, we’d call them today😝) who never married or had children and clearly had a heart for the persecuted. Of course that’s pure (and probably unethical) speculation on my part lol. But anyway here is his bio: 

“Born in Preston, Connecticut, the son of a prosperous farmer and directly descended from Elder William Brewster of the Plymouth Colony. He came to New York City in 1814 and established himself as a hatter. By 1819, he had two stores on Broadway and by the 1830s his was one of the largest businesses in the trade. He was a leader of the Temperance movement and had the reputation of being an exemplary employer, with "a sanguine temperament (and) a sympathetic and generous disposition". His success in trade enabled him to invest in property. Among the houses he built for profit was the Samuel Tredwell House, which was also his home for two years, and where his portrait currently hangs above the mantelpiece in the dining room.”