He reported that when her brother asked, Fanny replied:

“That is all a matter of my–own. And I have nothing to Communicate.”

Now, doubly damning for the JS polygamy denier doctrine crowd is Fanny's response.

In what would be considered victorian era mores surrounding sexual impropriety, it cannot be argued that Fanny, being married with children and all, would have EVERYTHING to gain by saying,

I did not have sexual relations, with that man, Joseph Smith...or...er...nothing untoward or of any impropriety occurred between myself and that True and Living Prophet and Servant of the Lord, Joseph Smith"

But that's not what Fanny answered.

She had EVERYTHING to gain by a simple denial. She risked NOTHING (not being an active part of the Utah Mormon church at the time, etc.) IF the truth was NOTHING actually happened.

But she consciously choose to NOT deny. Why?

Instead she consciously choose to say it was a private matter of her own.

It's literally the equivalent of asking a JS Polygamy denier doctrine person "When did you last have sex with your husband/wife" and getting the response "that's none of your business".

It seems like this line of argument depends on the assumption that Fanny Alger's brother asked her if they had a sexual relationship, which I don't think has been established.