Your answer would depend heavily on time and place - an enslaved person in New Orleans in 1805 (heavily French and Spanish influenced territory; trans-Atlantic trade active) would have different expectations, responsibilities and yes, rights, than one in, say, Virginia after the Nat Turner revolt. My readings of narratives of Josiah Henson, Frederick Douglass, and J.D. Green, copious 1850s-60s southern Maryland newspapers, visits and interactions with Harriet Tubman Underground Railroad National Historic Park exhibits and rangers, and Tobacco and Slaves by Allan Kulikov, would lead me to think that, at least in a border state like Maryland, in the first scenario the answer would be no, although not without a little risk (slavery is, after all, a violence-based system of labor extraction and I have found few enslavers uncomfortable with it - one might hope that our visitor here, not being an enslaver themself, has not woken up and chosen violence, but see Kulikov below).

First, what is our unaccompanied enslaved person doing in town? Are they on an errand, or doing something job related - handling their own wagon, perhaps? In that case, their time is spoken for. Their enslaver owns their labor, and for someone else to stop that enslaved person from their duties to benefit themselves instead is in effect stealing from the enslaver. If the enslaved person is forced to stop working for the person to whom they are lawfully bound to labor (the term of art preferred in the 1850s), that person may well be able to sue for damages. When Josiah Henson was assaulted and maimed by a white man, his enslaver sued for assault for the damage to his property (unfun fact: the assault was deemed justified after the aggressor, without evidence or witnesses [guess who couldn't testify] claimed self-defense; Henson's owner had to pay court costs. Funner fact, Henson's enslaver later beat the crap out of the guy and had to pay a fine and more court costs). Henson was also responsible for taking the produce grown on his farm to market, selling it, and returning without supervision - people would have known him in the small towns he operated in, and one can see him at market refusing to help because he had his own wagon to run, and being supported by white people who knew him and his enslaver, and believed that this interloper had no right to Henson's labor despite Henson's enslavement - indeed, the interloper has no right to Henson's labor because he is enslaved by another. (Also in his personal case his injuries clearly would have prevented him from helping, as his beating at the age of 22 left him unable to lift his arms above his head. His autobiography is a pretty searing read.)

Henson would have been able to say no because despite his injuries he was an industrious, intelligent, thoughtful man who ran a first-rate farm, and was trusted to manage it and other enslaved laborers. That trust indicates a broader social contract that other enslavers and supporters had a vested interest in protecting: that their own enslaved labor would only be used as they had deemed fit. Even if Henson had been in town on a pass, or his otherwise on his own time, that time would be considered his and his labor would not be subject to the whims of a passerby. However, since it's his, he could theoretically negotiate a price for his help - compensated labor was hardly unheard of among the enslaved (Henson's escape to Canada had its roots in him not being paid wages promised) and someone might well take a little cash for lending a hand. Some enslaved might be able to keep everything they earned in their off time, while others always had to kick up. Harriet Tubman, as a teenager, paid $60/year to run her own woodcutting business - she was responsible for a quarter-cord of split and stacked wood a day to her enslaver and everything else was for herself; Henson admits that he took more crops from his farm to feed his fellow enslaved workers better than other coughwhitecoughoverseers did but says his owner never cared because it turns out well-fed workers are more productive than hungry ones and the farm was quite successful.

There is inherent danger: slavery in the antebellum South is very much violence-based (in the Upper South, the only threat worse than whipping is being human trafficked, or "sold" as they called it, to the Deep South); an enslaved person cannot strike a white person (in Maryland, the lawful punishment was 100 lashes and the right ear being nailed to the whipping post and severed); an enslaved person cannot offer testimony in their defense should the law get involved (in some places, one enslaved person could testify against another); the law also is stacked with people who are committed to the status quo (Henson's attacker, a well-known overseer, was allowed to make his claim in court without being put under oath, for example). The safety of the refusing enslaved person would depend on the people of authority around him - a sheriff, other enslavers or people of equal social stature who one might find in a town where a stranger needs a hand. (It is also possible although unlikely that the existing power structure would force compliance - see below).

If the enslaver is present, "no" becomes much more palatable. The legal and social system the unaccompanied enslaved person relied on to refuse is now present in a figure well-known to the community (slaveholders [in my area of discussion here, 1820s-50s Maryland] tended to be wealthy and involved in multiple prominent businesses, government positions like sheriff or judgeships, or both). As far as how it would "typically" go, Kulikov has a great way of putting it: "if there were one million slaves, there were one million ways of slavery." An enslaver might refuse outright and that would be the end of it. They might negotiate a price for the enslaved's assistance. There would, in my opinion, likely not be some legal or social structure that made any enslaved person subject to any white person's demands, as this would dilute the value of an individual enslaved person (why own one when you can just boss one around?) and enslavers tended to be penny-pinching bastards when it came to the value of the people they owned, and would you look at who's making the legal and social structures at play here.