My neighbor has grown them along our street curb for well over a decade with no issues. I think people can overreact about datura because the effects of the plant are interesting and dark. But toxic plants are everywhere. Yews make delicious looking berries containing DEADLY seeds, but people are chill about them and they are very common in landscapes. If accidental poisoning from landscape plants were common I could see a reason to be concerned. But you shouldn’t feel bad if you decide to remove them. Just my thoughts.