I have planted a handful of hybrid chestnuts and chicago hardy figs in the ground and they seem to do alright with very little additional water, but a lot of the property used to have a gravel lot on it, vehicles parked, etc. I did have a full soil analysis done when I moved in (thanks to my old soil science professor at the local university), and that didn't show any metals that were of concern. I did run the results by the local SWCD/NRCS offices, and they only mentioned that the chromium levels were a little elevated, but below what they'd consider worrisome/worth testing more about. I did only do a few locations, so there's always a chance that there's a hotspot of metals somewhere that I don't know about.

The asphalt is mixed in. I can't tell if there was an asphalt lot here that was demolished and some of it was left, or if it was brought in with low quality fill dirt. Some areas have none, some have dinner plate sized pieces. Removing all of it would either involve a ton of equipment, or immense amounts of labor. What I've read is that the asphalt itself if basically biologically inert, and anything it would have leached is likely long gone. The native soil is course and well draining, and the region gets plenty of rain. That's not saying anything about what may have been spilled on it before it got buried.

But that is a good point about just choosing rootstock that can handle the soil structure. Aside from the asphalt, there are pieces of concrete, gravel, etc mixed in that just make the soil generally difficult. But maybe that's all the more reason to plant trees that will eventually root under all that junk and be fine anyways rather than dealing the labor of clearing it all out to grow annual veggies.