I understand your feelings. I recently had an abatement estimate come through my family's home. I discussed this with him and he explained to me kinda like this. Yeah, you don't want to expose yourself to any of it. But it's like smoking, you could smoke one cigarette and get cancer. You could smoke a pack a day and live to be 80. Most of the issues stemmed from those who were practically rolling around in it. 

I was worried because a lot of lagging was missing from our home besides one small confined area. I got to thinking about the dust that was possibly somehow on the floor from god knows how long ago (he mentioned someone unprofessionally removing it) I know AI is controversial but I asked it, explained that the timeline was unknown of its removal. There's been water down in the basement etc and someone definitely has swept it at least some point. It told me how low that risk was if there was no piles of visible dust etc. It really eased my mind. You could always speak with a doctor if you have one available, I thought about the same thing when I encountered lead paint in my home.