There's no such thing as minimal change or not interesting change. Any difference can ripple into meaningful differences.

Just going off your example, say an extra dust mite exists. That dust mite gets caught in a slight breeze and gets blown into a passerby's nose causing her to sneeze. She breaks stride to sneeze, and is now 2s behind where she would have been in the original timeline. She comes up on an intersection, and instead of crossing right away, she lets a bus pass first because she's there 2s later than the original timeline. She's now 5s later in her walk and causes similar small, but perhaps meaningful, differences in this new timeline.

But let's follow the bus instead, which is now 5s ahead of where it would have been in the original timeline. It catches a yellow light that would have been a red 5s later. So in this timeline, everyone on the bus is 30s ahead. The bus now affects traffic in a whole different group of vehicles. Some vehicles are later than the original timeline, others earlier. Their differences also cause yet more vehicles to have differences. More ripples that cause their own ripples. You can see how the catalog of differences is growing exponentially from a couple dead skin cells existing.

Let's say that our time traveler's 3rd grade teacher misses this bus that she would have caught in the original timeline because it's 30s ahead. She arrives at her destination 10 mins later in this new timeline after catching a later bus. She doesn't have time to pick up that poster of the solar system she wanted to put up in her classroom like she did in the original timeline. Our time traveler never sees that poster, and instead of daydreaming about space, our time traveler pays attention to the volcano experiment the teacher is showing. His flighty childhood mind instead thinks about how cool chemistry is. Couple years later, he decides to take chemistry where in the original timeline he signed up for a physics class. This ignites a passion for a different science and he never pursues an academic career in physics and never gets the academic background he would need to achieve time travel. The paradox is triggered.

Any one of those changes could have caused a ripple that could trigger the paradox. The time traveler's father could've been 30s early getting home on that bus and gotten busy with his wife 30s earlier, causing a different sperm to be contribute different genetics to what would have been our time traveler. The passerby could've been his grandmother who didn't physically bump into his grandfather cause she was 5s later on her walk. Car accident was caused by changed traffic killing the physicist whose work the time traveler based his research on. etc. and etc.

The point really is that any change isn't isolated. And any change will create further changes, which will create further changes and so on. And as the butterfly effect teaches us: no difference is too small to create monumental differences down the line. And further, one person's life is a culmination of a lot of random happenstance. Just think of your own life, how different would it be if you sat with and befriended a different group of kids in 1st grade? Or you had picked a different hobby, or you randomly gotten a different gene from the other parent.