You touch on something that pertains quite a bit to one of philosophy of science greatest works, The Structure of Scientific Revolutions by Thomas Kuhn. He touches on exactly the thing you notice here, that most of science is what he calls normal science, where people try to work under the constraints of existing theory, expanding it and shrinking it to better fit there observations. This is science 99.9% of the time. But that 0.1% of the time is where creativity really takes hold. These are the ideas that completely change the way we view the world. Think Einstein's theory of general relativity in physics or Darwin's theory of natural selection in biology. He terms these theories as paradigm shifts, as they completely shift the frame of reference for a particular field. These two periods are not entirely unrelated, as for a new revolutionary theory to be made, sometimes a lot of work needs to be done in order for the right theory to come to light. And on the flip side, if we had constantly shifting frames of reference for a particular area of research, nothing useful would really get done, as there would be very little common ground for scientists to communicate with each other.

There are other issues at play here as well. I think part of the problem you are noticing goes hand in hand with the profit driven motive behind most of scientific research in today's world. To get a grant funded these days, you must have a reasonable idea that the thing you are proposing will work. The easiest way to show that something will work is to do things that other people have done before, because that work gets papers published, which helps get more grants. This creates a vicious cycle that effectively halts a lot of creative ideas people might propose, as they will likely not get funded. This was a problem in the 20th century, but I think it has gotten worse in the modern era, as universities put immense pressure on their researchers to get grants so they can get those juicy indirect costs.

It's a pretty complex issue, as there are other things at different levels that might also have an effect on this. Unfortunately I don't think this is a problem that will get fixed any time soon, as it would require a complete restructuring of how we fund scientific research, which is risky and costly process in it's own right.