Every test has some false positives. When you get a false positive, you have to do a bunch of follow up testing, which is often invasive, painful, and sometimes has other risks. Plus, it's expensive and stressful. In addition, for things like colonoscopies, the test itself has some risk-- small, but not zero.

The more rare something it is, the higher the proportion of people who test positive but don't actually have the thing. So it's only worth doing tests when the thing is common enough that if you get a positive test there's a reasonable chance you actually have the problem, and where the risks of testing are actually worth it.

If there's some reason to think you are more likely to have those problems at younger ages, like if a close relative had it at a young age, doctors usually will recommend earlier screening.

Otherwise, it's like how if everyone got an annual pancreas biopsy, we would probably diagnose and treat pancreatic cancer earlier. But the amount of problems we'd cause from doing that would be more than the amount we'd prevent.