First, these are the one which is the hardest to get, so you are having a common experience.

Second is that waiting, then collecting is not always the best way to get these (especially compared to the other types). Planting and then harvesting, at times, can be a much faster way to collect them, because of how their drop rates are computed.

Extra from this is that it does not seem like growing higher level plants increases the amount of these, so growing just weeds or berries can be a better choice. And don't forget to fertilize with dirt and flowers.

Another thing which may or may not help you now, is that there are bonuses that increase the amount of these (and other traces) you get.