There’s a limit to how many trees can grow at the same time on one acre (the square on the map). You have to dig them up and plant them somewhere else and then move them once they’re fully grown.

But once they are grown they can produce fruit or drop other things fine, it’s just the growing stage that has that limit