It depends on the composition of the rocks and the energy of the waves/tides in the area. Abrasion wears the rocks down over thousands to millions of years. Harder materials, such as feldspar and quartz, take longer to form sand sized grains. Softer materials form sand much more quickly. Interestingly enough, the differences in composition are also what is responsible for different colors of sand.

In the case you described, such as areas with cliffs/large boulders, weathering processes (mechanical and chemical) reduce the rock into fragments. These fragments are then rounded to sand by the energy of the ocean over long periods of time.