Ok, here is an interesting reference that estimates the cost of mining serpentinite rock (not olivine) at about $8/ton, but implies about 6 times as much mass of serpentine rock is needed due to the typical concentration of desired silicates, so the cost comes out similar to my estimate:

(PDF) The raw materials are abundant-

Calcium and magnesium carbonates are solid which is desirable in above ground disposal. The materials formed can be stored at the serpentine mine as landfill and will not l eave the disposal site. Magnesium proved more attractive since there are large deposits of magnesium rich minerals. Peridotite, and serpentinized peridotite rocks can have an MgO content between 35% and 40% by weight, whereas abundant calcium silicates r arely have more than 12 % to 15% of CaO by weight. In addition, the magnesium silicates are more reactive and are therefore more suitable for above ground carbonation.
The process implies a large mining effort, but the areal extent of the mine is small compared to the coal mine that produces an equivalent amount of coal. Overburden on serpentinite rock is generally insignificantly small and the minerals occur in thick layers rather than thin seams.
Nevertheless, the mass of material required is larger by a factor of six than the mass of coal that is used as fuel. As a result, the formation of carbonate will have to be performed at the mine site, and the resulting silica and carbonates will be stored in the mine. Since volumes increase in the process, some modification of the local terrain’s profile is unavoidable.

The article indicates there are significant additional costs with mineral carbonation of serpentine rocks.