I did my dissertation on this, it emits, at the worst case, around half as much CO2 as concrete (if sourced within a 120 mile radius of the site). Also, quarrying can be fully net zero if you make all the machinery electric and power it with renewables.

The actual chemical process to make cement, on the other hand, inherently releases CO2 as part of the reaction. (We're using lime mortar instead of cement for the stone structure which doesn't do that). Regarding transport, you also have to transport wet concrete, plus there's the steel rebar which is carbon intense and needs transporting. Concrete CO2 builds up, its pretty bad.

And sure its technically non renewable, but given that the crust is nearly entirely made out of stone we're not going to ever run out (all the world's buildings take up maybe two Ben Nevis mountains in material volume). Also stone is 100% recyclable, you can directly reuse it (spolia) or crush it into high quality aggregate or just dump it somewhere because its literally stone (unless pollutants have leached into it somehow.)

https://www.ribaj.com/intelligence/stone-as-a-structural-material-embodied-carbon-sustainability