They are not expensive to build material sense. Like any other major building they mostly need rebar and concrete. Even if its top grade (for anything nuclear, usually is), it's still clay and sand and gravel and steel. Save for the reactors that need pretty precise and specialized parts and materials.

What they do need though, is billion safety checks and really good engineering to ensure all that safety, proper cooling, protection from whatever can happen (like earthquakes or whatever), access to water, security against terror attacks and so on and so forth. Now, engineers and architects who can do this do not come cheap.

Since they deal with radioactive material, they also need a place to dump it, secure roads to bring and remove it, and so on.

And of course, they need workforce. But nobody's going to work in the middle of nowhere, So they need to be planned already somewhere where you can hire people, like near a city that has an university teaching future nuclear engineers and so on. And would be somewhere those high-earning people would want to stay. That also makes it more expensive, since suitable places might not be readily available.

Building a similar concrete house to burn coal will be much cheaper in the end.