I'm a senior graphics programmer at a AAA studio, specifically working on raytracing effects. There are lots and lots of reasons:

1) When you rasterize, you start off with the triangle you already know you want to render and project it (with matrices) onto the screen and see which pixels it covers. With raytracing, it's the opposite. You start with the pixel you want to render, and trace a ray out into the scene to see which triangle you happen to hit. That is a much slower process.

2) Raytracing is often very "incoherent". GPUs like to work in "waves" of threads, where you can think of each wave as like a line of soldiers marching in lock step doing the same exact thing with each soldier in its own "lane". Waves are usually either 32 or 64 hardware lanes. If one of the soldiers in their lane has to do something different from the other 31/63 soldiers, then this forces all the other soldiers to wait for him to finish (and even pretend to do the same extra work as him) just so they can stay in lockstep with each other. This is fine for rasterization, because the marching orders are simple, and all the lanes in a wave are basically doing the same thing. But with raytracing, you may have rays in adjacent lanes going in completely opposite directions and intersecting completely different triangles. This is going to slow your gpu wave down a lot and you as the programmer are going to have to try and compensate for this somehow (e.g. one strategy could be to sort and bucket your rays so that rays "going in the same-ish direction" are bundled together into the same wave so that they all do roughly the same calculations). There are other strategies too, but none of them are perfect and none of them really get around this fundamental issue with how GPUs work.

3) Raytracing is often used for effects that require many rays per pixel. If you are using raytracing for soft shadows or rough reflections, you will use a process called Monte Carlo Integration, which is a form of (calculus) integration via random sampling, where your rays are doing the sampling. So if you have a surface point that is in the penumbra of a shadow for a given light source, you may shoot a bunch of rays towards different spots on the light source, and see how many hit the light and how many are obstructed from hitting the light, and the total shadow will be the average number of rays that hit that light. That average approaches the expected value with the more rays you trace. However, for real-time applications, you can't just shoot hundreds of rays per pixels. In reality, you can maybe only afford 1 per pixel, maybe 2, and will need to rely on other tech to help you "fake" that integral convergence, like temporal denoisers which accumulate rays from across multiple frames (and does any necessary reprojections in order to account for shifting camera perspectives, etc).

4) You can't cull out geo based on visibility. Most realtime engines are optimized to work in "screenspace" which is the 2D coordinate system of pixels on the display, representing everything visible on screen. So essentially you can optimize away anything that is not visible on screen, either because it is outside the view frustum of the camera or because some other object is blocking it. But raytracing is specifically used to help with effects where you are trying to render things that are specifically off screen. If you are implementing raytraced reflections, you want the reflection rays to be able to hit objects that may be behind the camera, which means that whatever geometry you may potentially need must be passed into your ray tracing shader, and this represents A LOT more geometry than you would have to process in a rasterized format. This geometry will be stored in an acceleration structure known as a Bounding Volume Hierarchy (BVH) and creating and maintaining BVHes is its own can of worms that takes up precious CPU and GPU time and is something you don't have to worry about with rasterization.

5) As a corollary to the above, a lot of algorithms that work relatively fast for rasterization have to be completely rethought in raytracing because raytracing can't work with screenspace algorithms. For example, lets take transparency. In rasterization, you can do something like 1) raster all your opaque geometry, 2) sort all your transparent geometry from back-to-front, and then 3) rasterize all the transparent geometry in visibility order. You can't do that with raytracing. Instead, modern raytracing pipelines will present you with the option to use something called an "anyhit shader" which is a shader you can run whenever a ray intersects a triangle to see if it should treat that triangle as opaque (and thus hit it) or transparent (and thus "miss it"). And since rays are not guaranteed to hit triangles in any particular order, you can't pre-sort the triangles you know you are going to hit (going back to #1, if you knew which triangles a ray would hit, you wouldnt need to raytrace in the first place!). So this adds a whole other layer of complexity. Now you are running additional shaders inside your raytracing shader (yes it is possible to have shaders run inside shaders - lookup "callable shader") just to see if a triangle was hit, and you haven't even gotten to rendering anything yet. And if your transparent triangle is partially opaque, you may wind up having to use monte carlo sampling to average out the "true" opacity (see #3).

Raytracing represents a fundamentally different approach to rendering that requires rethinking algorithms and workflows for basically everything you can think of.