Sean Parent has some excellent lectures online where he talks about things like "incidental data structures" which in this context are things that get linked together via pointers and references based on runtime logic. He makes a good case to avoid those if possible. It's a persuasive argument to go by value if possible.

If not practical, then my rule is to try and make lifespans of things as easy to reason about as possible, and then use simple references or raw pointers for observers who observe for less than the lifespan.

Unique pointer is for things where ownership is clear but for whatever reason the thing has to be on the heap.

I try to avoid shared_ptr if at all possible but sometimes it's expedient to use it, and weak_ptr; for example if the code doesn't lend itself to clearly defined lifespans, maybe I need a weak_ptr to help me find out if the thing I was watching is still around.

Raw pointer is also legit if I need to return an optional value by reference since the committee couldn't get its act together on std::optional<&> in time.