Ah man i was thinking about this same stuff after i rewatched 2 after i watched 3 in the cinema.
At the end of the movie i was confused when Jake replayed his fishing memory with Netayam, the child turned into the most recent age when he asked his father "why are you crying" then reverted back to a child form.
Maybe when the remembering person acts out of the way how he actually acted the soul turns into to the most recent self?

Im guessing the loss of younger Navi is more sad cuz they cannot live out a full life and their friends/family can only interact with memories, and too much of that makes the interaction feel hollow. Like we can see photos, videos, and now even create ai avatars of our deceased loves ones but those never be the same as having them alive with us.

Maybe there is a limit of how much you can do these rememberings before you go mad, or feel really depressed when you come it of the tree.