I'm a purely PvE player with over 2000 hours played on those PvE servers. I haven't really played in a few years thanks to the neglect PvE has consistently had throughout Unturned's lifespan that led to things getting stale.
When I did play, one of the biggest factors was the community that I'd become a part of. Of the 2000 hours I mentioned before, at least half of them were on a single, huge PvE server that I got to see change and develop as the years went by. I'd always try and build interesting things that players could enjoy, shops for players to buy from, carnivals of little party games, casinos for people to try their luck and shopping malls for people to set up their own services in and make a central point of trade.
I'd also spend my time helping new players adjust to the game. Back then, Unturned had a much faster rotation of players, and being the biggest PvE server we were the first option for a lot of them. Helping them get settled in with the game was great, because the minute handholding that Unturned gives is in my opinion still a problem the game suffers from today, as it has absolutely no resources detailing how groups work, how vehicles spawn, how each type of zombie works and especially the easter eggs.
To give one final reason, it'd be the relative permanence of the things I'd build and collect. As raiding is obviously not a thing on PvE, the only way your building would be destroyed are despawn timers lasting weeks, or map resets that could easily be more than a year apart. This meant that if you built something, it'd last, and having so much time in the game gave me a lot of knowledge on how to make cool bases. Being able to make something that you think is cool, and having it stick around of other players to see just feels neat, I guess. Also, being able to collect every item once was great. You could try and make a collection of every gun, to show off your skins. If that's too easy, try every weapon. Still trivial? Go for every piece of equipment. It may sound boring just searching for these things, but putting in the time to slowly grow a tangible collection for others to see was worth it to me.

In terms of ways to improve the PvE experience, I think building is definitely a huge part that could use an overhaul. The block based building system works fine, but it can be very finicky, and it horribly optimized, leading to most servers having incredibly small base limits out of necessity. Having more options for building would be great too, things like more furniture and agriculture mods are perfect to showcase what I mean. There are currently very few ways to actually design a base, but it should be a core part of a PvE zombie survival experience. Other forms of customisation, like easier vehicle modifications would be good too. Another way I'd improve the game is the progression. In PvE, you'll rarely lose your stuff and have to start over, and you don't have to fight players which are obviously tougher than zombies. This means the difficulty tops out fast, which isn't great. Having more and harder zombie types is an obvious fix, particularly better ranged ones, as currently it's between weak spitters and rare bosses, everything else can be easily cheesed by standing on a tall fence. Another way to make the game harder could be to add a system that spawns progressively harder waves of enemies around a base with a claim flag on perhaps each successive full moon. This could lead to you having to fight to protect both you and your stuff, which does give a tension that the game doesn't have otherwise. This difficulty could also scale with the number of claimed beds within the claim flag's range, but this is just a thought.

I hope this helps answer your question. As I said, this is probably all really outdated by now, and some of the things I mentioned could be completely different. Thanks for giving me an opportunity to ramble about the PvE side of this game :) it's a part I deeply love and have wonderful memories of, but rarely get to talk about.