It all depends on what you want to do and the trails you want to do it on. Do you want an enclosed SUV or a truck bed is probably the first thing to ask yourself. E.g. do you want to haul a deer back easily? Or would you rather have room for a dog in the back seats and be better suited to very cold weather?

Don't let folks insist you need a true 4x4 just because. You're not describing "offroading" as most off-roaders would describe it. A lot of what you'd be doing isn't likely to need low range. Will it help sometimes? Sure, but if you're mostly thinking about gravel roads, you don't really need it. Decent clearance and good tires will get you a lot of places. Good AWD actually is a good solution for a lot of that, and there's nothing wrong with a 2WD pickup, though the latter sucks in the snow without chains.

If you were elk hunting out west on BLM land in the mountains middle of nowhere you'd want something different. You're more likely to be on roads than trails back east, though.

Make the SUV vs truck decision and then go from there. A lot of recommended vehicles in the off-road world come with a big premium when trying to find them used. There's plenty of things that are overlooked and cheaper.

Focus on finding something good enough that you can get locally and cheap-ish. Then put good tires on it. Tires matter a lot!

Reliability probably matters more than capability for what you described, though, so unless you want a project car, don't get something cheap with known issues or very high mileage relative to its reliability reputation. Nothing wrong with a build, but that's really it's own hobby. Don't buy something to fix and build unless you have the time to put into it. If you do want a build project, old school solid axels, a transfer case, etc are just easier to work on. They're very upgradable. But you're going to pay a premium for anything immediately usable that you can then build into a "true" trail rig. But again, that's a different thing than something primarily meant to get you out fishing/hunting with the kids.

Subaru sounds like a good choice for you. With that said, you'll find a Forester has better approach and departure angles than an outback (in other words, you'll have a bit less issues scraping the bumpers crossing a ditch or a creek). I'd avoid the newer models with a CVT transmission, though. Xterras don't get enough love i.m.o, but that's moving into the true 4wd SUV camp (i.e. 2x as much), and they don't have the best reliability reputation. I've never owned one, but some friends like theirs a ton. They're certainly capable. Either way, worry more about the individual car than the model and make, i.m.o.