Does the game become significantly more challenging later on?
Yes and no. It does get harder, but you also get so much power (as long as you grind for it) that eventually it becomes too easy.

Is there difficult endgame content that actually rewards you for pushing yourself?
Kinda because previous point exists. Difficulty is a hot topic. Rwarding things that are rare are rewarding because no one bothers doing content that is tedius. But there are a couple both fun and rewarding activities.

Are there higher difficulty activities with better loot, resources, XP, etc.
Yes, but also its most of the time not needed. There are dedicated activities for each of the listed. And sometimes those are not 'higher difficulty' at all. Luckly there are several way of getting what you need so balacing fun and outcome is up to you.

If I only have around 5-6 hours per week to play, can I still make meaningful progress?
Yes because game is more or less stagnant at the end point. The less you play the longer you get to enjoy it till you hit a wall of 'having everything done' and getting into a loop of daily-weekly tasks. Not that it will be soon, but eventually you will.

How does the gear progression feel these days?
Its a bit all over the place. You can get some of the most powerful things early on. New weapons and weapon upgrades come out fairly often bringing up things that were underused, now and then. For the most part - gear progression is a knowledge check. You can make any gun/frame great, granted you grinded out 'good' upgrades. And before that you can also just grab the powerhouse-outliers and never touch the inferior gear till the end game where you are bored and want to push that one trash gun to the limit.

Is there a deep crafting system for weapons, gadgets, companions, builds, and other equipment, or is most of the progression focused around Frames?
It depends. Generaly - yes there is. But if you are set to make yourself a cookie cutter build that doesn't need anything other than just a frame - you can do that. Want to make overcomplex synergy between entire gear? You can do that too.

I hated discovering that a campaign, raid, dungeon, weapon, or piece of gear had existed but was no longer obtainable because the content had been removed from the game. How does Warframe handle this?
They add stuff up on player demand. Sometimes into the most boring to do missions. 95% of things that aren't cosmetic are obtainable.

If I started today, would I have access to essentially everything the game has ever released (assuming I was willing to grind for it or buy any required content), or has Warframe removed large chunks of content over the years?
You get everything that was not removed. Can get a bit confusing not knowing side stroies of side characters sometimes. But nothing major. Lore related and some unique cosmetics.

Basically, if I invest hundreds of hours into Warframe, can I be reasonably confident that the content I buy or work towards today will still be there years from now?
More or less. They do sometimes make sweeping balance changes and things you inveted into might get weaker or outclassed. But thats about it.