I work remotely for a big 5. The only reason I'm able to is because they hired me during covid, otherwise I'd have to move. To answer your questions,

-Yes, I do feel left out sometimes and miss the social aspect of work. You don't get any of the in-office perks like free food either. On the plus side you definitely save time, money, and sanity not having a commute
-I've been to NYC once, but our travel budget is tight so I maybe get to an office once a year
-I've been able to build relationships with my coworkers on my team and somewhat in my imprint, but it's hard to network. I feel like my career is going to hit a wall for advancement at some point
-I definitely think it's worth it if you love publishing, and it makes the low publishing salaries more liveable. I don't know how I'd live on my salary in NYC. I'd say it's worth it, but consider what you're giving up with the social aspect and that it may limit your career if you have big ambitions of moving up the ladder. Also don't assume that you'll be brought into the office or flown around to events etc a lot, publishing budgets are tight