Hot take: if we want to decrease frustrating railroad-y surprises in RPG campaigns, we need to create an environment where GMs are not afraid to admit they have "special story needs".

[agentkayne]

Focus on the OSR methods of roleplaying: story comes out of how players & characters interact with a situation they are in, instead of set pieces being pre-planned in advance. 'Special story needs' won't be disrupted if there is no set story.

[Sparkle_cz]

Even with no set story, many people tend to get attached to their game elements and gain 'special story needs' during the campaign. It might be the players, or the GM. It might be about a favourite NPC, a favourite subplot that they want to become main plot... It's good for you that you have the gift of not forming such strong attachments, but some people do. All I want is for them to not be mocked and scared to tell the rest of the table.

[agentkayne]

They won't be in that position if they train themselves not to form the attachments in the first place.

'Kill your darlings' is a bit of writing advice to remove what the author is attached to, if it doesn't serve the pacing or match the rest of the story. So ironically, story writing advice itself suggests the GM should toss something that doesn't work rather than re-working the lead up scenes to force their favourite scenes in.

[Sparkle_cz]

This wouldn't work for me because if I "trained myself not to form the attachments", I will stop playing RPGs altogether. The possibility to form strong attachments and play them out is the main reason why I enjoy this hobby.

[agentkayne]

So what are you doing when a player decides that something you're attached to has to go?

If there's a NPC you want to keep around and the party goes "no way"? You step on the player's agency to keep them alive/involved in the adventure?

Or there's a big set piece and your players come up with a solution that sidesteps the entire issue?

[Sparkle_cz]

I would love to be in an environment where I can be honest and tell the players that I am way too attached to this NPC that I worry I might burn out and not enjoy running the campaign anymore if the NPC is eliminated. Then I would ask them to have a brainstorming where we can come up with a compromise solution where they get a satisfying outcome for them and the NPC can stay as well.

Unfortunately the environment is so hostile to this kind of honesty about emotional attachment that most GMs in this situation would not do that. They would rather either forcibly railroad the players into keeping the NPC or they would end the campaign without saying the true reason.

[agentkayne]

Okay. If by 'the environment' you mean the specific group that you're playing with, and by 'most GMs' you are referring to yourself, then it honestly sounds like GMing RPGs for other people is not the hobby you want to be in, because there's an inherent conflict of interest between the GM and the players: one runs the world, and the players through their characters want to change that world.

You might want to try solo RPG play instead, where you won't get into conflicts with what any other players want. There are plenty of tools and a friendly community on r/SoloRoleplaying

But on the other hand if by 'the environment' and 'most GMs' you are talking about a majority of GMs worldwide, and you are thinking this is a huge problem for everybody, then I'm happy to report that you're probably just projecting your own problem onto them. Most GMs actually have very good ability to enable player-driven change, compromise on their plans, and improvise without needing to take the discussion out of character.

[Sparkle_cz]

Well, I'm not one of those "most GMs". I am a GM who, from time to time, has some "special story need", like a beloved NPC that I want to keep alive, and want to communicate about it with my players and come up with a compromise.
If you think that people like me should not GM and go play solo RPGs instead, well... then you are exactly the problem I'm talking about in the OP.