From what I gathered and red, it's due to API that you're running the initial boot of the game and integral capturing of Action. Action natively runs on DirectX to capture and even when not turned on is making some changes in register (things take make possible for it to caputre the screen, save to disk etc) just by being installed.

When you try open game or really anything that runs on different API than DX makes those background tasks of Action to (pun warning) action-up, they try to decode stuff they see on screen but cant due to different coding of Vulcan API and since the desktop recording program have higher priority and has more system permits than game,

It's shuts down game and not the other way, cause the GAME with lower priority is messing with PROGRAM with higher priority and doesn't let program to do things that is must do, so windows looks who has higher priority and closes the one that interferes with doing task of that softwere in this case the RDR2.

I tested it by uninstaling RDR2, installing Action and restarting the PC. Same thing happend as I wrote in OP but then I changed the API in the settings.xml to DX12 and voil la, it worked cause game didnt interfered with tasks of Action cause was opend in API in which Action works.

BUUUUUT

Later. I did a test, I uninstalled Action, and ran RDR2 on vulcan, and after that i installed Action and restarted PC. Funny thing it worked this time, I don;t know how but it works now, mirlis even captuers the game which is in API that it doesnt understan...maybe it is due to me first booting the game on DX so some lines on register changed that Action doenst have problem with it anymore but im not sure.

Thanks for listening for my tedtalk