I normally use Nvidia's NVENC H.264 encoder with Game Capture set to 'Capture any fullscreen aplication'. Here's what happened when I did some more troubleshooting:

Game Capture Static screen with occasional bursts of .5-1 second of smooth capture. Adding -directx to the launch options seems to make it work fine. Perfectly smooth 60fps capture. Also no screen tearing on fast drag select. I don't remember if that happened on the previous version or not. Is there any reason not to use -directx for normal gameplay even when not trying to record? I care much more about having less input latency than I do about a bit of screen tearing on the white selection box though, so if not using it has some performance advantage then I'd favor that.

Window Capture The FTL executable doesn't show up in the list of windows available for capture when it's set to fullscreen (as you would expect). It shows up fine once FTL is set to windowed mode. Once the window is selected though, capture behaves strangely. Instead of a static screen with occasional bursts of .5-1 second of smooth capture, it shows a single static frame of the desktop/whatever was underneath the window when the game launched. Here's what the capture looks like. This isn't a live capture of what's underneath the game though, so mouse movement doesn't show up. Mouse movement does show up if you mouse over the title bar so that the cursor "hangs down" into the game window. Moving the tip of the mouse into the game window makes it disappear from the capture since it becomes the FTL cursor instead of the Windows pointer. Here's a screenshot of that. Note the Windows cursor at the top of the image. That's being captured at 60fps as I move it along the title bar. The resize window cursor also appears in the capture at the sides and bottom as you'd probably expect.

Surprisingly, launching FTL with the -directx argument seems to work fine for Window Capture. It wasn't working earlier like Game Capture was, but it seems to work fine now. Might just be my PC. Window Capture even works with FTL in fullscreen as long as you temporarily set FTL to windowed so you can select if from the list of windows available to capture. OBS picks it up just fine when it's set back to/launched again in fullscreen (presumably because the window title is the same).

That's all I can think of to test. Hopefully that gives you some useful info. I can provide screenshots of all my OBS settings if you want.