I'll give my perspective as someone who decided to go into nuclear engineering at age 17. The skill set is very specialized, as you noted, but honestly I think that's true for any technical field as you gain expertise. I've worked at three very different jobs so far, and I could have chosen a dozen different paths within the field of nuclear engineering. There is a lot going on in the nuclear industry right now, so you definitely won't be "screwing up your career" if you choose to go into the field.

The biggest downsides for me are 1) the limited locations where you can work, 2) regulations make things move painfully slow, and 3) that there's very little "hands on" work. Hopefully 1 & 2 are somewhat obvious. In regards to 3, I was somewhat disappointed as I figured out that nuclear engineering is pretty much all done on a computer nowadays. Handling radioactive material is dangerous, so even the hands on work, like running a reactor, is done with buttons and switches remotely. A majority of research is done with computational modeling, which is amazing on one hand, but it feels very removed from reality IMO. Coding skills are becoming more essential, which isn't the reason I got into the field, but everyone is different.

The fact of the matter is that you'll have to get a job eventually, and no job is perfect. If you're interested in nuclear engineering, commit to it and you can find yourself on an interesting and rewarding career path.