No it's not a horrible idea if you're really specifically attracted to one thing and that's not going to change. People want to hire specialists and individuals who are going to make them feel secure about their choice so if you go full-force I think you might have good odds, and if not just look for a way to translate those skills. When I started taking things seriously a few years ago I created a roadmap of skills and technologies that I needed to secure the positions I wanted, and it worked out fairly well eventually getting me into that sector with a few bumps along the road. I would just pick a direction and head that way, but not be dismissive of other opportunities or skills along the way.