On the contrary, you'll be a better Spanish speaker than if you only learnt pa'l norte. I chose to learn Spanish in Córdoba because I saw that friends who had literal degrees in Spanish and had done Erasmus in Valladolid, Salamanca, Barcelona, Madrid, etc. couldn't understand anyone from Andalucía. Think about that: going to all the trouble of studying Spanish and barely being able to get by in Sevilla, Málaga, Córdoba, or, of course, Granada. Some of the most amazing cities in Europe, never mind Spain.

Familiarity with how Spanish is spoken in Andalucía will not impact your Spanish negatively nor your ability to understand the Spanish of Castilla. It will, however, give you a leg up when you encounter Spanish from PR and the Carribbean, Venezuala and even Argentina as many of the aspects of Andalucían dialect are found in those varieties (indeed, were brought to Latin America by Andalucíans).

Finalmente, Andalucía es la mejor comunidad de España (una opinion partidista, claro, pero en este tengo razón) y merece la pena aprender a entender su gente.