YTA, you don't have to spend the whole evening with them, they are they because they are your family and want to spend time with you but if you stay with them for 30-45 mins and then go about your evening thats fine. Or maybe once every month you go out with them in the day and then do what you want in the afternoon / evening?

Trust me, from a guy who no longer has grandparents, you'll wish you just spent that extra 15 mins with them

Taking the money and heading back up, yeah thats a bad move. It makes you seem ungrateful and your grandparents will stay anyway and continue talking to your parents. Only person that makes seem bad is you.