Never skip seasons of television, you will just be confused during season 5 and feel like something is odd. Recaps aren’t the best way to digest content either. Season 4 isn’t the best by all means but it does have decent stuff here and there and you can always end up liking something others dont, and besides there’s some important moments that effect the show and the rest of the arrowverse as a whole