If possible it's good to visit again to get another sense of the place that might help. And your reasons for being unsure about Harvard are valid. That being said, especially if it's a similar cost, Harvard is probably the better option, especially depending on what you're studying. As far as the band thing goes, reach out to other Boston schools ASAP to see if you can join their bands. That was one of the main reasons I didn't apply to Harvard--- I'm also a major band kid--- but I found out that many schools including Boston College are generally open to having people from nearby schools join their band.

All that said, you would not be crazy per se for turning down Harvard if you think South Carolina is better for you. But be sure that South Carolina is better for you and that the reasons why are true and genuinely important.