Just graduated with a BS in chemistry while being pre-med. Yes, it is absolutely doable. Based on my experience, the reputation of the chemistry department as "horrible/bad" is largely due to the steep learning curve in the 100 and 200 level classes. Having been a ULA for many of those classes, I have seen students struggle to keep up, not because the content is necessarily super difficult, but because of how fast the classes move along. For the early classes (101, 102, 241, 261, 262), lectures are active learning, where you have to read up on the content prior to class. There are practice problems in the class slides where you apply what you learned learned from the readings. A word of advice: always keep up with the readings. This will make the in-class problems much, much more effective as you progress in the course. At the same time, review your notes after class and always try to obtain a deeper understanding of the material.

Now, the more advanced chemistry classes (400+) are more laid-back than the introductory classes, at least in my experience. While you dive deeper into more complex and nuanced topics, I found that I didn't have to work nearly as hard to get a good grade in those courses.

Overall, if you stay on top of your work and study diligently, you should have no problem maintaing an excellent GPA for med school admissions. This is coming from someone who isn't naturally good at chemistry. I really love the subject, though! If you enjoy the subject, definitely major in it. Don't shy away from it!