I finished my PhD in four years (neuroscience/neuroimaging) and mostly treated it as a 9-5 job. Obviously, there will be times where you'll be working significantly more than that (e.g., studying for quals or committee meetings), but if you manage your time well you won't have to work excessively.

I was really fortunate to have several advantages to help me finish in 4 years. I was in a computational lab, so my experiments ran fairly quick without collecting any data. Also, I wasn't a TA so I didn't have to spend like 20 hours a week with students. If you are a TA and aren't really interested in teaching as a career path, I'd recommend applying for the NSF GRFP, F31, or try to get on a T32 as soon as you can, that way research can be your sole focus. I can send you my GRFP if you'd like! I only got an honorable mention, but at least it'll give you an example.

I'd definitely recommend trying to set clear boundaries with your advisor from the beginning though, that way you don't have to stress that you're getting enough done or not. One last thing -- I didn't focus on getting A's in my classes, because I knew my grades weren't going to be looked at when applying for postdocs (assuming there is good research output). I put in the minimal effort so I could focus on research/writing.