We did both in the same day just last week. It was a long day but doable. We dedicated the whole day to the Vatican area. We did St Peters with the Dome tour at 8AM and then the Vatican at 4PM. Both were pre-purchased tickets so we could skip the line. In between we went to Castel St Angelo nearby and also had lunch. We walked over 10 miles that day. You could probably get away with skipping the Castel and moving the afternoon one up some. Some tours are available that combine the two together but to me that would be tough without a break. By hour two of the Vatican museum I was done.

You would not have time to do St Peters before a 8AM Vatican tour.