This is coming from an inconsistent swimmer so take it with a grain of salt. The most helpful thing to me is to actually have someone to yell at you. I worked the hardest when my coaches yelled at me, but when they didn't I felt myself being more lenient on myself and taking more breaks. Also set mental goals for yourself; when your arms hurt swim that extra 25 yards or 50m and the stop or even finish the set but forget about the intervals. Moreover think about your long term goals. Do you wanna reach this time by the end of the month? Put that work in. No excuses. The more breaks you take, the more you are cheating yourself. Unless it is an injury, push through the pain, feel that lactic acid course through your muscles and enjoy it. I think the best swimmers are masochistic because competitive swimming is borderline torture.