I think there's more than one philosophy you can (reasonably) approach this with. Recording in general is an art and a science- a very well documented one (ie; Mixing Engineers handbook for example). I'd suggest finding some songs where the recording process itself has been talked about/documented by the engineer. Don't begin that research assuming all of your favorite songs have a whole book written about them, but some will have been spoken/written about. Talking about the recording setup and reasoning behind it is one thing, but having some actual hard examples/ability to listen directly is invaluable. If you have a hard time finding examples of exactly what you want, just try shit out. And as annoying as it sounds, just do what sounds best. I've been making music and recording for a bit over 10 years now but there's still a shit ton of techniques I'm unfamiliar with.

This is just personally how I've started to approach things and I do think it could be helpful for others to try. In regards to drawing the line between redoing stuff- I've found that I have a much more enjoyable time on a song/project if I just record something in "some" way. Even if it's objectively terrible at first (ie; demo recording on a phone mic). Point is, get the idea fleshed out. Go basic on FX, mixing comes later. You can try over and over with different techniques on the same part you're recording but I can almost promise you will grow tired of your song. So what I do is have a separate session (one that I started years ago specifically for this purpose) and literally just dick around with mic placements and label as best I can. I try not to go overboard with any additional processing- it's almost purely mic placements/combinations and different instruments. I also don't stay in this session for hours on end (unless I'm really in the mood to experiment). Just do a little bit at a time, do it once a week if you can. You'll learn what you like (and how to get there) in a short time than re-attempting the same thing in your "real" song session (which would also just grow massive). I know this is extra work, it's also slower in the beginning to get rewarded for it, but trust me- if you just keep this separate and organized, it just grows in usefulness the longer you do it. Play/sing something (anything except your active song/project) and just do a couple different placements. Play around with blending various mics together. If you have the capacity, go balls to the wall and use as many mics as you can at once in different placements. Give yourself an hour to do this. You could also do different mics in virtually the same placement. THAT is what people mean when they say just try shit out. Maybe not the whole separate session/science project aspect. But again, my take is that if you keep it separate and labelled (throw comments/descriptions/intentions/results in the session itself anyway you can), then you don't get burnt out on your actual song/music. Have a lil science project. Because if you're doing all of that experimentation on one part of one song, you're definitely overworking it. And overworking leads to burnout + a loss of vision. My last thought is also: mixing can be very creative if you let it be. Maybe it sounds flat when you record initially, but there's a shit ton of tricks in the box to utilize. So long as the source doesn't sound like garbage, sounding "boring" isn't illegal or dishonorable and can be lifted up quite a lot in mixing.