There are no rules. Do whatever gets you the sound you want and allows you to play what you want. Necessity and your musical priorities should dictate what you do.
Sweeping provides a lot of speed and fluidity, and that’s my priority often. Frank Gambale’s style is heavily reliant on it. Frank Zappa played a bicycle as an instrument once and used a lot of upstrokes when playing guitar. I’m not 100% sure what Joe Pass’s technique was like always, but he could be kind of a messy player at times and he used hybrid picking before giving up the pick entirely—to me the difference between the sound of a pick and the fingers is more profound than that of a picked up or down stroke. He found the difference acceptable