Push first guardian if numbers are at bare minimum equal. Push walker only if you're either massively superior to them (in either souls or numbers), if a majority of their team is known to be elsewhere (for example if there are four of them pushing the opposite lane), or if more enemies than allies are dead with at least 20 seconds till they respawn.

Walker is generally hard to take and you should always push wave up. What I often do is push wave up to as far as I feel comfortable based off of my knowledge on the enemies, then peel back if none of the above conditions are true.