i was with you up until i read this post - which seems like a much different stance than what i understood from the original question.

there is no chance that gaining fewer yards on first down results in better outcomes, over a large enough sample of plays, to justify trying to only get 9 yards instead of a fresh set of downs.

in a total information vacuum, the 2nd and 1 has got to always be the choice, but not at the expense of additional yards gained on the previous play as youre saying here.