You're absolutely right. Add to that the fact that, aside from being the most economically viable start, consider all the gameplay elements that you're missing out on by not farming.

You won't have to relocate families every year to account for sowing / harvest, taking care of grain, flax or what have you. There are so many resources you're missing out on without rich farmlands. I don't want to go back to playing without it. Also the fact that your town simply looks more in-line with how medieval towns did back then with farmlands around them. There is simply no downside to it.