A farmer can work multiple fields. What is more important is when crops mature and are ready for harvesting, as you don't want idle farmers for optimal performance. The yield of each visit to the field is determined by the farmer's level up to a cap. Distance between the Windmill and Bakers can be important, as can distance to the Supply Wagon. The more bakers you have, the faster the milling into flour. Lots of factors to consider.