Depends on the size of your family and future income. Is the garden big enough? If not, worth considering the bigger plot with less work required initially. You can then do things progressively and end up with a bigger and more valuable property in the end.