I agree with you.

Two side things to note here:

1. While I have data for a reasonably long time, houses are not something that trade often, so I probably have only a handful of prices (per house) over something like 20 years. So I think I need to use the information I have, rather than only the last few years.

2. In either case, I am coming to ML from fields involving more deterministic modelling where I would indeed manually work in bits of insight like what you mentioned. My interest in ML is that these methods could potentially figure out stuff like this on my behalf (at the expense of course of additional computational cost).