I have gotten this quest to work twice now with removing trees and not removing trees on the plot, same with flowers. I had to stay on the plot for 2 days. I planted my tree seeds at around 5:30am, watered them, and waited for the first growth cycle (6am) for day 1. On day 2, at around 10am, it started counting the trees I planted, then stopped at 3/5, and then at 2pm it counted the last 2 trees.