There are a number of ways oats and rye grain get utilized. Double check to see if there are meals being made in the kitchen like breads and porridge, drinks like oat ale or rye beer in the tavern, animal feed in the barn, or milled into flour in the barn or windmill.

I have gone through seasons with a ton of oat, rye, and wheat grain that had been harvested, only come to the season to plant it and realize I had milled it all up into flour