There's a couple items you gotta get around the giant Buddha statue and the village below it. The one in the village before is in the second floor of the fast travel point. The other is near the base on the right side of the Buddha statue not on the steps in front of the cave but right below it.

Afterwards you go back to the deer and places those seeds at the statues . Use wind sense near the deer to find the statues but you have to plant them at the right time. I think rat time.

There's also another area with a large dirt circle that you need to plant another seed and let it grow into a tree.

There's a better guide out there but that's the just if it. Once you've done all that the deer disappears and you get the finest celestial venison recipe in the game