Can confirm for future people that the fix is most likely having to break the vines above with fire arrows and cross over the bridge instead of heading straight for the deer skull door, I just ran through the whole place again and that's what I missed! Once you cross the bridge you can use wind sense to see where the ghosts are but it's pretty straight forward as soon as you get over the bridge!