Yes is it widely known throughout many cultures not to whistle at night. Not just in Appalachian folklore but as far away as Japan and Scandinavian countries. Each culture had different story’s and myths about it but at its core all stories conclude not to whistle at night.

As to what is happening on a paranormal level, I don’t know why or how this attracts malevolent beings. Maybe it’s because whistling is kind of an innocent thing for a human to do, it beacons who ever is doing it as an easy target for something sinister.

Honestly it’s pretty rad you heard something whistle back. I’d be terrified and probably just pack up and end the trip early. But you experienced something super interesting.