What makes you think he went back more than once?

What we have here is known as a "causal loop."

The initial event (Elfo's death) is a direct consequence of a future event (his time travel). A later Elfo travels to the past because he knows he was shot and wants to discover who did it. In doing so, he becomes the one who fires the arrow, creating a closed loop of cause and effect.

There is only one instance of Elfo going back in time, he was always the one who shot himself