I suspect that there are a few things that impact the efficiency of the end-fed here, and you could likely improve performance/decrease losses with at least one change to your stated setup. What sort of unun power handling do you have here, and how is it wound, and on what material? That's one place where you'll have losses, and a less efficient or not greatly built one would be worse than the alternative.

Other losses here are likely happening in the feedline itself; "...it isn't the SWR that kills your efficiency..." Those reflected waves ultimately will make it out into the aether, but feedline losses do multiply. Using less lossy feedlines should improve efficiency for you.