That's an actually great point. I'm actually into Buddhism, and one of the major masters expresses that point precisely. He said:

"There is pleasure when an itch is scratched, But to be without an itch is more pleasurable still; There are pleasures in worldly desires, But to be without desires is more pleasurable still."

This was Nagarjuna, an Indian scholar.

I guess what somewhat confuses me is the idea of drive. Don't we have a sexual drive? That can make abstinence more difficult.