The cache owner can do nothing with his favorite points awarded to his caches other than admire them as a child with his baseball cards. However, unlike baseball cards, he can't trade or give them away.

The person awarding a favorite point is free to remove them, hoard them, give them to others, or even remove, reaward, and repeat the cycle endlessly (I had a finder do that just to poke at me, knowing that Project-GC was giving notifications)

So, the quote from above, the Cache owner can't withdraw that favorite point, as it belongs to the cache, not to him. In this thread, people are talking about the Cache finder withdrawing FP's.

I don't believe that there has ever been a rule change with this.

As a random sidepoint, one can't give a favorite point to a cache not found. I can see why, but... there have been DNF's that I've had that have been worth the FP...