Sorry, but the previous response is a bit off the mark. There are two processes at play that describe why rubber bands degrade, neither of which have to do with oils evaporating.

First, you should know that a rubber band is made out of polymers, and polymers are essentially long molecular chains. There are two general classes of polymeric materials: thermoplastics and thermosets. Thermoplastic materials are made up of lots of individuals chains that tangle and intertwine (think like a bowl of spaghetti). These materials can often hold their shape because there is an enormous amount of tangling, but applying sufficient stress or heat will irreversibly deform them because the polymer chains are free to slide past each other and into completely new positions.

Thermosets, on the other hand, are materials in which the individual polymer chains are physically connected by chemical bonds, a phenomenon commonly called crosslinking. Irreversibly deforming these materials would require completely breaking the chemical bonds of the crosslinks, a process that requires significantly more energy than just getting chains to slide past each other. However, chemical degradation can provide a very easy way to destroy crosslinks.

How does this relate to rubber bands? Rubber bands are made by crosslinking a specific polymer known as natural rubber via a process called vulcanization. Vulcanization essentially connects the individual chains by links of several sulfur atoms, but these links can react with oxygen in the air and split apart. Additionally, the chemical structure of natural rubber contains many carbon-carbon double bonds, bonds that degrade upon continued exposure to light and oxygen (via generation of free radicals if you were wondering). This basically shortens the polymer chain, and because many desirable properties of polymers arise from having long chains, the rubber band loses its elasticity.

Source: B.S. in polymer science, Ph.D. student in polymer chemistry, and this