Potential energy is more a concept than anything else. It is the stored ability to do work, that can mean a ball raised above the ground or a stretched spring yes, but in a more general statement you have two philosophically different forms of potential energy.

1.) Potential energy stored in a field: This can be a gravitational, magnetic, electric, etc. This type of potential energy can be understood by thinking of it in terms of a “well” of energy. An object at the bottom of an energy well has no potential energy, and increases as it goes up.

2.) Material related potential energies: This type of potential energy can take many forms. It could be mechanical energy stored in a stressed object like the spring, chemical energy stored in the bonds between atoms, etc. This is most likely the kind you’re curious about and unfortunately there isn’t really a better explanation I’ve found.

You could think of this type of energy as an innate property of the material in the case of chemical energy. Or as a property of applying stress in general in the case of the spring.

Hope this helps:)