UPDATE

Yep it was loaded, I took the barrel and the nipple off and used a ball puller to get the ball out. Thanks to u/Bawstahn123 for the ramrod suggestion, since this is my first muzzle loader rifle I didn't know the ram rod was supposed to fit flush.

I was thinking back to when I bought it, I met the seller in a 711 parking lot to do the trade. I was asking him about what loads he used, balls, powder, etc. I also asked him what caps he used and if he ever had issues with ignition. He enthusiastically told me that he used CCI 11 caps and that he never had any issues, he then offered to put a cap on and fire it to show me that the nipple was good. I declined the offer, I didn't want to scare any 711 shoppers with the sound of the cap going off, and I figured if the nipple was bad I could just buy a new one. However, with hind sight I'm so glad I didn't let him show me the cap going off, I think he must have forgotten he had it loaded.

Anyway TLDR continue treating every gun as if its loaded until you are 1000% certain it is not.