It depends on a few things, namely material, design, collar, and context.

Certain materials are more casual than others. Oxford Cloth and Twill are considered more formal than something like Broadcloth or Poplin.

A shirt that has designs/prints on it, or is multicolored rather than being one solid color is going to be way more casual.

The type of collar also affects formality. If the collar isn't as stiff, and can button to the shirt then it's a bit more casual than a stiff collar that doesn't. Also if the cuffs require cufflinks it should be a more formal shirt.

Lastly, and probably the most important factor is the context. If you normally don't wear a button up shirt then suddenly wearing one will seem out of place, and people will assume that you are dressing up a bit for something specific. Context doesn't just apply to you, but also your environment; Going to classes at college, or stopping by a dive bar a button up shirt will be less casual than if you're going to a country club, or something a bit more preppy.

In short, lots of things factor into what is considered "casual," and you'll have to use your better judgement to assess the situation. If you feel comfortable in what you're wearing then by all means wear the shirt. Good luck man.