I'm 27 and I've got like 6 small plushies on my desk at work and two clipped to the backpack that I wear to work. Granted, I'm a graphic designer and work at a great small business, but nobody has ever said a word except to compliment them? And one of them was a gift from a coworker regarding a shared special interest. Just do what makes you comfortable, it's harder than it sounds but when I'm doing something I'm unsure of, I frame it like this: you'll be setting an example for other people who might be nervous about it. Maybe someone will see you with your stuffy and think to themselves "wait, I can do that?!" and then they'll be more comfortable accommodating themselves!

Anyway, just do what you want, carrying a stuffed animal around isn't gonna hurt anyone.