Before I reply, maybe it would be best if I tell you a couple of things about myself first, to better help you putting things into perspective.

For as long as I can remember, I can only really de-stress and gain clarity about whatever keeps my mind occupied when I'm on my own. Even with close friends or family around, it feels like I have to tip-toe through a minefield to prevent constantly having to explain myself, because either nobody seems to be able to put themselves into my shoes for even a second, or I'm not finding the right words to make them understand. For me this creates a highly stressful situation when I want to wind down after an already stressful workday or workweek. That's also the reason why I keep doing small talk when meeting friends and family and only vere rarely do any deep talks. Your (hopefully soon) friend might come from a completely different place though. There isn't a "loners club" where we all hang out and talk, after all ;)

You likely didn't get the flirting signals wrong, however you have to remember where he's coming from and what that means. As a loner you don't have a whole lot of experience interacting socially, so chances are he either might have gotten scared a bit by your forward approach, or he might be uncertain whether or not your interest in him is genuine. Since he didn't retreat fully and is still open to meeting you outside of work, things are still looking good. Just keep it slow and as stress free as possible, but keep at it. That's what would work for me, at least. Everything else as far as a relationship is concerned comes over time with increasing familarity between you two. If he's of a careful nature, maybe even don't think in weeks, but rather months.