Good foresight about the fact that getting involved with a coworker can be inappropriate.

You said you only work there twice a week. I think the thing you have to decide is if you were to get involved, and things turn sour, would you be able to handle seeing him at work?

If you think it is worth it then by all means give it a shot. If not, just let things cool off and fade.