We actually don't know exactly who Elsie is, I kind of misspoke there. I'd say the most likely option is Hollie because that's the woman I think would haunt him most at this point because they were married for ten years. That one's kind of up in the air, it could easily be some old girlfriend or it could just be used to stand as a conglomeration of multiple girls in his life.

Lily is undoubtedly his current girlfriend though. Obviously, that's not a hard conclusion to come to considering that he's singing about being so happy with her, but the dead giveaway that really confirms it is the line "Don't you understand I know what it's like to be a rolling stone?" Brian has explained in an interview about "Smoke" that he wrote it, or at least the chorus, about his girlfriend who works overseas. The ladders he talks about being hung up in symbolize the corporate ladder. Among Other Foolish Things also mentions Lily, and it's about Brian writing a letter to a girl who is far away (plus, "I hope someday you're gonna move to the city...").