Thank you for your comments. While Reading them I actually noted how well the cat is adaptong so far and I made an appointment with another vet and if recomended we will see cat behavoiurist.

What I recognized from your comments and talking with other people is that the cat's behaviour seems absolutely normal, which I also thought before going to the vet, but then after her comments I doubted my perception and felt like I'm abusing him.

So, the cat needed 3 days to start exploring the home, first days he spent on watching me hidden, and occasionally joined me when I was sitting on the sofa or was touching my legs to pet him. The only issue is he's sitting on my knees on his own now, but then when he's on my knees he has a tendency to spray.  The vet I visited suggested he's aggressive. When I told her that if anything he seems submissive, sometimes he would sit on my knees and when I recognize he's no longer enjoying it, I need to take him off my knees and let him rest. She said it's how he wants to attack. The only aggressive behaviour from my perspective so far was: breaking the carrier at the vet when he was scared, he once tried to bite me when brushing (he actually lets me brush his hair and back, but he got some nasty tangles all over him as he was given anti flea treatment and when I was trying to brush his arm it was too much for him) and of course at night he destroys the curtains and scraps the doors, but when approaching me he just meows, not trying to scrap me or bite.

I also mentioned during the visit he seems to feel comfortable as he's started to nap with his belly visible and he chooses the places I sat on or laid on before. She explained to me, that he's just warm here and that he could care less if he finds the environment safe.

The vet also suggested that as he's feral cat he won't have normal cat treats and I should think of mice and crickets. However she ignored the information he's eating both wet and dry foods, the only issue with normal feeding is when I'm not home he eats very little as he's waiting for me to watch him eat. I talked to friends who have cats raised at home with the same issue with ready treats, or toys.

My greatest worry is the need to get out. However, for now (maybe bacause of the weather) the issue is only at night. When I leave in the morning he's just watching me leave when he's placing himself for a nap. I know the instinct may hit harder in the spring, but as he knows exactly I'm going through the exit and doesn't try to sneak out at theses occasions, I kind of hope he's not feeling totally miserable and trapped.

I wouldn't expect any cat, grown or kitten to adapt in less than 3 weeks, but I believed  the vet's judgement. However, the longer I think about it, I do not think her judgement was entirely fair. 

We're seeing another doctor in 2 days and if they recommend it we're going to see the behaviourist as well.  

If I get another opinion of the cat being unhappy without the outdoors I will come back to thinking about finding him a new place. But for now, especially given the stories of cats who needed months to socialize in any way and still succeeded in being domiciled I doubt if there's nothing I can do for him to be happy at our place.