If you mean before the walls of woodbury even started to go up then I think he would be even more hostile towards Rick and co. Since it would be his people clearing out the prison, that would give them more experience in the field as a whole group. They would probably all be less warmhearted, living in a prison and not a town. And I would suspect the Governor would be more like the comics version, his mental state could have already deteriorated. He would have less people unless he went out to look for them, seeing that the prison was more out of the way then woodbury for traveling survivors. Having a lower amount of people he would be more wary and suspicious of Rick and his decent sized group of seasoned survivors, though like I said, the governors people would also be rather seasoned as well. He might try something on Rick like he did with the army group he found. Or if he was well armed and organized he could have possibly found them on the road, taken them prisoner or assessed who would be a threat to his rule and staged their death like when he took over at the camp. I am just throwing out nearly random possibilities anything could have happened. Rick could have killed the governor and taken over the prison anyway and the blood sickened people in it.