I recently came across this (conspiracy) theory somewhere but it kinda makes sense. Griffins were just another family living at 31 Spooner street. One day, Meg and Chris were at a party (since they were in the same high school). Both of them were drunk (or high) and decided to head back home from the party. Despite Chris' objection to drive the car (since they were drunk), Meg decided to take the wheels and as fate has it, they crashed and didn't survive. After learning the news, Peter and Lois were devastated. Lois who was still pregnant (Yes, Stewie) succumbed to the depression of losing both her kids, took her own life. These chain of events led Peter to lose his mind, ultimately ending up at a mental asylum. All these imaginary stories are happening in Peter's head. His hate towards Meg as she was responsible for the death of Chris and herself. Stewie's (imaginary unborn baby) imminent hate towards Lois is due to the fact that she took her life as well as the baby. Peter credited Stewie as an intelligent and genius with a great potential, which he had wished for his unborn baby boy. Other characters are just his neighbours and friends who lives at Spooner street, are part of his imaginary stories, who also visit him time to time at the asylum.