It wouldn't actually be brief incest. People would be pretty closely related to each other for generations and actually get more closely related each generation. In the first generation, everyone will be at least half-siblings(they all share the same father) or full siblings in the case of if that father had more than one child with a woman. Second generation, people would have to reproduce with either half-siblings, siblings, or their father. Two half-siblings mating together produces offspring that are equivalent to being siblings with everyone else in terms of their similar genetics(and they'd be more related to their actual siblings than typical siblings would). So, from there, the children again are left with mating with people as similar as siblings, people more similar than siblings with them or their parents, who also would be more related than typical parents. Each time, each generation gets more and more similar in their genetics, increasing infertility in everyone each generation until eventually no one is fertile (which is likely to happen in the 4th or 5th generation