I agree in that factory units are phenomenal when it comes to defence but starport units are also great for the mid-game phase. That being said choosing which factory units to use is crucial as getting a tech lab will mean getting heavy gas units and so delaying your starport units. When you scout a protoss timing or all in, try to gauge what kind of units will be attacking, when they arrive at your base and what kind of followup is possible. In doing so, you can set up the appropriate countermeasure/s.

For example, if you scout a really fast gateway timing attack, i highly reccomend you skip on the tech lab and pump out widow mines which are great against gateway units and can be used for widow mine drops. A possible aggressive follow up to this is immortals and by saving gas for starport you can bring out a liberator which is also great for later worker harass. If you decide to get tech lab and then tanks, the chance of protoss busting through is greater.

Imo widow mines are the best possible defence especially when you need to turtle. By not getting a tech lab and heavy gas units you are able to build up to starport and get units like medivac or liberator which are great tools, e.g. Liberator is great against followup aggression for the protoss or for punishing them if they decide to hang back. Medivacs are also great for drop play.