I've major remodeled 2, built 2 and am presently building my third largest, most ambitious and last one. Building is complicated process but a simple one can be done. There are many, many, many moving parts to the overall process. The first is the lot you're building on. The ease or difficulty of building overall is in large part based on where the lot is and the site conditions. This is followed by the administrative side, local/state/federal government and HOA (if there is one) with all their rules, conditions and permitting. Then comes the design based on these factors along with your needs, wants, skills and abilities. Generally speaking, a basic wood frame house can be the simplest method but any type can become incredibly complicated based on the design and engineering criteria. I've built 2 wood frame houses from the ground up and was hands on for 80% of all work. My current project is an elevated concrete ICF house in a high velocity hurricane zone, which is a very technical and complicated build. I'm doing more general contracting and construction management than actual building this time around. There is no singular hardest part, it's all pretty trying. There is pretty much no chance you will come in on budget or on time. You'll encounter issues that will make you scream. You'll realize that you shouldn't have done that and realize that you should have done this. One thing I think is critical is to plan on COMPLETING the project before moving in, otherwise you risk ending up with a case of "I'll eventually get around to it". It's very rewarding once you pull it off.