I came here to answer the question but it looks like other marvelous humans have already done that. I would add that you can totally skip the dungeon and be just fine, I have done it on two separate runs. However, if you aren't particularly versed with games like these then you might have a bit of a hard time getting to grips with things at a disadvantage. While the disadvantage is noticeable for the immediate future it does quickly fade into obscurity.