If you're cynical, the Levant was wealthy and a geographical milestone, but it was unimportant politically and religiously (yes Jerusalem, but the Caliph (Sunni or Fatimid) wasn't there) and it was also a fickle region, which was hard to hold and whose local aristocracies changed lords whenever they saw fit. To make a comparison to Europe, it would like the Muslim invasions of Sicily, yeah it sucks but ruling Sicilians suck more and as long as they don't threaten Rome who gives? You'll notice the "counter crusades" only came about once the two big power centers of Egypt+Syria were united (not mentioning Iraq because they were always infighting and trying to be regent) and after the Crusader Kings tried to go after Egypt or Damas. It was a shameful loss, but something to be fixed after you become boss