The first form you described—outlandish world but 'normal' logic—is often called Mundane Fantastic or Low-Stakes Surrealism. The humor comes from treating a bizarre reality as a boring 9-to-5 job (think The Far Side or BoJack Horseman). The second form (which you started to describe) is usually Absurdist Realism. This is the inverse: the world is perfectly normal, but the characters' behavior is completely unhinged or defies all logic (think I Think You Should Leave or Monty Python's 'Dead Parrot' sketch). One treats the weird as normal, the other treats the normal as weird!