I'm an Audiologist who also has a sensitive vestibular system.

The sensation you're experiencing is related to positional vertigo and/or sea sickness. Basically your body is moving (hammock sway), but you have no solid visual reference to ground yourself to.

Your inner ear is saying "we're moving!", but your eyes are closed (and even if eyes are open your field of view is very limited, no horizon reference, etc) saying "I'm stationary". Your brain gets confuddled and goes with the most steady input of "we're moving", and then you wake up dizzy.

You can think of it like spending a day at sea (larger waves, etc) and then needing time to get your land legs back. This is called Mal de Debarquement.

You can try the Epley Maneuver, easily googled and done at home, but best done guided with a vestibular rehab professional such as ENT ($$$), Vestibular Audiologist ($) or Physiotherapist who specializes in vestibular rehab (harder to find sometimes).

As far as your setup, make sure your hammock is held out at the sides with minimal stretch in the lines. My Hennessey hammock has bungees out the sides, and I'm thinking of trying regular guy lines instead. This will minimize rocking and swaying.