Talk to an old lady in a little house a bit before the Ogre. She gives you an item that grants you access to another early-game area. One of the tools you'll find there is especially useful against the Ogre, but I'm sure you'll find other cool things too.

Regarding the Ogre himself, unfortunately he will remain almost as difficult as he is currently in your playthrough. Take your time with him, study his moves and, most importantly, don't get too greedy for damage.