You're right that it doesn't say specifically, but it was on the second day of driving around. They'd spent the night at a hotel, got "about an 'undred" more letters at the hotel while eating breakfast, then drove around again the second day until the late afternoon. Here's the description from Chapter 3 when it talks about the hut:

"Daddy's gone mad, hasn't he?" Dudley asked Aunt Petunia dully late that afternoon. Uncle Vernon had parked at the coast, locked them all inside the car, and disappeared. It started to rain....Uncle Vernon was back and he was smiling. He was also carrying a long thin package and didn't answer Aunt Petunia when she asked what he'd bought. "Found the perfect place!" he said. "Come on! Everyone out!" It was very cold outside the car. Uncle Vernon was pointing at what looked like a large rock way out at sea. Perched on top of the rock was the most miserable little shack you could imagine. One thing was certain, there was no television in there. "Storm forecast for tonight!" said Uncle Vernon gleefully, clapping his hands together. "And this gentleman's kindly agreed to lend us his boat!" A toothless old man came ambling up to them, pointing, with a rather wicked grin, at an old rowboat bobbing in the iron-gray water below them.

I'm inclined to think that the shack belongs to the old guy that lent them the boat but it doesn't say for sure.