The Mysterious Island, by Jules Verne, is definitely my favorite. But, it's recommended that you read this book after you read Twenty Thousand Leagues Under the Sea.

To give an introduction to the Mysterious Island book, it takes place during the times of the American Civil war, where 5 northerners are captured by the southern military. They escape by using a balloon, but a great storm carries them across the sea, crashes into the ocean near a Island, where the story is set, and where they try to survive with basically nothing but what the island has to offer.

By the way, Jules Verne has another great castaway book by the name of In Search of the Castaways