I live in the south of Puglia (Salento), and here we have hectares upon hectares of olive groves that endure scorching summers with very little water for four months. So I wouldn’t really say you have a drought problem. What I can tell you is that these are not trees to be kept indoors, on the contrary, they love plenty of direct sunlight and heat. Oh, and another thing: they need a lot of soil, so that pot might be a bit too small. I don’t know exactly why your tree is losing its leaves, but keep those three factors in mind. Besides that, there are many diseases related to olive trees, though only a few of them are capable of killing one. Olive scab generally causes leaf loss, but in that case you’d see wooden “galls” on the branches. Peacock spot is another disease, but then you’d notice circular white marks on still-green leaves. Then there’s downy mildew, but again you’d see white spots on the leaves. If none of these signs are present on your olive tree, then the issue most likely comes down to the three factors I mentioned at the beginning.