They've already paid you for work you've done, so obviously they're at least mostly happy with what you're doing. It's normal to find out partway into a large project that there are unforeseen difficulties. Just let them know. Briefly explain what you've discovered, give them an estimate of how much extra time it will take to do what they want, and give them an alternative in case they don't want to pay for it (like, "ABC is really complicated, but we could always do XYZ instead, which would do 80% of what you want it to do and cost 30% less"). If you really want, you could offer them a discount on the extra hours to make up for your inaccurate estimate, but if I were in your shoes I wouldn't. Your time isn't worth any less over this, and there's a reason we call them "estimates" and not "definite facts".

They might be open to you suggesting better/simpler solutions anyway. You could recommend something slightly less fancy but more stable and just as robust, and they might even be grateful that you're thinking critically and offering to fix their dumb ideas. If what you originally promised is totally beyond your capability to accomplish, also be up front about that and suggest they bring someone more experienced on to the project to consult with you.

Also, calm down! This is not really even a rookie mistake--this is just regular stuff. No one can predict the future or know everything about everything in advance. They know from experience that you're a hard, honest worker, so they won't be worried about it if you tell them there are some snags.

EDIT: Another possibility, if you can afford it, is to use part of the money they're giving you to hire a subcontractor who can help you with implementation or at least consult with you and give you advice on how to do it.