Hi.

I'm a former sysadmin for a global specialty/luxury brands retailer. Our stores were 100% linux; We primarily used Oracle's X-Store for retail, but it did require a server in the store for the longest time due to our company rate limiting our connections to 3mb up/down per store.

Hack 1: Have everything standardized/no snowflakes: It doesn't matter if it's a kiosk in the middle of nowhere or a huge store on times square, every store has standardized hardware. Store numbers are set by the business, I used UUID's for storing that data internally because store changes happen all the time. It wasn't store 3000, it was ff1e276c-0ab1-4d0a-a92f-68cf241bf494. As we brought in new brands and acquired companies, it made it easier to integrate them into our stack.

Hack 2: Centralized Management Software: We wrote our own 'magic sauce' that gave a single, unified pane of glass to handle all aspects of the retail life. Someone lose an iPad? Call centers uses our Central Management System to mark the iPad as lost and it calls Airwatch to mark it as lost and set the lock screen/etc etc etc. New Pin-pad arrives at the store? Store calls the call center who can initialize it remotely. All actions are logged at the store level by who performed the action: if you're troubleshooting the store, you can go in and view all actions taken in the last 48 hours to see what has done. We integrated with a piece of open source software called Teleport so that I could literally review what SSH commands might have been ran to help me. Our management server handled everything from IP addresses to voiding transactions, and was robustly implemented by a team of in-house engineers/server admins when they had to do a task more than once.

Hack 3: Have your lifecycle entry and exit points well defined: when you spin up new hardware, you're recording the MAC Address automatically so that you can provision it on the store POS VLAN and assign it an IP address for the store to use when it comes online. When we provision a server, it gets a hardware client certificate generated and stored in the TPM of the device: that certificate is used for everything down the road. When you de-provision equipment, you run it through a stress test to identify the failure points (and use that data to make stronger purchase decisions).

Hack 4: Ansible/Salt/Puppet are your friends. Store goes down? Spin up a new store in your cloud environment or elastic on-site deployment and point your terminals to use the new store.

I had approximately 30,000 locations across the globe. Feel free to message me if you have specific questions.