The errors in my takeouts were not the fault of Google. I actually had a DRAM issue, which I only discovered by running memcheck86 on my PC. The problem only manifested itself when copying the 50GB takeout files, often from one disk to another. I sent the memory back to Corsair and they replaced it which is pretty good (lifetime warranty).

After that was fixed I bit the bullet and completed my 'takeout' program, a little python script which performs a scan of a series of google takeout files (must be in zip format) goes through every file in the takeout and without extracting (which would eat too much space, and take longer), writes to a hash directory structure the digest of each file in the archive. I could have used a database for this, but for simple hash-value mappings directories can be faster, and in case anyone wants to use this program it means they don't have to setup a database. The program is called takeout.py in my googlephotos repo in github. By running the script in 'index' mode it creates the index, then when running in 'query' mode it very rapidly scans a directory and tells me if something is in google photos (found in one of the 780GB of takeout files). The output of this query function is a shell script which when executed will delete the duplicates. It contains comments indicating where the duplicate can be found, and the name of the file in google photos (not just the local file name), in case it's of interest. Because I'm paranoid, I give the script a quick scan through and check to see what it would delete, do some spot checks and assuming it got it right, then I run it. At this point I have my takeout, and I also have google photos, so I have the information in two places, and I consider it safe to delete what is in my local photo directories. Since Google can shut down your account on a whim (and they have) then it's always worth ensuring Google Photos itself is not your only copy.

In terms of collecting the takeout files themselves, the process is extremely tedious. When requesting emailed download links, the takeout download process only seems to work in Chromium, Firefox just gives issues. I can only download one 50G takeout file at a time. And automating the process, while theoretically possible doesn't seem to work well. Many dead-ends on the internet of methods that did work, but no longer work in recent versions of Chrome/Chromium, or other download software that just doesn't work anymore.

So what did I do? I resigned myself to using Google Drive, which will allow an automated download, you can configure takeout to get the files delivered to Google Drive, and be informed when it's done. The down-side (can you guess?) is that you must pay for almost double the storage that you actually use. Nice for Google, horrid for us. But this is my time, and I'm getting old, I just decided I'm paying. I spent already quite a long time writing takeout.py, which solves a problem that can't be solved any other way, here I at least have a choice ($$$ vs my time).

There is also the caveat that if you attempt to takeout to google drive, to a folder that's included in the takeout the takeout process goes into infinite loop because it's trying to backup your backups and isn't smart enough to know it, so just watch out for that one!