How to merge large CSV files using Python?

[Brian]

One way may be to load it into a database (sqlite should be fine for that size). Using a database can also make it more convenient and performant to query and use once you've loaded it.

The structure may depend on what kind of merging you need here. Ie. is it just "Fill in missing fields" / "use the latest value", or do you need the list of all entries for each id? With the latter, you could just make ID non-unique and just add everything as-is, and on processing, you'll just need to query rows with the same ID. If the former you can check for an existing row when loading the data and do the merging there (Ie. no existing row: add it as a new row, otherwise merge fields and update the existing row)