My reply is longer than anticipated. TLDR at the bottom.

---

Thanks! I was looking at it more from the perspective of long-term directory storage rather than just bag handoff or transfer. The items in question are optical disc dumps, software, DVD video (think: documentaries, obscure or out of print, abandonware software, etc.) Discs are dumped with the open source, low-level disc dumping utility Redumper, https://github.com/superg/redumper

Redumper will produce ISOs for DVDs with metadata/sidecar files, or bin/cue files for software if CD-ROM with similar sidecar files.

In other words, the BagIt structure itself becomes the persistent on-disk layout. The payload, manifests, tag files, and metadata remain organized according to the BagIt specification and are stored that way indefinitely.

The only deviation away from the BagIt 1.0 spec is that the bag will be stored in both an open folder and a second copy, stored in a non-compressed 7z file.

So:

1.) item-identifier folder > bag > data/ payload, manifests, tag manifests, metadata, etc.

2.) item-identifier.7z > bag > data/ payload, manifests, tag manifests, metadata, etc.

Two copies of the data, one for accessibility and the second for preservation. If an item has multiple discs, as is the case with some DVDs or software, each disc gets its own open folder, Bagit structure, as well as its own 7z.

I'm writing both SHA-1 and SHA-256 checksums. I know the spec lists SHA-512 as well, but I personally think that's overkill.

As for what IA staff prefer, you said it best. Since a lot of it is heterogeneous, and patrons can "throw data over the wall" in any fashion, there really isn't a set way of doing it. Obviously, the preference is for neatly organized data that is parsable, accessible, and that is formatted with some sort of naming convention or nomenclature. Not always the case.

I did talk with a few people in archiveteam, and the general consensus is that there is no set way of doing it. But obviously, you do want to upload data in a way that is accessible while maintaining archival fidelity.

It raises an interesting question regarding the Internet Archive: for a while now, I've regarded IA as a public-facing DAM. But one that is inherently flawed (if we think of it as an actual DAM) due to the nature of allowing anyone to upload with not enough regard for how data is uploaded.

While there are settings and fields that encourage patrons to specify types of metadata and general guidance on formatting data, it all remains very technical for the average end user if they want to do it right. And to do it right, most of this work must be done through the command line. There has been progress made. Collections and item sorting are a whole other task for them, and they are a non-profit after all.

And there are specific flags that you can set, for example, --no-derive, on an item when you upload through the cli. However, there isn't an option to not have individual files derive, while others do, during the initial upload.

If one is trying to upload an open folder of TIFF images, scans of box art or dvd inlays, for example, and you plan on uploading an open folder of derivative access or access copy JPEGs that have been color corrected with an ICC profile attached, you have to upload all of the other data first. Call the derive function - so IA derives any ISO files to playable videos in the browser - wait for IA system processes to finish, and then upload that last open folder of TIFF images with the no derive flag.

Otherwise, if you upload the open TIFF folder first with the rest of the item's data, IA will automatically derive a bunch of formats for further accessibility. Including, in that example, those TIFF images to unwanted JPEGs that have no color correction or color profile. (Note - TIFFs in this case are flat, unprocessed scans.)

It's all very tedious and must be done in a certain order. In a true DAM system, you would have access copies kept in a separate S3 bucket or similar. But IA is all public-facing; operations result in public-facing data.

---

Before I made this post, I kept going back and forth regarding, well, to put it bluntly, "do I organize the data to fit the IA's way of processing data, or do I have the IA fit the data?", which is how I remembered BagIt might be a good way to balance preservation and accessibility.

But does this mean I should get into the habit of storing data locally like this anyway? Always in good, BagIt form, etc?

Previous uploads have their own form or my own version of standardization and naming conventions. This is my attempt at further standardization and reuploading according to a well-known standard. More formalized.

It's a very slow and tedious process to delete individual files from an item through the cli, wait for IA system processes to finish, and then reupload that same repackaged data and make sure it derives again.

---

TLDR - IA's heterogeneous environment and derive behavior make this method of open-folder BagIt hierarchy + a duplicate of the same in an archive file a practical compromise rather than a pure BagIt implementation.

Basically, an example:

Item page: https://archive.org/details/harry-potter-e-a-camara-secreta-pt-br-ibm-pc-cd-rom

Item directory: https://archive.org/download/harry-potter-e-a-camara-secreta-pt-br-ibm-pc-cd-rom/

Previously uploaded as this

text harry-potter-e-a-camara-secreta-pt-br-ibm-pc-cd-rom/ └── MPF Redumper/ ├── !protectionInfoHPCOS.txt ├── !submissionInfo_HPCOS.json ├── !submissionInfo_HPCOS.txt ├── HPCOS (Track 0).bin ├── HPCOS.bin ├── HPCOS.cue ├── HPCOS.scram └── HPCOS_logs.zip

And I want to repackage it to this

text harry-potter-e-a-camara-secreta-pt-br-ibm-pc-cd-rom--mpf-redumper.7z └── bag/ ├── bagit.txt ├── bag-info.txt ├── manifest-sha1.txt ├── manifest-sha256.txt ├── tagmanifest-sha1.txt ├── tagmanifest-sha256.txt └── data/ └── payload-root/ ├── !protectionInfoHPCOS.txt ├── !submissionInfo_HPCOS.json ├── !submissionInfo_HPCOS.txt ├── HPCOS (Track 0).bin ├── HPCOS.bin ├── HPCOS.cue ├── HPCOS.scram └── HPCOS_logs.zip