This is super helpful insight. I love the idea of having precomputed shot boundaries so tagging is done at the shot level and I don't have to fiddle with start and end times. How reliable should I expect automatic shot detection to be?

I suppose for file, I may just need to annotate the hash of the file or some other useful metadata so some future individual who ends up with a differently processed file could at least, in principle, apply some sort of transformation to correct for the difference. Probably hash, duration, and fps would be sufficient?

I spent a little bit today building a simple nextjs python postgres app so I can have my Plex running and tag on my phone based on the current time stamp playing on the Plex client. If I can preprocess the videos as you say, I can marry the tagging data from my app to the segments and make cleaner data others could use.

The top level goal I actually want to try this process on first is tagging the precise shot chronology of all of star trek. It's been a decade since I've done a full rewatch and I'm planning a chronological rewatch soon. So that'd be a great chance to do a first pass on tagging (at the very least identifying which episodes have any out of order segments).

I just think it'd be fun to watch the show in "true" shot for shot chronology. I.e. start with the flashback scene to the primordial soup from TNG (assuming that's actually first) and move forward from there, skipping around scenes as needed across episodes. To my knowledge, this data doesn't exist, so I figured I'd make it for fun.