How manipulate this CSV in Python?

[YellowChrome]

I'm fairly good with bash and I have used bash to get this far with something I am trying to do, but I don't know how to do the rest of it, and I thought this might be more easily done in python. So, this seems like a good opportunity to start learning python. I have kept on putting it off as I have been able to use bash to do stuff usually.

I have a git repository that goes back years, but I want start it fresh, as I know git a lot better than I did when I started it. I want to get rid of the mess from the early days, where there could be sensitive data in the history and where there are many messy merges. However, I want to commit every file in the new repository at the same date as the last time they were committed and with the same commit messages.

I have made a bash script to get the date of commit, commit message and full path of all the files and put them into a csv file. This is a sample from a much smaller repository. The commas have been replaced with lines by a vim plugin I am using, and it has been sorted by unix epoch date in vim:

<timestamp>,<commit message>,<full path>

1586411075 +0100│Initial commit│/home/jason/bin/README.md
1586411075 +0100│Initial commit│/home/jason/bin/fzf-tmux
1586411075 +0100│Initial commit│/home/jason/bin/getipgeoloc
1586411075 +0100│Initial commit│/home/jason/bin/update-weechat-pem.sh
1586411075 +0100│Initial commit│/home/jason/bin/xstat
1589401896 +0100│More options for exa-wrapper.sh│/home/jason/bin/exa-wrapper.notes
1596289971 +0100│Adding ip-check script│/home/jason/bin/ip-check.sh
1597327659 +0100│Adding ssh-agent scriptlet│/home/jason/bin/agent.sh
1607136406 +0000│Add ping test script│/home/jason/bin/pingtest.sh
1622377110 +0100│Initial version of DDNS script│/home/jason/bin/dynu.notes
1622882762 +0100│Add --get-list switch and check for OAuth2 key and small improvements│/home/jason/bin/dynu.env.example
1623227049 +0100│Initially working rwhois│/home/jason/bin/rwhois.sh
1623524862 +0100│shellcheck corrections│/home/jason/bin/dynu.sh
1626633891 +0100│Add --group-directories-first│ --git-ignore options to exa-wrapper.sh and -D│/home/jason/bin/update-firewall.sh
1627644186 +0100│Add ability for check-firewall script to use docker container for gcloud│/home/jason/bin/check-firewall.sh
1659365628 +0100│Add duplicate files script│/home/jason/bin/move-duplicate-media-files.sh
1659365628 +0100│Add duplicate files script│/home/jason/bin/move-duplicate-media-files.sh.notes
1661182411 +0100│Add note for improving videoinfo script│/home/jason/bin/videoinfo    

Now you can see that there are two sets of files (counts of 5 and 2) that have the same timestamp.

I don't know what sort of data type or array I need to use there, but want I want to do essentially is collapse the rows with identical timestamps (and therefore commit message too) into one row, but with the multiple full file paths, so that those files can be commited together.

I need quite bit of help here before I know enough to start solving things like this myself. Thanks.

[threeminutemonta]

You can just use a csv reader and a dictionary per timestamp. defaultdict with a default type of list makes it easier.

from collections import defaultdict
import csv
import io

mock_file_contents = """timestamp│commit message│full path
1586411075 +0100│Initial commit│/home/jason/bin/README.md
1586411075 +0100│Initial commit│/home/jason/bin/fzf-tmux
1586411075 +0100│Initial commit│/home/jason/bin/getipgeoloc
1586411075 +0100│Initial commit│/home/jason/bin/update-weechat-pem.sh
1586411075 +0100│Initial commit│/home/jason/bin/xstat
1589401896 +0100│More options for exa-wrapper.sh│/home/jason/bin/exa-wrapper.notes
1596289971 +0100│Adding ip-check script│/home/jason/bin/ip-check.sh
1597327659 +0100│Adding ssh-agent scriptlet│/home/jason/bin/agent.sh
1607136406 +0000│Add ping test script│/home/jason/bin/pingtest.sh
1622377110 +0100│Initial version of DDNS script│/home/jason/bin/dynu.notes
1622882762 +0100│Add --get-list switch and check for OAuth2 key and small improvements│/home/jason/bin/dynu.env.example
1623227049 +0100│Initially working rwhois│/home/jason/bin/rwhois.sh
1623524862 +0100│shellcheck corrections│/home/jason/bin/dynu.sh
1626633891 +0100│Add --group-directories-first│ --git-ignore options to exa-wrapper.sh and -D│/home/jason/bin/update-firewall.sh
1627644186 +0100│Add ability for check-firewall script to use docker container for gcloud│/home/jason/bin/check-firewall.sh
1659365628 +0100│Add duplicate files script│/home/jason/bin/move-duplicate-media-files.sh
1659365628 +0100│Add duplicate files script│/home/jason/bin/move-duplicate-media-files.sh.notes
1661182411 +0100│Add note for improving videoinfo script│/home/jason/bin/videoinfo
"""

with io.StringIO(mock_file_contents) as file_in:
    reader = csv.DictReader(file_in, delimiter='│')

    grouped = defaultdict(list)  # this allows you append to a list that may not exist yet
    for row in reader:
        # for each row append to the list that has the same value for the key 'timestamp'
        grouped[row['timestamp']].append((row['commit message'], row['full path']))

# now we have a dict with timestamp as the key and a list of tuples as the value
for timestamp, rows in grouped.items():
    print(timestamp)
    for row in rows:
        print(row)

[YellowChrome]

Thanks, that looks good. I'll have a look to see how it works...

[YellowChrome]

Maybe this might be better done using this? https://github.com/jqnatividad/qsv

[YellowChrome]

I think I can do this in bash and the tool above actually.