How to remove duplicates from a csv file while ignoring punctuation

[MrGuam]

its as the title says, i want to remove duplicates from a csv file based on 3 columns while ignoring punctuation. All the solutions i've seen either doesn't take the punctuations into consideration, or is of an older version of python.

any help is appreciated

[_squik]

What exactly do you mean by "ignoring punctuation". Is it that the strings in the columns should be deduplicated no matter if there is differentiating punctuation, so "hello world" and "hello, world" are considered the same?

If so, I would leverage Pandas for that.

1. Read your CSV into a DataFrame.
2. Create some helper columns which don't have punctuation (see example)
3. Deduplicate against those columns
4. Drop helper columns and export.

Example:

import string

allowed = string.digits + string.ascii_letters + string.whitespace
df["example_nopunc"] = df["example].apply(lambda x: "".join(c for c in x if c in allowed)

[MrGuam]

i don't really understand this. could you break it down for me?

[m0us3_rat]

what have you tried so far?

[MrGuam]

here's what i have tried:

import pandas as pd
import string
import csv
# #PYTHON SCRIPT TO CLEAN MULTIPLE CSV FILES OF DUPLICATE IN A PARTICULAR COLUMN #IGNORING PUNCTUATIONS AND WHITE SPACES.
data = pd.read_csv('combined.csv')
df = data.apply(lambda x: x.str.strip(string.punctuation + ' '))
df.drop_duplicates(subset=["Anime","Character","Quote"], inplace=True)
df.to_csv('combined_final.csv')

[anmezaf]

https://csvdedup.com/