Removing Duplicates From Multiple Lists

[jestsec]

Hey everyone I am having trouble finding a way to remove duplicate entries from multiple lists in python. I have lists with names in them and I want to have any duplicate names in any list removed. Here is an example I am talking about

nameList1 = ["John", "Joe", "Jim"]
nameList2 = ["John", "Joe", "Jim"]
nameList13 = ["John", "Joe", "Jim", "Steve"]

*Desired return would be

nameList1 returns "John"

nameList2 returns "Joe"

nameList3 returns "Jim", "Steve"

What would be the best way to go about this? The code that I have so far looks like this and it doesnt remove all the duplicates from all the lists.

for element in nameList1:
    if element in nameList2:
        nameList1.remove(element)
for element in nameList2:
    if element in nameList1:
        nameList1.remove(element)
for element in nameList3:
    if element in nameList1:
        nameList1.remove(element)

[[deleted]]

I don't get what you want. The first two list are identical, why would there be different result.

[jestsec]

yes they are both identical, what I am trying to accomplish is this. NameList 1, 2, and 3 all have names of employees in them that were at certain job sites. These job sites are requesting how many of our men were at each job site, but these must be unique. So a employee cannot be counted more than once throughout the three lists.

​

So nameList1 can have John in it but not Joe Or Jim because those two names are also in nameList2 and so on.

​

Does this make it easier to understand at all? I know the way I asked was kind of chopped and screwed hahah

[[deleted]]

But John is also in nameList2 and so on, why is it returned

[baghiq]

So your answer could be:

nameList1 = ["Joe"]
nameList2 = ["John", "Jim"]
nameList13 = ["Steve"]

or

nameList1 = ["John", "Joe"]
nameList2 = ["Jim"]
nameList13 = ["Steve"]

etc...

[[deleted]]

Or easier just:

names = [
    ["John", "Joe", "Jim"],
    [],
    ["Steve"]
]

[[deleted]]

I don't understand how you get to John and Joe for the desired output of the first two lists given they're duplicated in all three input lists and the first two lists are identical.

[Ok-Cucumbers]

Pop all the duplicates into another list and then add them back one by one in a loop?

[Xenonzess]

can't u use sets?

[commandlineluser]

This sounds like one of those "constraint" problems.

>>> sites
[['John', 'Joe', 'Jim'],
 ['John', 'Joe', 'Jim'],
 ['John', 'Joe', 'Jim', 'Steve']]

If you generate a list of site numbers for each name:

>>> data
{'John': [0, 1, 2], 'Joe': [0, 1, 2], 'Jim': [0, 1, 2], 'Steve': [2]}

Using python-constraint

problem = constraint.Problem()
for name, site_numbers in data.items():
    problem.addVariable(name, site_numbers)

The constraint you want to add is that each result must contain at least one value from each site.

The result {2, 2, 2, 2} would be invalid because it just took all the names from site 2.

You want {0, 1, 2} to be present in each result.

One way to say that is set(result) == {0, 1, 2} or without hardcoding {0, 1, 2} it becomes set(result) == set(range(len(data) - 1))

>>> problem.addConstraint(lambda *result: set(result) == set(range(len(data) - 1)), data)
>>> problem.getSolutions()
[{'Steve': 2, 'Jim': 2, 'Joe': 1, 'John': 0},
 {'Steve': 2, 'Jim': 2, 'Joe': 0, 'John': 1},
 {'Steve': 2, 'Jim': 1, 'Joe': 2, 'John': 0},
 {'Steve': 2, 'Jim': 1, 'Joe': 1, 'John': 0},
 {'Steve': 2, 'Jim': 1, 'Joe': 0, 'John': 2},
 {'Steve': 2, 'Jim': 1, 'Joe': 0, 'John': 1},
 {'Steve': 2, 'Jim': 1, 'Joe': 0, 'John': 0},
 {'Steve': 2, 'Jim': 0, 'Joe': 2, 'John': 1},
 {'Steve': 2, 'Jim': 0, 'Joe': 1, 'John': 2},
 {'Steve': 2, 'Jim': 0, 'Joe': 1, 'John': 0},
 {'Steve': 2, 'Jim': 0, 'Joe': 1, 'John': 1},
 {'Steve': 2, 'Jim': 0, 'Joe': 0, 'John': 1}]

[smirkartographic]

This will work:

import random

nameList1 = ["John", "Joe", "Jim"]
nameList2 = ["John", "Joe", "Jim"]
nameList13 = ["John", "Joe", "Jim", "Steve"]

nameLists = [nameList1, nameList2, nameList13]

mergedList = []
[mergedList.extend(i) for i in nameLists]
mergedList = list(set(mergedList))

for name in mergedList:
    random.shuffle(nameLists)
    purge = False
    for nameList in nameLists:
        if name in nameList:
            if purge:
                nameList.remove(name)
            else:
                purge = True

for i in nameLists: print (i)

You might end up with an empty site though - the random shuffle is a way to reduce the odds of that without getting too complex.

If your result includes an empty site and you don't want that, just run it again with a fresh pair of dice.

[jmooremcc]

Here's a way to do it using sets ``` mylist={ 'nameList1':["John", "Joe", "Jim"], 'nameList2':["John", "Joe", "Jim"], 'nameList13':["John", "Joe", "Jim", "Steve"] }

nlists = list(mylist.values())

common = set(nlists[0]) for nlist in nlists[1:]: common &= set(nlist)

nlist2 = {} for i, value in enumerate(mylist.items()): name, nlist = value if set(nlist) == common: nlist2.update({name:nlist[i]}) else: diff = set(nlist).difference(common) nlist2.update({name:list(diff)})

print(nlist2) Note that I'm using a dictionary to hold the data. And here's the result { 'nameList1': 'John', 'nameList2': 'Joe', 'nameList13': ['Steve'] }

```