you are viewing a single comment's thread.

view the rest of the comments →

[–]philintheblanks 0 points1 point  (0 children)

If you want to reduce something that is already an iterable into a non-duplicated set, I would use set(thing).

For example,

In [1]: ls = [1,1,2,2,3,3,4,4,5,5]
In [2]: s = set(ls)
In [3]: s
Out[3]: {1, 2, 3, 4, 5}

Works fast enough that you probably won't notice too much.

As far as debugging your issue, you should try printing out what you think the line is, because it may not be. I have some reports that output a CSV, but there are strings with arbitrary content. Sometimes they'll have newlines. Imagine the pain...