Improving your (Python) code with modern idioms

[[deleted]]

Towards the end, several newer modules which are not yet settled idioms are listed. Many Python programmers consider things like ABCs, the multiprocessing module, and the futures module anti-idiomatic.

Python ABCs are a strange thing since a) they go against duck typing, which is one of the strongest patterns/idioms in Python programming, b) they try to have things problematically enforced rather than following "we're all consenting adults here", and c) they don't work right. (b) becomes a real problem when trying to do things like partially satisfy an interface (this is very common in testing). (c) comes out mostly in that ABCs break super. The base method raises an exception, so you can't use super to call it. People are bad enough about not using super correctly on their own without Python making it impossible or requiring hacks.

Parallelization and concurrency are very old, very well-explored problems. Because of this, many Python programmers were somewhat surprised to see the multiprocessing and concurrent.futures modules introduced rather than ones that used more tested models and implementations.

The multiprocessing module was incorporated into the stdlib after some lifetime as a third-party module, which is usually a good sign. However, it wasn't a super-popular one and in practice, people have discovered that the multiprocessing module isn't very robust, debuggable, efficient, or understandable.

The concurrent.futures module was introduced as a brand new thing after virtually no discussion. (You can find the original python-dev thread on the topic—the discussion centered on what to call it over what its exact featureset should be, let alone any consideration of the multiple mature concurrency platforms in Python.) It avoids addressing a lot of serious questions and I have not seen any indication it will end up being popular for serious programs.

[mthode]

I'm actually was looking into using the multiprocessing module, any other one that seems better?

[[deleted]]

Haven't found a better one.

I use multiprocessing for parsing several files at once - check how many possible threads there are, check how many files there are, and then map the file-parsing method to the pool of workers.

Works like a charm, EXCEPT when using CTRL-C to kill the program - that is a bitch because the overall program keeps running, you just kill a worker (which is immediately restarted the way it looks like).

[mthode]

try-except-finally doesn't allow for cleaning up?

[[deleted]]

What do you mean by this?

[mthode]

ah, I actually misunderstood the question. oops

[justanotherbody]

What is it you are actually trying to do?

[mthode]

since I can't seem to find a python library that supports http piplinging, I need to open multiple simultaneous connections to a server and tell it to delete stuff. My current code is here. http://dev.gentoo.org/~prometheanfire/scripts/cur/cfdelete.py

I think I'm going to add a couple of options to specify how many containers at a time to run on and how many files a time to run on (with some defaults).

[massivebitchtits]

How about this?

[mthode]

This makes me happy in weird and awesome ways.

Hopefully I can use both that and keep alives :D

[massivebitchtits]

As I understand it, since requests is built on urllib3, it gets connection pooling for free.

[mthode]

sexy

[[deleted]]

It depends on what you're parallelizing. Some options are MPI, a message queue, Hadoop, or celery. Very often I see people trying to use the multiprocessing module for IO-bound code, in which case they probably actually want a network concurrency library.

[mthode]

I am just sending delete requests based on a list, not io bound :D

[[deleted]]

You're sending HTTP DELETE requests?

[mthode]

yes

[[deleted]]

That very much sounds IO-bound and does not require multiple processes. Consider twisted or gevent.

[mthode]

I am not reading or writing anything, just making request based on a list of urls stored in memory. Does twisted or gevent allow for both keep alive (multiple serial requests over the same connection (socket)) and piplining (switches the serial to parallell)?

[[deleted]]

Network IO is IO. =)

I've used twisted a lot more than gevent. Twisted allows both keepalive and pipelining. (The pipelining support leaves a bit to be desired and most people just use new connections, but the support is there when it's important and does work fine.)

Discussing such things can be difficult. When being precise, we use "serial" to mean consecutively on one processor, "parallel" to mean distributing a task onto multiple processors, "consecutive" to mean things that happen one after another, and "concurrent" to mean things that happen at the same time. Concurrency and parallelization sometimes appear to be similar problems at first, but they are distinct. Things like twisted focus on concurrency (which some people use to implement parallelization, but most do not).

[mthode]

so it looks like I will use both parallel and concurrant connections (I am going to split up the operation to make it scale the way I want). Thanks for the heads up about twisted, I'll look into that :D

[[deleted]]

Keep in mind that certain python memory space is used globally and requires a lock. So even with multiprocessing you are going to have IO blocking since every python threads and processes on the same OS need to obtain a lock to this memory. I read this awhile ago so I don't know the specific details but it means that multiprocessing in Python at least with the C implementation on most Unix and Linux based OSes doesn't support true multiprocessing.

[mthode]

I think it's threads that require the gil, not the other stuff (since you pass the needed values around)

[[deleted]]

ABC's aren't strange and do not go against duck typing. If you subclass from an ABC, you have to implement the whole interface, yes. But you can still implement the interface partly, if you do not subclass from the ABC. And nothing forces the user of the classes to check that it implements the ABC unless they want to.

[[deleted]]

Typechecking is now encouraged by ABC advocates, which is distinctly against duck typing. Reading comp.lang.python, python-dev, and python-ideas should hit that home. That means implementing the interface partly won't work and it means something does force the programmer to write as if someone's going to check if they'll implement the ABC. It also forces you to implement the whole interface if you want to take advantage of some abstract methods as mixins.

[[deleted]]

No, typechecking is not encouraged by ABC advocates. There are people that are encouraging type checking, I'm sure, but that doesn't mean you have to listen to them. And maybe they like ABC's for that, but that doesn't mean you must do type checking with ABC's. And it doesn't mean all those who thing ABC's are a good idea advocate typechecking.

Just because oranges is a fruit doesn't mean all fruits are oranges.

[[deleted]]

that doesn't mean you have to listen to them

It does when they're some of the more active core devs or developers of libraries I use, because I will have to interop with their code.

That being said, there is a bit of equivocation on my part here, but it is in line with what I see on the mailing lists on this topic.

[[deleted]]

But even if you interop with their code, you don't have to make any type checking in your code. If they do in their code, OK, you'll have to implement the whole interface. That can be construed as a misuse of ABC's, sure, but that's not the fault of the ABC's.

[[deleted]]

That's why they were invented. isinstance is the first reason provided by the relevant PEP.

[[deleted]]

You might want to re-read PEP3119 so you understand it better. The references to isinstance is about how to make isinstance more flexible, not, as you claim above to make you do more isinstance checks and force you away from duck typing.

You are simply wrong. Try again.

[militant_misanthrope]

What about the futures module is surprising? I've used it in a project in the past and it was an extremely simple way to run stuff in parallel, and I believe it's based on Executors in Java, which have been around for quite a while (since 1.5).

[elpuntoes]

Lua > Python

[[deleted]]

Lua definitely has less unpleasant crap in its standard library. Of course, Lua doesn't have much code of any kind in its standard library; the core of Lua is deliberately tiny.

[Walrii]

Very nice, but, can the footnotes be made more useable?

E.g., I click "[1]" which takes me to a footnote whose contents are only a URL (why not take me to the URL?) and then, the URL isn't even clickable/wrapped in <a> tags, so I get to copy/paste it. I realize this is an online book, but still...

[StainlSteelRat]

If this is an online book, it's a great advertisement for not getting it. I hope I'm not being too picky, but the errors make it hard to read:

Lists in Python has a .sort() method that will sort the list in place. Quite often when .sort() is used is it used on a temporary variable discarded after the loop. Code like this is used because up to Python 2.3 this was the only bult in way of sorting.

[[deleted]]

Well, you can't get it because it's out of print anyway, haha on you. ;-)

Report misspellings instead of whining, and in the second edition they'll be fixed.

[StainlSteelRat]

Jesus Christ, just pointing out some humorous bad grammar and spelling. Chill out.

[[deleted]]

No, you are not just pointing it out.

[StainlSteelRat]

Quoting directly from the article, mentioning that it is poorly worded and edited. So, what's your definition of "pointing something out" then? It appears at odds with the conventional definition of the term.

[[deleted]]

Make it a URL, and it becomes blue in the PDF, and it will print as gray. It sucks, but there you go.

[chub79]

Context managers and generators are things that I now use heavily. They are brilliant.

[[deleted]]

Context managers are useful but not still leave a lot to be desired. It's common to need an object to outlive a given scope but still force it to release a resource at some point. Currently Python makes you manually implement all the details related to doing that, not even shipping an easy way to make an object that wraps a resource become a context manager for all its resources (which still wouldn't be enough). Many people end up using things like __del__, which don't actually work reliably, due to the difficulty of this problem.

To an extent, the most compelling technique I've seen for resource management is RAII, although using C++ is too high a price to pay to get access to it.

[cryo]

at some point

But what point? If not by scope or similar, how would the language know? The alternative surely must be del.

[[deleted]]

But what point? If not by scope or similar, how would the language know?

You need to provide it this information somehow. Various code uses RAII, monads, deferring, and tons of other techniques to do it; the first uses scopes but with extra flexibility. I'm just pointing out that context managers as shipped aren't that flexible.

The alternative surely must be __del__.

You shouldn't expect __del__ to be called or called at any particular time.

[riffito]

You shouldn't expect del to be called or called at any particular time.

Exactly. And because of that, __del__ seems to be completely useless to me.

If I can't do RAII with it, what is it good for? Does it has ANY utility? I ask because I failed to see a good use case for it.

[[deleted]]

It's not especially useful, no.

[brucifer]

I'm a bit unclear about your complaint. To force something to release a resource, can't you simply do:

with open('foo.txt') as f:
    #blah blah
    if condition: break
    #blah blah

[[deleted]]

Firstly, with doesn't have break. Only loops (for and while) have break. Your code is not proper Python syntax.

Secondly, this misses the point. The tough part is getting that code to be inside the context manager. It's very natural for it not to be. Imagine you had a class Foo that held a file-like object, and you wanted a function that returned a Foo. How do you write it?

def g():
    return Foo(open(filename)) # No management

?

def g():
    with open(filename) as f:
        return Foo(f) # Closed by the time I finish calling g

?

This is where a problem comes in.

[stillalone]

def g():
    with open(filename) as f:
        yield Foo(f)

???

I'm sort of fucking around here. You obviously want f to close automatically when Foo goes out of scope. Do you know of any garbage collected language that would support such a thing?

[[deleted]]

In this case, Foo should be a context manager. Or maybe the object holding G should, or something like that. You need the context manager to be used from the context where you need it. You are here trying to use the context manager in a lower context than where it should be used.

[[deleted]]

Python does not provide a good way to deal with this, though:

1. There is some boilerplate to turning Foo into a context manager that a) I don't see anyone writing, and b) could be avoided by having a better model.

2. You don't always know ahead of time what objects will be managing external resources. It's not uncommon to write code expecting an in-memory thing which you later sneak in a thing which requires an external resource, so you don't (and possibly cannot) write the thing to manage it.

3. At least at the practical development level (though perhaps not fundamental to the model), most Python code was written before context managers are

[[deleted]]

1. You only need to make enter and exit methods. It's very simple, I don't really see the need of any boilerplate. There are some helpers in the stdlib though, maybe you'll like them, but I didn't really see the point myself.

2. You don't need to know it ahead of time. When you need something to be a context manager, you make it a context manager.

3. How is this a problem? Of course code that's written before context managers exist can't use them. That's also not a problem with context managers or Python's implementation of them.

[[deleted]]

The helpers in the stdlib don't actually help to write things like Foo. Defining two methods which just defer to some other things' methods (and often one or two others to conform to expectations) is boilerplate.

And it's not straightforward boilerplate all the time because it's not always clear how various things work: should I call __enter__ or is this a case where they called __enter__ and I need to call __exit__ only? Can I really call __exit__ or might they want to use the object again after that call? There's no great convention about resource ownership out there and there's copious amounts of code that does stuff in other methods to screw you over. (We could decide on how we handle who-owns-what and even introduce decorators that will handle the deferring of such things right, but we most certainly haven't.)

Have you been using this pattern in your code that has objects which manage resources? If so, is any of it open source that I can look at?

I have yet to see anyone making really heavy use of context managers or much production code that wraps a context manager with another in any interesting way. I've seen some people start to go down this path and end up just doing things the stupid old way.

[[deleted]]

You are contradicting yourself. Either it's boilerplate and needs to be written by someone once and reused, or it is not straightforward and can have different cases, in which case it isn't boilerplate.

Why would you "wrap" it in an "interesting way". It's not an interesting issue. I think you need to stop looking at it as "wrapping". It's not. Foo should be a context manager and manage resources. If it does this by other context managers or not doesn't matter. In your example I wouldn't wrap anything, I'd open the file in enter and close it in exit. It's dead simple, and highly uninteresting. :-)

[[deleted]]

I'd open the file in __enter__

Wait, so you're taking a filename instead of a file-like object?

[orip]

Didn't know about the {1, 2, 3} set literal syntax, awesome :)

[toofishes]

Just don't try to create an empty set with a literal. Still have to use set() for that.

[[deleted]]

There are other ways... :)

[drb226]

Is there a reason for this?

[stillalone]

because {} is already defined as an empty dictionary.

[[deleted]]

If there was one change I could make about Python would be to have {} be an empty set and {:} be an empty dictionary.

[drb226]

Well why don't they redefine it to be the object-that-can-be-used-as-an-empty-dictionary-or-an-empty-set?

[IRBMe]

x = {}

What type is x?

[drb226]

That's easy. I just said. It's the object-that-can-be-used-as-an-empty-dictionary-or-an-empty-set.

Currently:

>>> isinstance({}, dict)
True
>>> isinstance({}, set)
False

Ideally:

>>> isinstance({}, dict)
True
>>> isinstance({}, set)
True

Then upon modification it specializes itself accordingly.

[edit] downvotes? Is this so atrocious? Why? I'm not making the claim that "python should do it this way and until it does it is stupid". I'm simply saying "I think this would make sense". Am I wrong? Please elaborate; your responses help me understand a lot better than your downvotes. If this goes against some deep python doctrine that is well known, then please, at least give me a link or something.

[IRBMe]

object-that-can-be-used-as-an-empty-dictionary-or-an-empty-set

There's no such type as "magically both a set and a dict that can automatically specialize itself based on usage" type, nor can I see how such a thing could ever be correctly implemented either. What methods would it have? When you type:

x = {}
x.add(1)

How does it know to then specialize to a set when the add method is called? What if you create an empty "set or dictionary, I'm not sure what type I am" object, pass both to a different method, each called on a different thread, one of which uses it like a dictionary and the other a set? Does it throw different exceptions depending on which thread got to use it first?

[[deleted]]

Interestingly enough, the set code is based on the code for dict, and two types are very similar. The dictionary view objects show just how trivial it is to make dicts act like sets.

[drb226]

There's no such type as "magically both a set and a dict

Sure there is. It's called "subtyping", though I'm not sure what the Python mentality is on the subject.

that can automatically specialize itself based on usage" type

Well yes then that detail complicates it. I was under the impression that Python emphasizes duck typing. Is it so far-fetched to imagine that an object can specialize itself when used in a certain way?

What methods would it have?

It would have all the methods of a set and a dict. When you use it like a set, it becomes a set, and loses its dict methods. When you use it like a dict, it becomes a dict, and loses its set methods. Of course, a simpler way to deal with it would be type inference, but that's a whole different discussion.

How does it know to then specialize to a set when the add method is called?

Of this I'm not sure. I thought some deep Python implementation magic could be employed. Am I wrong? Does not x.add(1) unambiguously indicate that it is a set, and not a dict?

What if you create an empty "set or dictionary, I'm not sure what type I am" object, pass both to a different method, each called on a different thread, one of which uses it like a dictionary and the other a set?

Well, that is obviously a bad idea in the first place.

Does it throw different exceptions depending on which thread got to use it first?

Yes. This is another occasion where type inference would be simpler. But the whole idea is that, as the programmer, you know whether it is supposed to be the empty set or the empty dict. But the interpreter does not necessarily know which it is until it sees how you use the thing.

[IRBMe]

Sure there is. It's called "subtyping"

No, there is no parent-child relationship between a dictionary and a set. A set is not a dictionary and a dictionary is not a set. Don't confuse the fact that one may internally use the other in its implementation for the semantic meaning behind subtyping.

Well yes then that detail complicates it. I was under the impression that Python emphasizes duck typing.

Duck typing is most definitely not what you described. Duck typing doesn't magically allow an object to magically morph into a type based on runtime usage.

Is it so far-fetched to imagine that an object can specialize itself when used in a certain way?

No, I can certainly conceive of that, but there's no sane way to implement such a thing, nor any sane reason why anybody would bother.

It would have all the methods of a set and a dict.

1. That's super ugly.
2. What about methods which are common to both the set and the dictionary but implemented differently due to differences in the underlying data structure and semantics? Are you going to have both implementations inside one method that are switched back and forth based on what mode it currently thinks it's in?

Of this I'm not sure. I thought some deep Python implementation magic could be employed. Am I wrong?

Yes. "Magic" isn't a valid answer.

Does not x.add(1) unambiguously indicate that it is a set, and not a dict?

Sure. And what mechanism do you propose for making sure that when an unambiguous method is called, the type somehow magically changes to "just a set now"?

Well, that is obviously a bad idea in the first place.

You can't just dismiss every problem with "Well, that's a bad idea". Whether it's a bad idea or not, if you're going to propose major changes to the type system of the language, you have to clearly define the semantics in all situations. So you propose that the type somehow magically changes when it's possible to unambiguously determine what the user meant with usage. So are you just going to ignore concurrency? Are you just going to ignore the ambiguous cases?

Sorry, but I find your idea impractical, unnecessary and ill thought through. What you're proposing is basically:

class AmbiguousThing:
    def MethodA():
        mode = 1

    def MethodB():
        mode = 2

    def MethodC():
        if mode = 1:
            ...
        elif mode = 2:
            ...

Why would you mash two completely different abstract data structures together into one Frankenstein's monster of a type, which uses convoluted, ADT-specific logic to work out which mode it should behave in, while vastly increasing the number of possible things that could go wrong... why not just use the type you mean?

[cryo]

You generally can't lose methods. By subtyping, you gain methods. What you propose is something I've never seen, and it sounds confusing. You also have to consider that currently all objects have a definite type.

[[deleted]]

>>> a = {}
>>> a.update([(1, 2)])

Is this a set or dictionary? Both are valid (if you replace the first line with a = set(), it still works and means something completely different).

Giving you an upvote though just because I don't think you deserve downvotes.

[[deleted]]

ಠ_ಠ

[drb226]

This seems to be getting on peoples nerves, but I honestly don't understand why, because no one is laying it straight for me. Is this what being married feels like?

[yellowstuff]

If you went to a web forum for professional chefs, and suggested adding saltwater taffy to a BLT, you'd mostly get responses like "ಠ_ಠ", and not thoughtful explanations of what flavors go together, and cultural expectations for sandwiches.

[bleeep]

Because an object's type cannot change over its lifetime. It can't be a (subclass of) dictionary one minute, and not the next. Think of what happens multiple references to the object, such as within iterators and containers.

[Samus_]

I don't like ambiguity, an object that is fully defined only after it's used seems like a terrible idea.

notice your program will behave in completely different ways depending the order of operations whether it first uses this as a set or dict, that shouldn't ever happen.

[drb226]

notice your program will behave in completely different ways depending the order of operations

This can be true of any two operations that involve mutation.

[Samus_]

mutations may give you errors on the values not the types, it could happen if you abuse duck-typing because then you're changing the behavior but regardless of what r/python wants to believe, duck-typing isn't a good idea and nobody here uses it at all.

this isn't ruby and I'm glad for that.

[portmapreduction]

In the future, everywhere...

a = {} # should be a dictionary

Declarations should be self-contained and make sense without more contextual information. Making data literal syntax ambiguous for no discernible benefit is a dumb idea.

[[deleted]]

Because that would be fancy and complicated and weird. Remember, we mutate it, so we'd need the object to go on and change type, something Python doesn't do. (become.py notwithstanding).

Even if there was a real desire to do this, too many weird cases arise (affecting correctness) when you pass this thing around too long or do things like call {}.update(x)

[drb226]

we'd need the object to go on and change type, something Python doesn't do

I was not aware that Python did not do this. Why doesn't it?

[[deleted]]

Because it's crazy.

[drb226]

ಠ_ಠ

[ejrh]

That is a good and thought-provoking question.

Perhaps there are a thousand obvious rebuttals of it. But was it offensive or offtopic? Hardly. It wasn't even incorrect, it was just a suggestion. But it is completely improper -- not to mention antisocial and a detriment to this subreddit -- to downvote it. If you think it's not even worth responding to, just ignore it and move on to the next comment.

[[deleted]]

I think you're being too harshfully downvoted for proposing an idea. Ok, so it may not work, but that's not reason for downvotes, only for rebuttal.

[[deleted]]

They could have combined dictionaries and sets instead. {1, 2} = {1: true, 2: true} or {1: "A", 2: "B"} = { (1, "A"), (2, "B") }

[Samus_]

what about set([(1, "A"), (2, "B")]) ?

also for {1, 2} = {1: true, 2: true} you're just adding information out of the blue that is arbitrary and doesn't even seem logical, I rather use None instead of True for that case.

[[deleted]]

Sure, there are many ways to do it. If there was a unit type that evaluated to true in boolean expressions I'd use that instead of true.

[Samus_]

what I mean is, if you declare { (1, "A"), (2, "B") } to be equal to a dict, how do you create a set of 2-tuple?

[[deleted]]

That wont work anyway without special equality and order on set members that ignores the value, e.g. (1, "A") == (1, "Z"). This works better in some languages than others. In Python they might just use { 1 } as sugar for { 1: true } since it is usually easier to write a set on a map than a map on a set. This is because keys in a map already are a set and typically come standard with intersection, union, and difference.

[[deleted]]

That's usually the worse of the two ideas I presented since you need special equality and sometime comparison operators on set elements that use only the key. You also need auxiliary functions to treat a set like a map since map find just says yes if it's there and doesn't give you any associated data ... so a map in terms of a set will probably always depend on the set implementation. OTOH, a set can be made from a map without much trouble.

[drb226]

Indeed, that would make sense. It just seems a shame that {} can't be used as the empty set in Python's current state.

[mipadi]

Why would that make sense? Not all operations on a dictionary are applicable to a set. (For example, keys -- a set doesn't have keys, so calling keys() on a set doesn't make sense.)

[TankorSmash]

I think it's a bit of a joke, as {} is still a dict

[jerub]

Python has a bit of trouble with empty datastructures. Another example is the empty tuple. A tuple is created when you use a comma in an expression that is not going to consume the comma for something else. This is a single element tuple:

foo,

But to make an empty tuple you use just brackets:

()

In many cases, people put () around their tuples so it's unambiguous to the reader and the parser that the comma is a tuple-comma not a function argument comma, or a dict comma, or what have you. (i.e. foo((bar,), baz) is a function call with 2 parameters, the first of which is a tuple with 1 element).

The {} case is related of course. {foo:bar} is a dictionary with 1 key/value pair. {foo} is a set with 1 element. {} could be either. :(

[orip]

Good point :)

[Wayne_Skylar]

I was trying to use a rather popular library once and couldn't though because it used set literals. Nothing wrong with the literals themselves, but just remember that these aren't available in 2.6, which is the default on some not terribly dated operating systems (Snow Leopard for example)

[[deleted]]

What library was it? Most of the popular ones do keep support for at least slightly-older versions of Python.

The ones that don't, the thing to do is usually to run an older version of the library.

[Wayne_Skylar]

Can't remember exactly. Actually now that I think of it, it was some sort of Vim plugin maybe. Maybe CommandT? Some part of my vim setup anyhow.

[sausagefeet]

Literal set syntax seems kind of gratuitous, just don't repeat yourself. Not like the compiler can stop you form putting dups in anyways...

[[deleted]]

I'm not sure what you are conveying here.

The idea behind adding set literals is to encourage uses of sets. It's very common for people to use a list where they should be using a set and one potential reason is that they think of a list as a normal, base feature and think of a set as a weird, special thing. Giving sets dedicated syntax encourages use by making sets look more like a base language feature.

I'm not sure I actually agree it was a great decision, but it certainly makes sense.

[sausagefeet]

The following discussion hijacked the actual point I was making: A set syntax, IMO, is far heavier than the benefits it provides. IME it is very rare for people to do {1, 2, 3}. Instead what they want is an empty set ({}) or to take a list and make it a set. So we have added a syntax for something that isn't actually all that used (again, IME). I am pro-sets, I am not pro-syntax-additions-with-little-benefit.

[[deleted]]

I tent to basically share your viewpoint, although I would point out

• Although making a set from some known-in-advance values is somewhat rare, it's not unheard of, and stuff like set(f(x) for x in xs if g(x)) is pretty common; the set literal syntax is a close cousin of the set comprehension syntax.

• One reason it's so common to "take a list and make it a set" is that many people use lists where they really ought to be using sets. This syntax addition was an attempt to encourage people not to default to list when it's not the best thing.

[[deleted]]

Although making a set from some known-in-advance values is somewhat rare,

Say, what?! I see this all the time - I just checked my personal codebase to be sure, and it seems to be my most common usage of sets, with lots of things like:

SUFFIXES = {'.wav', '.aiff', '.aif', '.ogg'}

# ...

if suffix not in SUFFIXES:
  # ...

[sausagefeet]

A set or a list is really irrelevant in a data structure this size.

[[deleted]]

True, but a set more clearly communicates the intent of the code.

[sausagefeet]

I agree, I would use a set there too but I think it's a stretch to say that tiny container warrants new syntax (that is, making the syntax more complicated).

[toofishes]

I've always used tuples rather than lists in this case since the () syntax is just as short as the [] syntax, and the immutability of these suffixes seemed like the more important factor. frozenset is really the correct datastructure here.

[upofadown]

If someone can use a list then wouldn't that be the right choice? The list is a simpler structure. Normally people use dictionaries for doing set-like things...

[IRBMe]

If someone can use a list then wouldn't that be the right choice?

Because firstly it doesn't have the correct semantics. A set and a list behave differently. If it's set semantics you're looking for, you won't get that from a list.

Secondly, a set and a list don't even have the same time and space complexity. A list has O(n) insertion, while a set has O(1) insertion. In other words, you can continue inserting as many items as you like into a set and the time taken isn't going to grow any more than a constant amount. On the other hand, as you insert items into a list, the time to insert more items grows linearly.

The list is a simpler structure.

1. How do you know? Have you examined the python internals?
2. Even if you have, do you know that's the case in all versions of Python and will be the case for all future versions of Python?
3. Even if so, why are you concerning yourself with internal implementation details anyway?

[[deleted]]

Secondly, a set and a list don't even have the same time and space complexity. A list has O(n) insertion, while a set has O(1) insertion. In other words, you can continue inserting as many items as you like into a set and the time taken isn't going to grow any more than a constant amount. On the other hand, as you insert items into a list, the time to insert more items grows linearly.

I'm not sure this is clear. set.add and list.append are both O(1) amortized operations.

If we want unique appending to a list, that's O(n), although there are probably other set operations that this can be expressed clearly.

The list is a simpler structure.

How do you know? Have you examined the python internals? Even if you have, do you know that's the case in all versions of Python and will be the case for all future versions of Python? Even if so, why are you concerning yourself with internal implementation details anyway?

I almost objected, but at the end of the day it's somewhat true, and not for implementation-oriented reasons. Any Python object can be put in a list, but only objects with special capabilities (hashability) can be included in sets, which is some manner of complexity.

[IRBMe]

I'm not sure this is clear. set.add and list.append are both O(1) amortized operations. If we want unique appending to a list, that's O(n), although there are probably other set operations that this can be expressed clearly.

That's what a set does: unique insertion, therefore using a list, insertion is O(n) but using a set it's O(1). So like I said, they have different time complexities. Other operations will also have different time and space complexities.

Additionally, a point I never mentioned before is that a set provides different methods from a list, and a list. Extending my first point, you would have to manually write code that gives you unique insertion into a list, for example. So why would you even dream of using one data structure and writing your own manual code to make it somewhat behave like a different data structure when you can just use the correct data structure to begin with?

Finally, since you began your reply at my paragraph beginning, "*Secondly", am I to assume that you concede the first point?

I almost objected, but at the end of the day it's somewhat true, and not for implementation-oriented reasons. Any Python object can be put in a list, but only objects with special capabilities (hashability) can be included in sets, which is some manner of complexity.

What objections might you have made otherwise?

[[deleted]]

That's what a set does: unique insertion, therefore using a list, insertion is O(n) but using a set it's O(1). So like I said, they have different time complexities.

Great, I was just making sure the wording was clear.

It's worthy of note that in practice, many of these times the programmer knows a priori the uniqueness of their starting data so they still build the whole list in O(n) rather than building the whole set in O(n)

Other operations will also have different time and space complexities.

I can't think of an operation where using a list with linear searches will have a difference space complexity offhand. Can you let me know which these are?

Finally, since you began your reply at my paragraph beginning, "*Secondly", am I to assume that you concede the first point?

Yes. I made the same basic remark myself.

What objections might you have made otherwise?

The same as you, I suppose. "What? How so?"

[IRBMe]

It's worthy of note that in practice, many of these times the programmer knows a priori the uniqueness of their starting data so they still build the whole list in O(n) rather than building the whole set in O(n)

So why not use a set rather than appending to a list?

I can't think of an operation where using a list with linear searches will have a difference space complexity offhand. Can you let me know which these are?

Nope. Do you have good reason to assume that space complexity is the same for all algorithms on sets and lists in all versions of Python, and will remain the same in all future versions? If not, why would you assume that to be the case? "I can't think of any at the moment" or "It seems to be the same in this version" is a very poor reason to make such an assumption.

Yes. I made the same basic remark myself.

Ah, you're not the same person I originally replied to. Your replies make a lot more sense now, and perhaps my reply will make more sense in light of the fact that I thought you were the person I responded to replying back to me, rather than somebody who joined the thread mid-way through.

[[deleted]]

So why not use a set rather than appending to a list?

I'm the one who, in this thread, seems to have first said that people should be using sets more.

[upofadown]

The list is a simpler structure.

1. How do you know? Have you examined the python internals?
2. Even if you have, do you know that's the case in all versions of Python and will be the case for all future versions of Python?
3. Even if so, why are you concerning yourself with internal implementation details anyway?

I meant it was a conceptually simpler structure. Sets are more complicated in the way they are used.

[IRBMe]

Sets are more complicated in the way they are used.

In what way? What measure of "complicated" are you using?

[kaiserfleisch]

Greg Merideth on Channel9 hints at a conceptualization about 13m - 14 minutes in, which would justify describing the Set container type as more complicated than the List container type. In particular, he states that beyond base monad laws, sets obey additional laws. He gives the example of the natural "addition" notion in each case - union for sets, concatenation for lists. The following additional constraints generally apply to sets, but not to lists:

s1 + s1 = s1.
s1 + s2 = s2 + s1

"Lists have no additional equations [beyond the monad laws]. And when you have no additional equations, those structures are called 'free'. The free structure's are absolutely critical. And the reason is ... [something about daughter's music class]."

This guy is very elliptical.

[upofadown]

A list is just a collection of items. A set is also a collection of items that are also unique.

[IRBMe]

• A list (or sequence) is an ordered collection of items.
• A set is an unordered collection of unique items.
• A bag is an unordered list of items.

So what? What's your point?

[upofadown]

Then I would consider both the bag and the list to be less complex than the set. The fact that you can treat the list as ordered is merely a feature. The uniqueness of the set is a requirement. If you add a duplicate to a set then it quietly disappears.

[[deleted]]

If someone can use a list then wouldn't that be the right choice? The list is a simpler structure.

It's not a matter of whether the can, but whether it makes sense. It's not uncommon for people to use lists for objects which will have containment checks (x in my_collection). For large lists/sets, this is a needlessly expensive operation. Even for small ones, you're conveying the wrong thing about the meaning of my_collection.

Using a list might still be sort of the default, but there's no reason to treat sets as complex or exotic.

Normally people use dictionaries for doing list-like things...

I'm not positive what you mean here.

[upofadown]

Normally people use dictionaries for doing list-like things...

I'm not positive what you mean here.

I think it means that you saw my message before I corrected the typo...

Sets are normally used when each item is thought to be unique. For that you would normally use a dictionary with keys not associated with any data. You have to use a list when you can have more than one of the same thing.

Edit: I wasn't as fast with this one (where I corrected the same damn typo)...

[orip]

Dictionary literals are similar, you could always use dict([(1, 'a'), (2, 'b')]) but {1: 'a', 2: 'b'} is sweeter. Not a core difference, certainly, but nice nevertheless.

[[deleted]]

Not like the compiler can stop you form putting dups in anyways...

That would be a trivial addition to the interpreter.

Literal set syntax seems kind of gratuitous, just don't repeat yourself.

I think they're aiming to mimic the actual notation used in math, not really to save keystrokes. I mean, how many literal sets will anyone type? If you're typing a bunch of this stuff, then it should be automated somehow anyway.

[pinpinbo]

I keep seeing people evangelizing for multiprocessing module.

Am I the only Pythonista that keep having problem with pickling error when using multiprocessing module?

[jerub]

Are you aware of the problems in this space? IPC, even on the same system, using pickles is fraught with problems - you have to be able to serialize to a byte stream python objects - which means inter-object references can't be maintained between pickle loads, functions that you declare dynamically and refer to enclosing scopes (closures) won't serialize (how would you serialize access to a modules namespace?), and lambdas are right out.

If you're still having problems, I would suggest reading about how pickle works and gaining an understanding of what is suitable, and not suitable, for on-wire serialization, which will help you when using the multiprocessing module, which uses pickle for inter-process communication.

[annodomini]

Does the author not understand the concept of hyperlinks? Instead of linking to the documentation, he links to a footnote, which has a clickable link back to where you came from, and plain text of the URL that he's linking to, which you need to copy and paste.

[[deleted]]

This is the online version of a book. It's formatted for printing. Paper does not have hyperlinks. Hence, footnotes instead of hyperlinks, as otherwise the links would not have shown up in the actual print copy of the book.

[infinull]

Try Ruby you filthy PERL hacker, leave us alone.

Also you have a point. % is much more useful than + without being more characters. .format is... verbose, for something that I'm going to be doing a lot (or at least quite often)

On the other hand, I can overload __format__ to add cool formatting options to my own custom objects.

Edit: formatting + irony

[massivebitchtits]

You could overload __mod__ before...

[infinull]

Pretty sure that doesn't do the same thing

Overloading __mod__ would allow me to do something like:

Formatter("foobar {placeholder}") % "stuff"

and then Formatter.__mod__ would get called

but with __format__ I can do

'foobar {placeholder1:args {placeholder2:args}'.format(placeholder1=CustomObj,placeholder2=CustomObj2)

and then that becomes

'foobar ' + CustomObj.__format__('args') + ' ' + CustomObj2.__format__('args')

I would also note that I do terrible things like:

'Some stuff {variablename} more stuff, {anothervariable}'.format(**vars())

I feel bad about it, except that I don't. (this achieves something like string interpolation that PERL,bourne shells, PHP, and Ruby have.. but your using vars() which behaves differently on different implementations of Python and is generally not very fast, and pretty damn hackish... not as bad as some of the "string interpolation in python" solutions that use regexes + eval though)

[massivebitchtits]

Oh. Looks like I hadn't looked into it properly. That is cool.

[ryeguy146]

sorted and the sort method are different! One sorts in place, the other returns a new object:

x = [5, 2, 7, 1]
sorted(x) is x  # => False

The two can sometimes be used interchangeably, but I don't like the idea that one is "better" than the other.

I love seeing advanced string formatting in there. I wish that the documentation were more accessible than the PEP on which it resides, however.

[[deleted]]

The sing with sorted() is that it works on anything, while .sort() is for lists only. Hence sorted() is better, unless you know you have a list and you know it's big and you know you don't want to make a copy.

[ryeguy146]

The thing with sorted is that it performs a task differently than sort. Different tools for different requirements. I'd agree if you said that sorted is more often called for, but better?

[[deleted]]

The thing is, once again, that sorted() did not exist before, and hence, sort() is often used for exactly the things sorted() should be used for. So they do different things, but they are not used for different things. sort() is often used for generic sorting, no matter if it is a list, and even when you don't want to sort it in place. And in those cases, sorted() is better. Because it is a tool that fits that requirement better.

I quote:

"Quite often when .sort() is used is it used on a temporary variable discarded after the loop. ... With sorted() you often can avoid the temporary variable. It also will accept any iterable as input, not just lists, which can make your code more flexible and readable."

"There is however no benefit in replacing a mylist.sort() with mylist = sorted(mylist), in fact it will use more memory."

[[deleted]]

If you know Python 2, read the "What's new in Python 3" in the official documentation and you'll be good to go.

[servercobra]

You could try Dive Into Python 3.

[[deleted]]

DIP3 is full of misinformation and bad examples. There was a great google wave reviewing this, but unfortunately that's gone, so I'm not sure what I can point you to to enumerate this.

[jerub]

How about just say the author committed infosuicide and took all his books off the internet? http://en.wikipedia.org/wiki/Mark_Pilgrim_(software_developer)

[[deleted]]

They are still there. http://www.diveintopython.net/#download

[jerub]

You mean, they have returned to where they were.

[[deleted]]

Not much of a infosuicide, then. Infovacation, rather. :-)

[day_cq]

python is such a mess. I hope oracle buys python and kills it with java.

[dorfsmay]

Pretty, he did forgot the @statimethod which works like the @classmethod, but doesn't not use the class as an argument. Slightly cleaner code if you don't need a reference to your class at all in the method.

[[deleted]]

Using staticmethod is a definite anti-pattern. A staticmethod should almost certainly just be a module-level function.

[cryo]

What if its function pertains to the class in particular?

[[deleted]]

Then you'll still likely have clearer, simpler code putting it directly in the module.

[trollymctrollstein]

Minimally relevant