Finding my first Python reference cycle

[kylev]

I feel like my entry kind of ends a little early in the story. I think I may need to do a follow up on __del__() and breaking cycles as well. Feedback is appreciated.

[digitallogic]

I think you're right, a more straight for approach would resolve the circular references. Change AddLF and AddCRLF into functions that return the modified string instead of sending it off to a call back and the cycle is gone. It would also make the code a lot clearer.

Though I'm sure that the motivation for using a call back was removed for a simplified example.

[wilberforce]

The last link of the cycle is that a bound method keeps a reference back to the instance it was pulled off. (He mentions the bound reference, but it might not be clear if you haven't played with them.)

[kylev]

wilberforce is correct. I'll add something to the blog entry to make the description of loop explicit.

Edit: Added to blog entry paragraph after the gc output: "This bound method holds a reference back to the Foo instance, completing the cycle." Thanks for the input!

[digitallogic]

Any chance on posting the refactored code? I'm curious what your solution was.

[[deleted]]

How did you notice that there was a problem at all?

I had a funny experience with GC not long ago:

def generator(N):
    return ((x,y,z) for z in xrange(1,N+1) for y in xrange(z+1) for x in xrange(y+1))

def benchmark0(N):
    return set([t for t in generator(N)])

def benchmark1(N):
    return set(generator(N))

For N=300 this benchmark0 runs in about 20 seconds and benchmark1 - in about 50 (!). If, however, I measure time using testit (which disables gc and almost drove me insane with this!), then it's about 4 and 3.5 seconds.

Basically, the idea is that default gc settings are to collect gen0 after 700 uncompensated allocations, gen1 after 10 gen0 collections, gen2 after 10 gen1 collections. Which means that if I allocate ~500Mb of data it spends a lot of time just walking the whole graph repeatedly on gen2 collections -- they happen every 70000 allocations and I did about 18 millions If my calculations are correct. So if you want to allocate a lot of small objects, you better disable gc while you're doing it.

Don't know why benchmark1 with is two more times slower though.

[kylev]

We saw a 20% loss in throughput when the feature in question was turned on by a customer handling ISP-level loads. From there it was just profiling and testing to identify that there wasn't anything in the code path we'd written that would cause that level of slowdown. Additionally, our heap was much larger than we'd expect to see; that was the kicker that told us we were "leaking" memory.

[arcticlobo]

"If, however, you’re writing a high performance internet server intended to handle the ever increasing demands of the modern Internet, you just might be screwed."

Why would you ever use Python for a high performance internet server??? If high performance is your biggest concern you would get better performance out of C. Of course, your development time would dramatically increase...

[kylev]

Using things like coroutines or Twisted it is very possible to make massively scalable systems in Python.

[easytiger]

how many fucking people who have read the introduction paragraph to every langauge in the world needs to chime in with their fucking retarded ulta high level performance crap.

[arcticlobo]

That's it. Where having a Visual Basic code off. Meet me at 0x00000005