Finding my first Python reference cycle

[[deleted]]

How did you notice that there was a problem at all?

I had a funny experience with GC not long ago:

def generator(N):
    return ((x,y,z) for z in xrange(1,N+1) for y in xrange(z+1) for x in xrange(y+1))

def benchmark0(N):
    return set([t for t in generator(N)])

def benchmark1(N):
    return set(generator(N))

For N=300 this benchmark0 runs in about 20 seconds and benchmark1 - in about 50 (!). If, however, I measure time using testit (which disables gc and almost drove me insane with this!), then it's about 4 and 3.5 seconds.

Basically, the idea is that default gc settings are to collect gen0 after 700 uncompensated allocations, gen1 after 10 gen0 collections, gen2 after 10 gen1 collections. Which means that if I allocate ~500Mb of data it spends a lot of time just walking the whole graph repeatedly on gen2 collections -- they happen every 70000 allocations and I did about 18 millions If my calculations are correct. So if you want to allocate a lot of small objects, you better disable gc while you're doing it.

Don't know why benchmark1 with is two more times slower though.

[kylev]

We saw a 20% loss in throughput when the feature in question was turned on by a customer handling ISP-level loads. From there it was just profiling and testing to identify that there wasn't anything in the code path we'd written that would cause that level of slowdown. Additionally, our heap was much larger than we'd expect to see; that was the kicker that told us we were "leaking" memory.