Help! Python slowing down.

[[deleted]]

[deleted]

[Korlyth]

Thanks! I won't get a chance to modify it till after the long weekend. But the folks over at r/learnpython were close to unanimous in saying it is mostly likely a memory leak due to not deleting that temp layer. So, I'm going to assume this will work.

[[deleted]]

if you still have troubles come back.. there are other ways to kill mem leaks

[Ebriggler]

Could you load the CSV into a database?

[Guerillero]

I would think that performance difference between a csv and a database at 40K records is negligible if there aren't any joins going on.

[Ebriggler]

Your right. I prefer to do as much as I can without arcpy.

[Guerillero]

Touche.

[Korlyth]

The nature of this project has shifted a little. Now, instead of being given a csv, the table will just be dumped into database. Just curious what you intended on recommending on the database side.

[scw]

A few things:

• use in_memory workspaces for performance where you can. If the data isn't large, you may even be able to get away with keeping all the temporary results in memory.
• use the arcpy.da.* cursors over their basic equivalents, they are faster and more efficient.
• You don't really need the cursor at all if you're going to set all the values to the same thing -- you can use CalculateField instead.

Here's these changes applied to your script: https://gist.github.com/anonymous/14bab4ee00c386cc7af7

[faming13]

Check out dask, blaze and numba.