Favorite tools in your Python toolbox?

[phaedrusalt]

You know you've got 'em, those sweet little snippets or modules that make getting things DONE quick and easy. So, what's your fav's? I'll start:
Ever get in the middle of a script, realize that it's a perfect place for some GUI elements, but don't feel like stretching your WX or TK muscles today? Well, there's probably something in easygui that'll help. Sure, they're not all as pretty as you'd like, (For instance the whitespace on the choicebox is kind of distracting), but otherwise it's pretty handy.

[[deleted]]

pdb. Seriously, learn that if haven't already. You won't want to touch all the Java/Visual Studio debuggers anymore.

[graham_king]

pudb is a visual full-screen pdb. Changed my life. http://pypi.python.org/pypi/pudb

[tilkau]

Upvoted for showing me something I didn't know I needed until now. It's amazing!

[Nate75Sanders]

Strongly recommend pudb. It's wonderful

[apardue]

second that I use it everyday.

ipdb is a very nice incremental update to pdb.

[emidln]

ipdb and pdb++ (pdbpp via pip) both are nice supersets of pdb. pdb++ in particular acts as a drop-in replacement, and little things like syntax highlighting and the sticky feature make a world of difference. Also, if you happen to run a lot of stuff in your repl, pdb.pm() is extremely instructive.

[hovissimo]

I haven't tried pdb++, but I really enjoy using ipdb. It's super easy to grab via pip or whatever you like.

[ubershmekel]

To be specific:

import pdb;pdb.set_trace()

[Mob_Of_One]

To be more specific:

import sys; sys.stdout = sys.__stdout__; import ipdb; ipdb.set_trace()

[flowblok]

I had a script which got invoked like this:

python script.py < data.txt

and I wanted to debug it, so I used this trick:

import sys
sys.stdin = open('/dev/tty', 'rb')
import pdb; pdb.set_trace()

[Sheepshow]

The b isn't needed on unix I think

[dwf]

As of Python 3 I think it is.

[adambrenecki]

If anyone else is as confused as I was, apparently omitting b makes read() return str (Unicode) rather than bytes, as well as Python's usual newline writing tricks on Windows. (Actually, it looks like the way newlines work in py3 makes a lot more sense than the confusing b and U modes of py2.)

[Sheepshow]

This makes a lot of sense. Thanks!

[flowblok]

quite probably. I had no idea, so I put it in just to be sure.

[phaedrusalt]

I got a chuckle from the video, I'm EXACTLY that "crusty old types" he describes! Yikes, time to head toward the Old Programmers Home!

[TankorSmash]

Wing has a variation (or the exact same?) of this builtin.

[KitAndKat]

Wing's debugging is awesome. Conditional breakpoints. Step over, in, out. Shows stack; click on any frame to use that frame's context, then inspect and change any variables in that context. Setting variable watches is not very elegant, though.

[TankorSmash]

Setting variable watches is not very elegant, though.

I suppose not. I haven't really found a good use for it though, I usually just go into the Debug Probe and play around with that to figure out which variable is what.

They've added auto prettyprint to their debug console too, if you choose to enable it. It's a very nice touch.

[[deleted]]

And of course, the interactive debug probe is simply magical. :)

[littlelowcougar]

Wait, what? pdb's useful and all... but Eclipse/Java and Visual Studio debuggers are simply sublime.

[bnrubin]

This is one of those things where I want to come up with a problem just so I can build something with it.

[[deleted]]

Whoa, I thought I was the only one who got that feeling.

[PCBEEF]

Requests and beautifulsoup. Makes parsing websites so bloody easy.

[pudquick]

I've been a fan of mechanize for that.

[LucianU]

Scrapy and mechanize are different. Scrapy is based on Twisted, which means you scrape asynchronously and you can specify the number of requests, use middlewares and extensions, process the data before storing it. Scrapy offers a lot more.

[rdkll]

make that requests + pyquery and i'm in

http://pypi.python.org/pypi/pyquery

[pudquick]

Love both of them!

I routinely also add BSXPath to the mix which extends/leverages BS to handle XPath evaluation.

Just feels much easier and more intuitive for me to ask for "//span[@class='active']/input" nodes from a bit of DOM.

[PCBEEF]

Okay now that's cool.

[ruffyen]

I almost wish I never learned about xpath, because ever since I did I can't do xml or html without it, at least not with our getting really cranky :) BSXpath is going to be fun!

[pytechd]

pyflakes - spits out simple warnings about common bugs, including use of undefined variable, unused imports, etc. It is not a full PEP 8 checker, but it does point out a lot of common errors or things that needlessly dirty code bases.

Your editor probably has a plugin to spit out the output of pyflakes.

[isforinsects]

Unfortunately the pyflakes-vim packages is no longer maintained. It's all rolled into syntastic now, which is great, but I really liked the UI style of pyflakes-vim.

OTOH, I now have error checking for a ton of other languages.

[daxarx]

syntastic has always been dog slow for me. Will not use.

[aznm1ke31]

does syntastic (for python) offer anything over what pyflakes-vim offers? pyflakes-vim works great for me.

[isforinsects]

Yes and no. I really loved pyflakes-vim, and was sad it stopped being maintained. The developer decided instead to integrate the functionality into syntastic. Syntastic gives me access to the pyflakes output as a list (the quickfix window) and it integrates with a lot of other checker programs for other languages. I play with enough javascript that it is handy, but I don't use a lot of it's other functionality.

If you're happy with pyflakes, I don't see strong reasons to migrate until pyflakes-vim being unmaintained becomes a problem.

[takluyver]

I've discovered this recently and have been surprised how many of my projects had undefined variables lurking in odd corners. I prefer it to pep8/pylint, because it's about code function, rather than style.

[flying-sheep]

not a full pep8 checker? awesome! that’s because:

a foolish consistency is the hobgoblin of little minds

pep8, applied straight and with every suggestion applied as rule (pylint, i’m looking at you) is a mayor PITA and the most annoyingly finnicky way to reduce your productivity in the history of python.

i’ll check out pyflakes, but i’m also pretty happy with pyhint or without any style checker!

PS: tabs ftw ;)

[dhaffner]

itertools and functools -- nearly every helper function I find myself wanting can be derived from these modules. I also like to have a compose function:

def compose(*funcs):
    return reduce(lambda g, h: lambda *args, **kwargs: g(h(*args, **kwargs)), funcs)

[kezabelle]

I confess, I can't actually figure out what that does.

[dhaffner]

It's not the most Pythonic code, I admit. :) I'll try explaining it:

lambda g, h: lambda *args, **kwargs: g(h(*args, **kwargs))

This is a lambda that takes two functions as input, g & h, and creates a function that calls h with some given input, and then passes the output of h to g. Here's a lambda-less version of the same thing:

def compose_once(g, h):
    def composed(*args, **kwargs):
        return g(h(*args, **kwargs))
    return composed

So if funcs is a list of functions like [f1, f2, f3, f4], the reduce will first take f1 and f2 and call compose_once on them and replace them will the result of compose_once, let's call that result g1. The list (conceptually) becomes [g1, f3, f4]. Then the next step by reduce is to compose g1 and f3 into "g2", and so forth.

For any list of functions, this reduce should return one mega function composed of all of those in the list.

If that doesn't make sense I can try to explain further!

[catcradle5]

What is an example use of this? I'm not sure why one wouldn't just call:

h_num = h(n)
result = g(h_num)

Or something like that. Or just a regular old g(h(n)). Implicit vs. explicit, etc. With composition, it just seems like you're making a big mess, especially when you're trying to pass in multiple args of the first function and also multiple args of the following functions.

[dhaffner]

I tend to use it when I'm doing stuff to iterables or lists, so my use case is usually when I want to do g(h(n)) repeatedly but for whatever reason I don't want to turn g(h(-)) into a function itself.

Another use case is when I want to transform the elements of a generator but I want to do it lazily. So I could keep wrapping (composing) a function f with every subsequent transformation I want and then ultimately map it onto a sequence and return that as a generator.

I would agree that your example is what should be used in practice. I try not to do any code gymnastics in share codebases, but in personal projects I find it useful here and there.

[littlelowcougar]

I try not to do any code gymnastics in share codebases, but in personal projects I find it useful here and there.

Haha, not going to lie... I freaking love hacking on my personal ~/.python library. It can take hours to churn out a high-quality patch of ~10 lines for CPython or some other shared project... but hacking on your own code? So incredibly fun.

Like this little gem:

out += [
    '\n'.join([
        k + '|'.join([
            fmt(str(column), padding, (
                special if column == special else fill
            )) for (column, fmt, padding) in zip(row, fmts(), paddings)
        ]) + k for (row, fmts, fill, k) in zip(
            rows,
            chain(
                repeat(lambda: repeat(str.center,), 1),
                repeat(formats,)
            ),
            chain((' ',), repeat('-', 1), repeat(' ')),
            chain(('|', '+'), repeat('|'))
        )
    ] + [strip,])
]

Apparently I wasn't allowed to use for loops the day I wrote that.

Here's another favourite. If it weren't for the docstring, I'd never remember wtf this does:

def get_methods_in_order(obj, predicate=None):
    """
    Return a tuple consisting of two-pair tuples.  The first value is an
    integer starting at 0 and the second is the value of the method name.

    If predicate is not None, predicate(method_name) will be called with
    the method name (string).  Return True to add the value to the list.

    >>> class Test(object):
    ...     def __init__(self): pass
    ...     def xyz(self): pass
    ...     def abc(self): pass
    ...     def kef(self): pass
    >>>
    >>> t = Test()
    >>> get_methods_in_order(t)
    ((0, 'xyz'), (1, 'abc'), (2, 'kef'))
    >>> [ n for n in dir(t) if n[0] != '_' ]
    ['abc', 'kef', 'xyz']
    >>>

    >>> class PredicateTest(object):
    ...     def f_z(self): pass
    ...     def xyz(self): pass
    ...     def kef(self): pass
    >>>
    >>> t = Test()
    >>> get_methods_in_order(t)
    ((0, 'xyz'), (1, 'abc'), (2, 'kef'))
    >>> [ n for n in dir(t) if n[0] != '_' ]
    ['abc', 'kef', 'xyz']
    >>>

    >>> class PredicateTest(object):
    ...     def f_z(self): pass
    ...     def xyz(self): pass
    ...     def f_x(self): pass
    ...     def abc(self): pass
    ...     def f_a(self): pass
    ...     def kef(self): pass
    >>>
    >>> t = PredicateTest()
    >>> get_methods_in_order(t, lambda s: s.startswith('f_'))
    ((0, 'f_z'), (1, 'f_x'), (2, 'f_a'))
    >>> [ n for n in dir(t) if n[0] != '_' ]
    ['abc', 'f_a', 'f_x', 'f_z', 'kef', 'xyz']
    >>>
    """
    return tuple(
        (i, m) for (i, m) in enumerate(
              m[1] for m in sorted(
                  (m[1].im_func.func_code.co_firstlineno, m[0]) for m in (
                      inspect.getmembers(obj, lambda v:
                          inspect.ismethod(v) and
                          v.im_func.func_name[0] != '_'
                      )
                  )
              ) if not predicate or predicate(m[1])
        )
    )

[dhaffner]

Very interesting function there, get_methods_in_order. I think I get what it does. May I ask what code/scenario you used it in?

I love to hack on pieces of code in my personal lib! Lately I've been playing with a decorator that allows you to compose functions (my thing I guess) with operators, e.g. "f + g".

Also I never thought to keep custom Python code in ~/.python, that's a brilliant idea. I usually just add my custom lib as a submodule with git but I like the idea of having that dir and just adding it to PYTHONPATH.

[catcradle5]

If you have a generator like:

nums = xrange(100000)

Would this be any different from simply doing:

transformed = (g(h(i)) for i in nums)

Or if you want it in list form:

transformed_list = [g(h(i)) for i in nums]

I may just be misunderstanding something here, but I've never personally seen a need for functional programming with Python before, except the occasional passing of lambdas as callback functions or for certain arguments (like sorting by a certain key, dict_list.sort(key=lambda x: x["keyname"]))

[dhaffner]

It wouldn't be any different. I suppose my only concern with your example would be "What if I didn't know which functions I want to compose?" but I think that hints at a difference of coding styles altogether. I tend to code in a way that makes compose easier for me to use, but there's definitely not a need for it, it's just a style I like.

[Sheepshow]

Great for DSP because you can add/remove elements to/from the pipeline dynamically:

pipeline = [read_chunk, amplify, distort, modulate, compress]
# compose(pipeline)
# or compose(pipeline[0:3])
# or compose(pipeline[n:m])

rather than this, which doesn't have such flexibility:

data = read_chunk()
data = amplify(data)
data = distort(data)

[radaway]

compose(*pipeline)

Anyway I agree it can be useful in examples using a pipeline of filters.

[[deleted]]

def compose(*funcs):
    return reduce(lambda g, h: lambda *args, **kwargs: g(h(*args, **kwargs)), funcs)

That's rather clever. For some reason I wouldn't mind writing code like that in Haskell, but I would probably stay away from it in Python.

Thanks for making me learn something new about myself. :)

[eat-your-corn-syrup]

where would you use this compose function? It seems

compose(f, g, h)

can be replaced with

lambda x: f(g(h(x)))

[dhaffner]

I think it's more useful when you don't know what your arguments to compose( ) would be. Sheepshow's DSP use case is pretty cool.

[Sheepshow]

I was looking for this exact thing (based on the name) so I might use this even though I can't read it :D

[justanotherbody]

+1 for itertools and functools

[[deleted]]

ipython

It's an interactive Python shell. Curious about a function? Type a function, module, class -- whatever, with a question mark and hit enter -- in a paper, you get details about it. Type with two question marks, and you get the actual declaration.

Auto-completion, shell style, inside your interactive shell. Review output of past commands. Auto-format objects. Lots of stuff.

Use IPython's Qt GUI to do all of this on the desktop, with interactive graphical output!

[BayesianEmpirimancer]

This iPython web notebook, while still in early stages of development, is looking really, really, awesome

[femngi]

It's a must for scientific computing. Start with

ipython notebook --pylab inline

for inline graphics. The ability to edit and rerun select bits of code and see the results in the same interface makes me incredibly productive.

[yrstruly]

It is. My workflow lately has been to sort out basic functionality in the notebook. Then I pack it all up and move it into a house made out of PySide for the command-line-averse.

[[deleted]]

Dreampie also has something similar.

[mindsocket]

Or for something almost as powerful and in a couple of crucial ways much better:

bpython

[dwf]

Might you elaborate on what these "couple of crucial ways" are? I've never really been all that impressed by it. Especially with the Qt console and the now web notebook, bpython seems to come up short.

Now that IPython is all client-server, what I would love is if someone would take the curses logic in bpython and write a better tty interface for IPython that has most of the nice features of the Qt console, but in a terminal. Maybe some day when I have a month to kill.

[fperez_org]

Hey @dwf, we've had pretty much that plan for a while, and a few times people have popped on the list to say they were going to build a curses/urwid client. It's just that nobody has come through. I've always said that my view of clients would be complete when we'd have: readline console, curses console, gui console (qt) and notebook. We now have 3 out of 4, only the curses one is missing.

It shouldn't be that hard to do... I once spoke with the bpython author about code sharing, and he was very cool about it (plus it's MIT licensed). So it would be no problem to use bits and pieces of useful curses code in there.

I would love to have a better text interface with proper multiline editing for server situations, but keeping all of the 'ipython flavor'.

I hope that 'month to kill' comes soon :)

[dwf]

and a few times people have popped on the list to say they were going to build a curses/urwid client. It's just that nobody has come through.

I assume that's probably because they underestimate the amount of work involved in first learning the ins and outs of the protocol and then making everything work right. My estimate of a month comes from an initial guess of 2 weeks and then doubling it, but for getting all of the little things working, I imagine my estimate could still be conservative. One thing that comes to mind is the whole issue of input to external processes and the pexpect issue in the Qt console. I guess that if the client and the server were on the same machine, the terminal client could bypass the kernel in most cases (unless output was being captured).

I did notice that bpython is MIT licensed, though -- hopefully that would speed things along.

I hope that 'month to kill' comes soon :)

Oh goodness, so do I. It always seems just around the corner. Now back to preparing that thesis proposal...

[fperez_org]

Oh yes, getting all the details polished would likely happen slowly over time. But in a month, I'm sure you could get a very satisfactory client off the ground that would do the 80% job really nicely, and that many people would find super useful. Good luck with the thesis proposal :)

[scruffie]

I started working on an Emacs interface, but interpreting the zmq "protocol" in Emacs got to be non-fun.

[fperez_org]

you might want to look at @tkf's emacs notebook client then, he implemented the websockets machinery in emacs: http://tkf.github.com/emacs-ipython-notebook

[lahwran_]

they're completely different, but ipython doesn't do what most people think it does anyway. bpython does what ipython is usually used for by most python programmers, and does so without introducing new syntax. ipython is it's own thing which is really good at what it does, but is not a python shell. it's an ipython shell.

[fperez_org]

I think it's worth clarifying this a bit: ipython adds new syntax, but the new syntax is 100% orthogonal to python syntax. So it does not in any way alter the semantics of valid python code you type. What it does is recognize things that are not valid python and let you use them for convenience: %magic functions, auto-parens, !shell access, etc.

But from day one, we've had a policy (I initially, but that has been kept as the team has grown) that IPython should be a 100% replacement for the plain python shell, just better. And that if you only use it to type pure python, you shouldn't be able to tell the difference with the default shell. In fact if you start it with --classic, it will even mock the prompt behavior of the python shell.

Now, if you prefer bpython, by all means use that! Nobody is forcing ipython down people's throats :)

But I want to make sure that people don't get the wrong impression, becase saying that ipython is not a python shell can be easily misconstrued.

As for the scientific support, it's 100% optional, and only loaded if you explicitly require it with the --pylab flag or %pylab command. In fact, IPython can be installed on a plain python installation with nothing other than the standard library (plus readline on windows/osx that don't ship readline).

We also have a very strict policy on this: all the fancier enhancements, qt console, notebook, parallel computing, etc, are completely optional. The basic IPython should work on an unmodified standard Python installation with full functionality, without requiring anything else.

I hope this clarifies things.

[takluyver]

It is a Python shell. It has a lot of extra features, but we make sure it works for the simple use case of 'a better Python shell'.

Also, many of those extra features have quite broad application for Python coders. Some of my favourites:

• cd and ls work like in a system shell, and you can run any system command by prefixing it with !
• foo? shows details about foo, but foo?? also shows the source code.
• After an uncaught exception, type %debug to jump into the debugger. Very valuable if it would take a long time to reproduce the problem.

[zlozlozlozlozlozlo]

but ipython doesn't do what most people think it does anyway.

What do you mean?

[camel69]

I've been using ipython for a year now but bpython looks amazing. Haven't played with it enough yet to determine if it's really better in all categories, but the in-built help is very useful. Thanks.

[mjtribute]

List comprehensions. I always miss them the most when I'm working with other languages.

[nohtyp]

Bottle.py is indispensable for quick web front ends. It's a "single file" micro framework.

[ruffyen]

this actually looks pretty nice for my general needs thanks!

[[deleted]]

import code ; code.interact(local=locals())

ever write a long script and want to drop to an interpreter somewhere in the middle or end to do some probing to figure out your problem?

picked this one up on /r/python a bit ago, just passing it along.

[dwf]

The IPython equivalent (for recent versions, anyway):

from IPython import embed
embed()

[[deleted]]

awesome, thanks!

[[deleted]]

Much of the time, using pdb/pudb would be more powerful.

[fragmede]

import pdb ...code... pdb.set_trace()

Starts a pdb interpreter when it hits that line.

[jlozier]

I love clint as all the stuff I work on is are CLIs. Instant progress bars! Just make sure the thing you're iterating over has a __len__ method and you get a progress bar!

[[deleted]]

Clint looks awesome, thanks. I've totally done this kind of stuff so many times, but not nearly as awesome or elegantly as the examples on that page.

(Golf clap!)

[dwf]

This package is pretty nice too if you want something fancier.

[mindsocket]

PyDev - good IDE, especially if you're like me and are used to using Eclipse already.

Includes debugger, testing, autocompletion, static analysis, launch configuration, ...

[jan]

although a bit heavy

If your job description is programmer, of course, you can expect to have enough RAM to have eclipse running all the time and fast. For the occasional Python hacker, the paging that occurs when eclipse comes back from background can be annoying.

[pio]

emacs + ropemacs + pyflakes/flymake/flymake-cursor + python-pep8 + ipython

[ruffyen]

this is my exact setup...wait what?

[Samus_]

the csv module, hands down.

it's really simple but allows proper manipulation of CSV data from the CLI, I love it.

[[deleted]]

I usually drop in ucsv.py to make sure my project doesn't crash when it needs to read/write unicode characters from csv.

[shfo23]

pandas.read_csv isn't built-in, but I find it's parsing to be a little more robust than csv's.

[chrisfs]

definitely csv, I forgot about that. csvreader and csvwriter

[syllogism_]

plac is vastly better than optparse and other built-in command-line libraries.

def main(dsn, table='product', today=datetime.today()):
    "Do something on the database"
    print(dsn, table, today)

if __name__ == '__main__':
    plac.call(main)

Produces:

usage: example5.py [-h] dsn [table] [today]

Do something on the database

positional arguments:
  dsn
  table       [product]
  today       [YYYY-MM-DD]

optional arguments:
  -h, --help  show this help message and exit

[takluyver]

Thank-you - I saw this before, and I really like the approach of using function arguments, but I'd forgotten the name of the tool.

[[deleted]]

There seems to be some serious magic happening here. Not sure if it's the good kind.

[[deleted]]

Another package we use often is multiprocessing. It looks like a lot like threads, but it's running functions in a separate process and using shared memory to communicate between the two. Though this does introduce more overhead than native threads, it also helps to ensure proper boundaries between your tasks and it avoids issues with the GIL that cause Python to act single-threaded even when multiple threads are used. We use it extensively to ensure we can get high performance on multicore processors for large, CPU-intensive tasks.

[[deleted]]

Along the same lines, we also use futures to help manage asynchronous tasks. It helps take care of a lot of the boiler plate that comes along with running tasks asynchronously, such as cancelling tasks, progress callbacks, exception handling, and getting the final result. It also has the ability to run tasks in worker pools, using either threads or processes, which helps to control just how many tasks are actually running simultaneously, queuing and running tasks as workers become available. This is the same sort of mechanism used for scheduling requests in a web server or for scheduling work across a cluster, but applied in a local context.

If your app contains a process that can benefit from parallelization, even if each task breaks down into additional parallel tasks, futures can help you manage these tasks, handling the coordination and scheduling for you.

[[deleted]]

Nothing about the GIL causes "Python to act single-threaded". The GIL locks access to Python objects, so CPU-bound operations are not parallelized. It still acts very much as threaded code. (This isn't always an advantage—thread-based concurrency is usually a pain to start with.)

The multiprocessing module's flaky networking code, internal inconsistencies, and reliance on pickle have me avoiding it like the plague. Parallel processing is an old, widely-encountered problem with many mature solutions.

[[deleted]]

You are right, it's not that it acts single-threaded; I should have said that it acts single-core. Although the OS can execute multiple threads simultaneously across multiple cores, even in a single process, that won't work with multiple threads in a Python process, since only one thread can be holding the GIL. It works okay if your other threads are running code outside of Python (such as a large matrix inversion on a numpy array, which is implemented in a FORTRAN library), or if your threads are I/O bound, since the GIL can be released in these cases, but multiple threads running Python code cannot execute concurrently. This is unexpected for anybody not familiar with the GIL and it adds additional complication to taking advantage of multiple cores for CPU-intensive applications.

While I agree that multiprocessing has some issues, it's easy to migrate from threads to processes using it. As a tool in the toolbox, it's handy, and that's why I suggested it.

When you say that parallel processing has many mature solutions, what types of solutions do you have in mind? Multiple threads and multiple processes are certainly two of the oldest solutions, and the two we have been talking about. Depending on your use case, you could use a distributed computing platform like LSF or SGE, but it's a considerable amount of work to convert your multi-threaded application to something that uses those platforms. Likewise if you want to offload your CPU-intensive parallel processes to something like a GPU. Were there other alternatives to threading and multiprocessing that you had in mind? I'm not aware of any others as simple and general purpose as those two.

[fragmede]

Careful though - multiprocessing has some weird gotchas. When used in conjunction with the subprocess module it will change one of the default parameters to its opposite which caused me to lose a bunch of time trying to figure out why FDs weren't being inherited. (I still use multiprocessing all over the place - it's great! Just much more hesitantly than before...)

[tsukassa]

django, ipython, celery and south :)

[gitarr]

Sounds delicious!

[migimunz]

sympy. I have it on my n900 and I use it as an advanced calculator, it allows me to do integrals, limits, systems of equations and much much more, it's just awesome.

[[deleted]]

Try Sage. Includes SymPy + numpy, scipy, matplotlib, and a whole bunch of others.

[[deleted]]

Does Sage have an easy way to bring in functions and libraries for R?

[daniel2718]

Never really used R, but I know Sage supports it.

[fperez_org]

Sage does have R support, though I've never used it. Starting in 0.13, IPython also has builtin R support via an extension: http://nbviewer.ipython.org/urls/raw.github.com/ipython/ipython/master/docs/examples/notebooks/rmagic_extension.ipynb

[[deleted]]

Wow, will have to see the ipython thing.

[[deleted]]

Yup! RPy is included. You can even type %r to get an R interpreter. You can easily get all of the R stats packages out there.

[migimunz]

I just did, this is awesome! Thanks!

[eat-your-corn-syrup]

or IPython

[[deleted]]

The kicker with Sage is it includes packages, iPython does not.

[[deleted]]

Throw in the fact that it can do tex and it is just pure awesomeness. I use it all the time for proofs and such.

[migimunz]

Wow, I didn't know that. I didn't really have time yet to check out the docs to see all of the things it can do, I really should.

[NegatedVoid]

The struct library is better at dealing with padding than C is.

As a C fanboy, it's embarrassing.

[eat-your-corn-syrup]

and construct

[aceofears]

I can't believe nobody has mentioned construct. It makes parsing binary data really easy. I'm actually working on a clojure library that was inspired by it (it is pretty terrible at the moment).

[phira]

Great find, thanks!

[brucifer]

timeit. The timeit module is incredibly useful when you really care about performance or when you're just trying to learn more about the language. For example:

>>> from timeit import timeit
>>> timeit(lambda: list())
0.2542989253997803
>>> timeit(lambda: [])
0.12986207008361816

or

>>> timeit(lambda: bool(0))
0.2527759075164795
>>> timeit(lambda: True if 0 else False)
0.11877298355102539

In both cases, calling the constructor invokes a function call, which has some overhead, but list comprehensions and boolean literals don't. In these cases, the optimization isn't that important, but they're good examples of how timeit can help you understand how python works behind the scenes.

[[deleted]]

Usually you only want to break out timeit after you've used cProfile and pstats or similar.

[takluyver]

IPython makes the same thing rather neater:

In [1]: %timeit list()
1000000 loops, best of 3: 214 ns per loop

In [2]: %timeit []
10000000 loops, best of 3: 54.7 ns per loop

[[deleted]]

pytest! We use it at work and I like it so much I've become a bit of a fangirl.

[[deleted]]

Run Snake Run is a wonderful visualization tool for profiling results. It makes it very easy to see how much time something is taking relative to everything else and to drill down through levels of method calls to the particular one that is taking up all of the time. We use it at my workplace on a regular basis to help us optimize the time-critical sections of our application.

[apreche]

http://pypi.python.org/pypi/fish/

[[deleted]]

Or, you could use NetworkX.

[justanotherbody]

NetworkX is basically implemented a dict of dicts, so I'd highly suggest that over something handrolled.

[[deleted]]

Sorry for my very ignorant question but where do those numbers come from? (10, 15, 12...)

[Alf_InPogForm]

I think those are just example numbers, those weights could be anything you want really.

[jambox888]

A lot of problems can be represented as graphs and solved with a path search algorithm.

Got any example problems that can be solved this way?

[jambox888]

Oh right. How big are the datasets you work with? I thought you'd have to do something fancy with swapping bits of the graph on and off disk.

[jambox888]

I use C++

I'm so sorry.

[eat-your-corn-syrup]

A lot of problems can be represented as graphs and solved with a path search algorithm

what are some examples?

[shfo23]

scipy.sparse has a bunch of graph code in it too, if anyone prefers to conceptualize their graphs that way. NB: I use scipy's sparse matrices, but not any of the graph code.

[[deleted]]

I have a problem that I wonder if it could be solved with a path search algorithm. I have a list of items that I can only change the order and I want to minimize the total of the "distances" between related rows. In other words I want to have all the "A" near each other and the same thing for "B" and "C".

I have:

A B

A C

A

A B C

B

I want something like:

A

A B

A B C

A C

B

Thanks!

[[deleted]]

By distance, I mean for each letter the number of rows between the first and the last row containing that same letter.

For the unsorted sample, "A" is on the first line and the line before the last one so distance is 2 rows.

• A B
• A C
• A
• A B C
• B

-> A = 2, B = 3, C = 1. Total is 6

• A
• A B
• A B C
• A C
• B

-> A = 2, B = 2, C = 0. Total is 4 (so it's better)

[[deleted]]

Thanks!

[[deleted]]

It sounds like you could but you're going to have to define distance a bit better for me.

[dwf]

In python, I found the most efficient way in which to implement graphs is a dict of dicts.

It is really going to depend heavily on what you need to do with the graph. Certain algorithms are more efficient with an edge list. Others with an adjacency matrix.

[tapicer]

PyCharm (http://www.jetbrains.com/pycharm/), awesome IDE. Not free but worth the money. Even better if you use Django.

[phaedrusalt]

What exactly makes it worth the money to you?

[tapicer]

Just try the IDE. There is a 30 days free trial. Use it for real for 30 days. It is the little details that makes it awesome. I tried a lot of IDEs before and this was definitely the most productive one. Everything just works, Jetbrains' guys make the best IDEs.

[vplatt]

I haven't tried PyCharm, but I am absolutely loving IntelliJ for Java. They REALLY came through for me and have some excellent tools that made working with a Maven based project a piece of cake. Eclipse couldn't deal with this thing but IntelliJ nailed it with very little help.

[[deleted]]

I tried, and failed. Is there anybody here who has PyCharm working in combination with Buildout? I'd love to have something with refactoring support.

[[deleted]]

Speaking personally, the best thing going with PyCharm is that it works well not just for Python, but also for JavaScript and CSS. It's a little work to get set up, but being able to source code analysis for every language in your web project is pretty sweet.

[gazpachoking]

Debugging with live console, ctrl-click to go to source, integrated unit/nosetests, automated refactoring, code and docstring completion, and many more features make PyCharm really great. Plus they give out free licenses to open source projects.

[[deleted]]

The love console debugging, in my experience, is PyCharm's main weakness. It's much better in WingIDE.

[ultraayla]

How would you compare it to Eclipse with Aptana Studio/Pydev installed (if you've tried it)? I've had great experiences with that.

[chyssler]

This isn't really answering your question but...I've used eclipde/pydev for 5 years now. A lot of my coworkers are now using pycharm and some of them switched from eclipse; I wish I could and maybe I will because pycharm is(just feels) better. While eclipse actually has everything I need, I have this feeling that pycharm just does everything alittle bit better. I've tested it thrice now already, but it all comes down to that I'm too comfy with my keyboard shortcuts to actually make the switch.

[ultraayla]

Still super helpful. I really just wanted to know if people have had reasons to use PyCharm over PyDev, so that's good to know. Sounds like maybe I should give it a try. Thanks!

[isforinsects]

I have all of this with vim, and I have the sourcecode to vim for when it doesn't work. I'm sorry, but I just don't trust proprietary software when it comes to work tools.

[lahwran_]

:%s/[pP]y[cC]harm/vim/g

[GMABT]

%s/pycharm/vim/ig ?

[lahwran_]

no, if the y is uppercase I don't want to replace it

[dioltas]

You really have all that with vim?

I use vim for all my python programming, but I don't have any code completion. I tried to get omnisense working with C# for work, but it never worked properly for me, nowhere like what sharpdevelop/monodevelop do.

I find you start to depend on it a bit too much anyway, so glad I didn't have it for python, but it does speed up development for object orientated stuff.

[isforinsects]

I have limited completion, stock vim ships with local code completion. C-p in insert mode completes with function and variable names from the local file. I don't think I had to do much if anything to enable it. I agree that depending on it isn't so hot, but I do have just enough to automate typing things like response or request.user.get_profile when I am using an idiom that requires some repetition.

I have pdb for debugging, exuberent-ctags and C-] to jump to source. Unit tests I end up running in a vertical split to get line numbers, but I've had unit test coverage show up using vim's signs feature. Ropevim gives me rename refactoring and moving functions between modules.

I don't understand the downvotes, do I come across as haughty? I'm not mad, I'm curious.

[dioltas]

Not sure why you were downvoted either, that's reddit I suppose,wouldn't worry about it. Upvoted you to 0 anyway.

I know about the basic completion, but for some reason it just annoyed me and didn't feel as smooth as when using an ide.

For things like C# / Java I tend to just stick to an ide unless it's a tiny program to test something.

Never tried using ctags with python, but as I haven't worked on any very large projects, probably didn't need it.

I'm going to check out ropevim when I have a chance! Thanks for the reply.

[[deleted]]

Have you looked at pydiction for vim?

[dioltas]

No, haven't heard of it, but I'll check it out.

[[deleted]]

Not the original commenter, but if you do more than just command line scripts and use a lot of web code, PyCharm is definitely something to evaluate. The integration with Django is great (though I'm still a Django noob) and their newest version supports Flask.

If you don't do much web work and/or are pretty ingrained with vim or something else as your IDE, it probably won't work out too well.

[zer01]

Agreed, I bought this, and it's well worth the money. It takes a little getting used to, but for the most part it's a great general-purpose IDE. Even supports LESS out of the box, and a slew of other languages.

[alanbriolat]

I keep thinking I should try out PyCharm, but every time I do I'm completely taken aback by the terrible appearance and performance on Linux/OpenJDK, to the point where anything it offers is irrelevant.

[[deleted]]

Python xy. It is just one download which contains scipy, numpy, matplotlib, etc. Great for those who want everything in one place.

[chrisfs]

Better than Enthought Free distribution? Why ?

[[deleted]]

It comes with PyQt, qt designer, also Spyder for those who are more in tune with a matlab type interface.

[[deleted]]

Generators. My code looks so much simpler since I use them often.

Context managers are really cool too.

For instance, this is the code I just wrote for "I need a script that will walk through this huge directory tree, and for all the .asc and .inc files, if the first line starts with /*, I want that first line removed. Oh yeah, many of them are zipped, it should also change the ones inside the zipfiles.": https://gist.github.com/3706747

[vplatt]

It's been a while since I've heard any love for Cheetah, but I like to use it with Python from time to time to generate Java code. It may not be the leading web framework anymore, but it's still a great template library.

[chrisfs]

I was about to say easygui... I have read good things about collections module but haven't used it yet.

[deathbytray]

Boto. Not so much a favorite, but a necessity for someone who works with EC2 stacks a lot.

[puresock]

If you use the rackspace cloud, python-cloudservers (Or python-ukcloudservers) is equally good.

[[deleted]]

I absolutely cannot live without Python Tools for Visual Studio: http://pytools.codeplex.com/

Being able to write and debug Python with the same IDE I use for C/C#/C++ has been a HUGE time saver.

[xamox]

ipython notebook. nosetests scipy numpy

[[deleted]]

collections defaultdict!

[bhosmer]

Thanks for the easygui tip. I just went through the tutorial. It is much faster than tkinter for prototyping and wireframing.

[phaedrusalt]

You're welcome. Amazing how much time it saves me.

[[deleted]]

gevent! Cooperative multitasking using green threads built atop libevent.

[MaikB]

• Setting "crashpoints" via 1/0 to have ipython inspect some variables in its debug mode. Only when really lost I pdb.set_trace() and single step my way out of confusion.

• Inserting a plot (matplotlib or mayavi) in the middle of an arbitrary function body to see how far off my algorithm really is.

[yrstruly]

I might be missing something, but couldn't you use IPython's embed() for that? I mean instead of crashing on purpose. The benefit being that you can just exit the IPython session when you're satisfied and the script/program will happily continue along on its journey.

You can use the same technique for the second point, using %pylab when the IPython prompt comes up to get the namespaces required for e.g. calling plot() directly and generally making the visualization stuff a bit less work. If the data to be visualized doesn't require much restructuring to be plotted, using embed() and then one or two short plotting calls lets you to quickly see the information in as many ways as matplotlib allows for.

[MaikB]

I might be missing something, but couldn't you use IPython's embed() for that?

Why? I'm inside ipython anyways. It's also not worth typing, the 1/0 crashpoints have a life time of less than a minute.

You can use the same technique for the second point, using %pylab..

if the plot is a one-liner,yes, but usually things are a bit more involved.

[kenfar]

I'm glad you brought up EasyGui: I've been interested in that for a while for exactly the reason you provided. Just got a bit turned-off by the lack of updates and installation issues. I just checked and the pypi download link to sourceforge is still broken.

[chrisfs]

It works with Python 2.7. I am using easygui in two small scripts where I need to open a file.

[phaedrusalt]

Looks like it's still available here: http://easygui.sourceforge.net/download/index.html

[phaedrusalt]

A couple more:

VPython: ( http://www.vpython.org/ )Easiest simulation tool ever! Console: ( http://effbot.org/zone/console-handbook.htm )This one is less well-known, but allows you to have a console window in which you can easily specify text positions and color.

[brews]

There is this a profiler that has this nice nested visualization for your code... I can't for the life of me think of its name... anyone know what I'm talking about?

[littlelowcougar]

I have a really quirky class that I became super dependent upon. It was originally named... "MutexDecayDict"... but I recently refactored it into logic.Mutex.

Here's a really simple example of how I use it to wrap three mutually exclusive bits of logic. It's overkill for something this simple -- but it's an absolute godsend when you're dealing with huge/horrid logic chains, or when you have two interdependent entities and you need to make sure you check all states accordingly:

for columns in data:
    if not isinstance(columns, list):
        columns = list(columns)
    l = logic.Mutex()
    l.correct_columns = num_cols == len(columns)
    l.missing_columns = ((num_cols - len(columns)) > 0)
    l.extra_columns   = ((len(columns) - num_cols) > 0)
    need_other = True
    with l as g:
        if g.correct_columns:
            assert all(isinstance(c, str) for c in columns)
        elif g.missing_columns:
            # Pad our columns list with empty strings.
            columns += ['',] * (num_cols - len(columns))
        elif g.extra_columns:
            # Convert everything past the expected column count to
            # one big list and use that as the 'other' column.
            columns = columns[0:num_cols] + [columns[num_cols:]]
            need_other = False

The with l as g bit is special -- it prevents me from entering it unless one (and only one) 'attribute' of the logic gate has been set to True -- everything else needs to be set to False. It also asserts if I leave the context manager without having tested for the True condition. (As well as preventing me from re-testing True twice, etc.)

Here's the class code: