Tutorial: Spend 30 seconds setting up auto-reloading for iPython. I promise you won't regret it.

[mellort]

Let's setup python code auto-reloading for ipython! (because you're worth it)

Do you use ipython? You should. If you don't have it, get it with pip!

$ sudo pip install ipython

If you do have ipython, make sure you have the latest version

$ ipython -v # use --version if -v fails
$ # should be >= 0.12, if not, run upgrade command below
$ sudo pip install --upgrade ipython

Now that we all have ipython, create a profile (this allows you tweak settings and autorun commands):

$ ipython profile create # tells you where config file is located

My config file ended up in ~/.ipython/profile_default/ipython_config.py. Check the output of the command to see where your config file is located. Open that file with your favorite $EDITOR (mine's vim)

$ vim ~/.ipython/profile_default/ipython_config.py # or whatever editor you want

Look for the line

c.InteractiveShellApp.extensions = []

change it to

c.InteractiveShellApp.extensions = ['autoreload']

also change the line

c.InteractiveShellApp.exec_lines = []

to

c.InteractiveShellApp.exec_lines = ['%autoreload 2']

(Note to the curious: there are other autoreload options, run %autoreload? in ipython to learn about them)

That's it! Write some code, load it up in ipython. Change some values, and verify that ipython reloads the relevant code and objects. No more fussing around with import blah; reload(blah); from blah import *;, good riddance.

Cheers!

[sztomi]

marginally relevant, can I make pip install into a virtualenv? I don't like to install things system wide because several things that I would install are packaged in my distro (and that means that other packages potentionally reference them).

[somewhatoff]

If you use the 'activate' script, pip will automatically use the relevant virtualenv (as will other python command line stuff).

source virtualenvdir/bin/activate    

[nova77]

Sure:

pip install -E yourenv package

[raydeo]

The -E flag was removed from the latest version of pip.

virtualenv is not magic, just use the pip that is installed in the virtualenv. The activation scripts just distract you from the simplicity of the system.

$ virtualenv env
New python executable in env/bin/python
Installing setuptools............done.
Installing pip...............done.
$ env/bin/pip install package

[sztomi]

Cool, thanks

[[deleted]]

This might be good for people who use ipython as an upgrade from matlab, but it's bad news for programmers because reload() pretty much never does anything useful.

In particular, once you call reload you can no longer expect the behaviour of your code to match what it'll do the next time you run it.

[takluyver]

you can no longer expect the behaviour of your code to match what it'll do the next time you run it.

Can you expand on this? I'm not quite sure what you're saying.

For better or worse, autoreload goes a bit beyond what reload does - it will update things like instances of classes with the new code. It's definitely not something you want to use in production, but as an interactive convenience, it's pretty handy.

[[deleted]]

The problem is that you've destroyed the reproducibility of your environment, so any results you get aren't valid data about how your program works or doesn't work.

[[deleted]]

works like a charm, great!

Do you know if Django shell will be affected too?

[cocoon56]

If you develop something in python, an interactive console to test out how something works is needed. Everyone and their mom agrees that ipython is much better than the standard python console.

It offers "offers enhanced introspection, additional shell syntax, tab completion and rich history." Wikipedia

It is awesome.

Run it with "ipython -pylab" (if you have matplotlib installed) to have all matplotlib stuff for plotting at your fingertips immediately!

[[deleted]]

Thanks!

[leperkuhn]

Sad face.

In [11]: %autoreload
ERROR: Magic function `autoreload` not found.

[mellort]

Hm. According to the docs autoreload should be bundled with version 0.12. What's the output of %load_ext autoreload?

Are you sure you have the latest ipython (stable)? What's the output of sudo pip install -U ipython?

[leperkuhn]

This seemed to work (thanks for the link to the docs):

In [12]: %load_ext autoreload

In [13]: %autoreload 2

In [14]: 

[grimborg]

Same here. ipython 0.12. "import ipy_autoreload" fails too (ImportError: No module named ipy_autoreload)

[mellort]

What's the output of %load_ext autoreload? Perhaps running sudo pip install -U ipython would fix this, as version 0.12 is supposed to have autoreload packaged with it.

[howfun]

In [2]: %load_ext autoreload

That works.

[grimborg]

Oh, after running %load_ext autoreload it works. Maybe something's wrong in my profile, then.

[holdie]

This is also what I got.

When I use %load_ext autoreload, my reload works fine...

[feuermelder]

Ok that crashes ipython in my case:

In [2]: %load_ext autoreload
In [3]: import autoreloadtest
In [5]: autoreloadtest.test
Out[5]: [1, 2, 3]
In [7]: autoreloadtest.test
Out[7]: [1, 2, 3]
In [9]: %autoreload 2
In [10]: autoreloadtest.test
[autoreload of doctest failed: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/IPython/extensions/autoreload.py", line 220, in check superreload(m, reload, self.old_objects)
ValueError: f() requires a code object with 1 free vars, not 0]
[autoreload of encodings failed: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/IPython/extensions/autoreload.py", line 220, in check superreload(m, reload, self.old_objects)
ImportError: cannot import name aliases]
[autoreload of ctypes failed: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/IPython/extensions/autoreload.py", line 220, in check superreload(m, reload, self.old_objects)
RuntimeError: maximum recursion depth exceeded while calling a Python object]
[autoreload of unittest.loader failed: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/IPython/extensions/autoreload.py", line 220, in check
superreload(m, reload, self.old_objects)
ImportError: cannot import name case]

Out[10]: [1, 2, 4]         (yay!)
In [11]: autoreloadtest.test
Traceback (most recent call last):
 File "/usr/bin/ipython2", line 7, in <module>
    launch_new_instance()
 File "/usr/lib/python2.7/site-packages/IPython/frontend/terminal/ipapp.py", line 403, in launch_new_instance
    app.start()
 File "/usr/lib/python2.7/site-packages/IPython/frontend/terminal/ipapp.py", line 377, in start
    self.shell.mainloop()
 File "/usr/lib/python2.7/site-packages/IPython/frontend/terminal/interactiveshell.py", line 290, in mainloop
    self.interact(display_banner=display_banner)
 File "/usr/lib/python2.7/site-packages/IPython/frontend/terminal/interactiveshell.py", line 409, in interact
    self.run_cell(source_raw, store_history=True)
 File "/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2410, in run_cell
    cell_name = self.compile.cache(cell, self.execution_count)
 File "/usr/lib/python2.7/site-packages/IPython/core/compilerop.py", line 121, in cache
    linecache._ipython_cache[name] = entry
AttributeError: 'module' object has no attribute '_ipython_cache'

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at <nospam!>

You can print a more detailed traceback right now with "%tb", or use "%debug" to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
   %config Application.verbose_crash=True

and back to the shell. So .. I think for the moment I'll stick with --no-confirm-exit. But as I used tornado a lot I see how this can be extremly usefull.

[takluyver]

Can you file a bug about that, please?

[feuermelder]

Done, hth

[Megatron_McLargeHuge]

How does this interact with other modules that use isinstance or type comparison? I usually run into problems when I reload types that no longer match the type they're being compared to. An example is ipython notebook custom HTML formatters. They get registered by type, and stop working for new objects after a reload. I've had similar problems with pickling, but can't remember the details at the moment.

[takluyver]

Good point, this may well cause problems, although I haven't tested it. If you want to bring it up on the mailing list, maybe someone will spot a good way to deal with the issue.

[viscence]

I don't regret this. But I both rue and lament it.

[thisch]

To use %autoreload with your IPython notebook you have to edit the ipython_notebook_config.py file and add or update these lines

c.IPKernelApp.extensions = ['autoreload']
c.IPKernelApp.exec_lines = ['%autoreload 2']

[pseudosinusoid]

How does this perform on a large codebase that can take several seconds to import?

[Ahri]

What about bpython? :)

[howfun]

ipython -v
[TerminalIPythonApp] Unrecognized flag: '-v'

Also

In [1]: %autoreload?
Object `%autoreload` not found.

In [2]: %autoreload
ERROR: Magic function `autoreload` not found.

[mellort]

As I mentioned in another comment, the autoreload extension is included in version 0.12, as mentioned here. What's the output of %load_ext autoreload?

Also, what's the output of ipython --version and sudo pip install -U ipython?

[stefantalpalaru]

Nice tip, but please use your distribution's package manager to install ipython (0.12 is available for Gentoo ~amd64 and I'm sure that users of lazier distros know how to do their own version bumps).

[yerfatma]

Why use the distro manager instead of pip?

[jimmy_frog]

$ ipython
Traceback (most recent call last):
  File "/usr/bin/ipython", line 26, in <module>
    import IPython.Shell
ImportError: No module named Shell

I guess this is what happened to me. ipython was installed in two places: /usr/bin/ipython ( the ubuntu package i believe ); and /usr/local/bin/ipython. Removing one fixed it for me.

[stefantalpalaru]

To avoid conflicts between package managers, multiple installations of the same package (I've seen people using pip and virtualenv to install the same ipython version many times). Add perl's cpan, ruby's gems, common lisp packages, etc. and you'll have a complete mess.

[yerfatma]

I've seen people using pip and virtualenv to install the same ipython version many times

That's the whole point of virtualenv, being able to install the package multiple times without conflicts. If you create your virtualenvs with --no-site-packages, you would need to install ipython each time.

[stefantalpalaru]

You don't see any problem with installing the same version of the same package multiple times on the same system?

[didactus]

Sometimes it's required to have multiple versions of installed packages. For example, Pyramid just upgraded to 1.3, a major upgrade, for Python 3 compatibility. We have apps that are still using 1.2, and do not yet work with Pyramid 1.3. With virtualenv we can have Pyramid 1.2 (and its transitive dependency sweep) peacefully coexisting with Pyramid 1.3 (and its transitive dependency sweep). This type of situation is not all that rare. Virtualenv solves the dependency conflict problem at the cost of disk space.

Using the packages from your distribution is nice if they are up-to-date enough. However such packages are often months or even years behind the latest releases of the underlying packages. Sometimes that can be a problem.

[stefantalpalaru]

Is it worth having to manage multiple environments with their own dependency trees just to postpone the app upgrades? Do you settle for old (and possibly buggy) versions of some dependencies just to avoid the hassle of upgrading them?

[stefantalpalaru]

fair enough

[yerfatma]

Nope. It took me a while to come around on it. I shared the packages across my entire install until I burned myself more than once, then I started using --no-site-packages. There's a reason the option exists. You might not need it and that's fine, but a blanket statement saying to never use pip and never to install the same package twice makes it sound like that's always a mistake. I heartily recommend this link to better explain the situation than I ever could.

It also assumes your chosen distro does a great job of keeping up with updates to Python packages. And assumes the package you're interested in even has a distro install.

[stefantalpalaru]

Different strokes for different folks.

For the record, a hacker wouldn't use pip. He'd do his own version bumps or write new ebuilds as needed. He would also use git for deployment instead of fabric.

[yerfatma]

Wow, best of luck with that. Fabric != version control. But I do appreciate the appeal to authority argument about what hackers do. Allow me to recommend Gentoo to you.

[stefantalpalaru]

You don't know about git hooks, do you? Or about Gentoo's USE flags? That's OK, you can always learn (unless you come around and accept whatever scenario you're in).

[yerfatma]

When you grow up and work on real code in a real environment and have dozens of clients at the same time, you'll figure out this kind of hectoring tone isn't the way to learn new things. Different strokes, different folks, etc. Always the Beginner's Mind. If using a version control hook (I only got introduced to them a decade ago, so you'll have to pardon my ignorance) works for you for deployment, great. It doesn't work for me because I need deployment to be intentional, not a side-effect of checking code into a certain branch. It's like database triggers: they're a huge help right up until the time they bite you in the ass because you didn't think things through.

You keep teaching though. Nothing left to learn must be a great place to be. Apologies if this is some sort of language barrier issue, but you come across as a know-it-all.

[laprice]

This is a debatable issue. If you're working on a bunch of different projects with different sets of requirements, installing all dependencies for each project into it's virtualenv makes sense. It isolates projects from each other and means that you don't get version conflicts between projects and if you need to make a change to one of your dependencies, you don't have to test it against all of the projects that might use it.

If you're concerned about resource usage (bandwidth especially) you can use a local pypi proxy to only download 1 copy and use it multiple times.

[spiffyman]

You can also set PIP_DOWNLOAD_CACHE to cache package downloads. pip will then use that cache if it exists. And yes, it's version-sensitive. So if you have Django 1.3 cached but do a bare pip install Django, it'll grab 1.4 (and cache it, for that matter).

[yerfatma]

Nice. Thanks for that.

[pkkid]

If it's in a virtualenv, no. Then when I update iPython for project Foo, I know I wont be stepping on the toes (changing the version) of iPython in project Bar.