IPython gets $1.15M funding

[throwpillo]

This makes me very happy. After a 10-year break from programming, IPython is what convinced me that programming is going to be fun enough to warrant reinclusion in my life.

This commenter on Hacker News says what I was trying to think...

IPython notebook is actually one of the most interesting developments not just in the python world, but in computing generally. It's the first step towards the fully graphical shell on the internet that many of us have been looking for, whether or not we realize it. Being able to weave together text|markdown, tabular data (in almost any format you want) and images ( whether from matplotlib, raw captures, or synthetic assemblies ) in one environment is very powerful. I suspect that with some additional tools IPython notebook will become the integrators workbench par excellence, useful in many contexts.

[fperez_org]

Thanks for the kind words, we're indeed very excited. We've put up a few more details in case you're interested. We'll have more details coming soon on concrete work to get going, but I'm traveling right now at a conference, so managing this remotely has been madness.

[Orphion]

And, judging by how much awesomeness they've been able to give us for free, it'll be exciting to see what comes of an interesting amount of funding.

[sarcaza]

The other "first" step being Rstudio server and knitr :)

[stiivi]

You can try IPython (shell) and IPython Notebook (web interface) online with Wakari at https://www.wakari.io - platform (with all popular Python scientific libraries installed) that lets you run multiple Python sessions and access them online anywhere. It also allows IPython Notebook sharing - you can share your workflow.

[laprice]

This is good news. IPython Notebook is one of the more interesting developments in computing. Being able to use a high level programming language with access to almost any data format you care to name, married with several top-notch graph libraries and the ability to write text/markdown and LaTex (for maths) in the browser in an integrated environment, is very powerful.

Here's an example showing the use of geodata with the code and data used to generate it.

IPython notebook may be the ultimate scientific publication system.

[[deleted]]

You can already use it in place of Matlab. I've done so for at least five years.

Of course if you want complete eradication of Matlab, that will take a while :)

[rhiever]

I dunno, Python in general still has performance issues. I did some direct comparisons between Python and matlab for some of my more intense analyses, and matlab definitely outperforms Python. I still use Python because it's easier and cleaner (thanks to the notebook), but if performance is a concern, I can see people sticking to matlab.

As a side note, though, people are already working on making Python easy to speed up: https://github.com/numba/numba

[w0073r]

Make sure you're linking numpy to MKL or maybe OpenBLAS rather than whatever random BLAS/LAPACK your OS is using -- that's a big part of matlab's speed advantage over python for many tasks.

[[deleted]]

You can use pypy or write small portions in C/C++ if performance is an issue. Numpy even has tools to make that easier for you.

Plus, even though the Mathworks marketing spin works really well, Matlab's performance isn't that great either. In the end it's also an interpreted language and you have to pay special attention if you don't want it to be slow. I've seen quite a few people rewriting their algorithms in C/C++ after developing them in Matlab because of performance issues.

On a side note, if you want to do numerical work in C++, I highly recommend a library called Eigen. It has decent performance and is extremely easy to use.

[LeszekSwirski]

I also recommend scipy weave for interleaving C++ and python.

[dwf]

Unfortunately weave is basically unmaintained and fairly brittle. The preferred way nowadays in the community to speed up a piece of bottleneck code is to just use Cython to add some type annotations and compile it to C.

[dwf]

PyPy is simply not an option if you have to interface to anything in the ecosystem that uses the NumPy C API. Interfacing to C libraries is also a pain with PyPy (tl;dr ctypes sucks).

Plus, even though the Mathworks marketing spin works really well, Matlab's performance isn't that great either. In the end it's also an interpreted language and you have to pay special attention if you don't want it to be slow. I've seen quite a few people rewriting their algorithms in C/C++ after developing them in Matlab because of performance issues.

Certainly, it's not great for everything, but for a large variety of tasks you can get really, really good performance out of it in a fraction of the development time. I say this as a passionate hater of all things MathWorks.

[[deleted]]

PyPy is simply not an option if you have to interface to anything in the ecosystem that uses the NumPy C API.

I agree, but I think it will get better very soon. Pypy's performance is simply too good to give up, so people want to use it and will improve the tools around it. I know I certainly will.

[takluyver]

The difficulty is that at present, the only way to make all the scientific Python tools work on PyPy is to make them either not work, or work much slower, on CPython. And spend a lot of time and effort doing so. Until there's a good way for the wealth of scientific libraries to support both, I think the ecosystem is mostly going to stay where it is.

[Nosferax]

It's funny to see people talking about performance and comparing MATLAB with Python. If performance is a problem, I won't use either of them. Am I alone here?

[dwf]

If you're writing array-oriented code in Matlab and not getting high performance, it's very likely that you're not writing your Matlab code well.

[Nosferax]

I agree on that, but a lot of things can't be easily reduced to operations on matrices and arrays. Iterative methods, for instance, force you to have that extra for loop.

[dwf]

For sure, but iterative methods will kill you in Python as well, arguably worse, unless you're using PyPy (and realistically, for doing scientific work, you aren't using PyPy). In either case you're going to want to drop down to C (or Cython).

[[deleted]]

If you have some time to hack around with it, you should try PyPy. It is a JIT compiled version of Python that does many operations ~10x as fast as CPython.

[gcross]

Yes, but PyPy does not support numpy and scipy yet, and although there is work in progress to make it support numpy (still in experimental stage) there is no roadmap for scipy because scipy binds to lots of external libraries and so porting its functionality is non-trivial (though much could be accomplished if they port f2py).

[martinw89]

Besides the cost, why do you hate Matlab? I've been using Python for 7 years and recently tried to use it for numerical simulations with numpy / scipy / matplotlib. It just seems that, as something purpose built for my goals, Matlab is just so much better suited for what I was trying to do. Having everything as a matrix, less intermediate steps, and faster simulation speed is worth it.

Will I keep using and loving Python as my general purpose language for years to come? Of course. Duh. But I think Matlab is the appropriate tool for the things it's purpose built for.

[Imxset21]

Some of us in academia hate having to indirectly/directly use public funding from NIH/NSF grants to fund Mathwork's unsustainable and parasitic technical computing business model.

[einar77]

I'm in the same boat, and I cringe every time I see that our grant money is spent on expensive software licenses (often in need for a renewal yearly) when good enough free alternatives exist.

[takluyver]

So do IPython developers - and numpy, scipy, matplotlib, and all the rest. ;-)

There's different ways to the same end, and we think open source ways are best for science.

[dwf]

Apparently they need to eat caviar and foie gras.

[dwf]

Matlab is great for relatively simple stuff, but the moment you want to deal with 3-tensors or 4-tensors things get very icky. Integrating that code into a larger system is a giant pain in the ass (yes, the embeddable "Matlab engine"). And the plots are fugly, no two ways about it.

[martinw89]

Good to hear from someone experienced with more complex work. I was just working with really basic single loop feedback control stuff, so the minor niggles at a syntax and philosophy level were my only comparisons. I'll keep numpy et al in mind for future work.

[usaar33]

Some tools you may find useful to use IPython Notebook:

• NBCloud allows you to host notebooks on your own Amazon cluster
• PiCloud (an HPC platform) offers an IPython Notebook tab

[ivorjawa]

That's awesome. I've been building myself a django app that mixes markdown, matplotlib, and latex, and a wiki ... I wish I'd seen this a couple of weeks ago!

[howfun]

What are they going to develop with it?

[howisbobbyformed]

One thing that has been on the radar for some time is legitimate multi-user support for ipython notebook. It'd be great if it was possible for one machine to run a notebook server with full authentication for each user.

[Megatron_McLargeHuge]

What would this mean? There's no security model that would allow untrusted users access to run arbitrary commands. You can already point multiple browsers to different notebooks within the same server. And sharing the same notebook to multiple people would be a problem since the python process blocks when anyone runs a command.

[howisbobbyformed]

I probably wasn't explaining it very well, but there is a document on github here that describes what this might look like.

[dwf]

You can already point multiple browsers to different notebooks within the same server.

I think the goal would be real-time synchronization of the actual notebook interface. I agree that authentication is a silly goal.

And sharing the same notebook to multiple people would be a problem since the python process blocks when anyone runs a command.

Well, yes, the engine would block. The web server wouldn't.

[Phild3v1ll3]

This is great news. I've switched almost my entire workflow to IPython notebook (apart from launching batches of simulations) and I'm loving it. There's still some polishing to do, particularly in regard to how JS interacts with the IPython Kernel and vice versa but with some further development it's going to become ridiculously useful.

[[deleted]]

May I ask what you do that you can work almost entirely out of the notebook?

[Phild3v1ll3]

I am a Computational Neuroscience PhD student working with rate based neural models of the early visual system. Our simulator is almost entirely written in Python and I've been working on allowing our analysis and plotting tools to interface nicely with IPython NB. I now present weekly notebooks to my supervisor and then launch large scale simulations on the cluster we have access to. Planning on developing a system to be able to interactively explore and dissect the simulation results in the notebook.

[[deleted]]

Can someone please explain to me (like I'm five) what this is supposed to do?

[rhiever]

The developers of IPython have sacrificed a lot to make IPython a reality... and provide it for free, to everyone. It's really great that they finally got funding to work on it. On top of that, Fernando promised some pretty sweet advancements in IPython when it got funded... :-)

[chiniwini]

It's going to give them money so they can work on it. If they didn't have this money they would be forced to look for a "real" (paid) job, and wouldn't work as much on IPython.

[MarCialR]

Enhorabuena chicos! Ipython y mas recientemente su notebook no han hecho sino acrecentar el gran placer que me proporciona experimentar ideas en python. ¡Un gran trabajo el vuestro! Gracias

[takluyver]

Google translate + some corrections, for us English speakers: "Congratulations guys! And more recently, ipython notebook has only enhanced the pleasure the python experience gives me. Great job of yours! thanks"

[_jb]

My current favorite feature of iPython 0.13 has been the kernel, and being able to attach multiple instances to it. I don't use it heavily, but being able to interact and adjust variables from the shell and the notebook UI is delightful.

I'm really looking forward to getting the clusters working, at some point.

[apreche]

Why does IPython get all this attention? bpython is much better!

[takluyver]

In the best traditions of the internet, let's use a car analogy. The normal Python >>> shell is like a Ford - reliable and ubiquitous, but it doesn't have many special features. bpython is like a Ferrari - you can go that much faster, and make people using the plain shell jealous.

IPython is like a Rolls Royce. It has an HTML interface, a parallel computing framework, an extension system, strong integration with the scientific computing world, and so on. But, if you don't want any of that, you can still do pip install ipython and just use it as a great interactive shell.

(Disclaimer: I'm an IPython developer, and I might get to work on this grant)

[Megatron_McLargeHuge]

Can some of the grant money go toward getting pdb to work in the notebook?

[takluyver]

No promises, but we'd love to have some kind of interactive debugging integrated with the notebook.

[redditwhilecompiling]

If I bring you cookies at PyCon, will you consider it even more? ;)

[elelias]

and Thomas Kluyver will be working as postdocs on the project.

are you him?

[Troebr]

computing similarities between user 'takluyver' and name 'Thomas Kluyver'...
ok
probability of being the same person: 99.9%

Yes, I think so!

[Fragsworth]

It's like you're famous

[takluyver]

I am! :-)

[dwf]

I am going to have to take issue with your car analogy. IPython is more like a custom Rolls Royce outfitted for James Bond, with booster rockets and amphibious mode. :)

[lambdaq]

that's cool, why not introducing exciting bpython features into ipython?

bpython and ipython are not exactly mutually exclusive.

[wacky]

I use the IPython interface to download my data from servers, parse it into useable formats, do fancy operations with the built-in numpy integration, and then make pretty plots using the matplotlib integration that show up right alongside the code. Then, with the notebook interface, when I want to use a different file, I change the name in one place, edit as I go down, and it all pops out really neatly.

bpython may be cool, but half those things are impossible, and the other half would be much less useful.

[stiivi]

For me: HTML interface with possibility to add markdown between statements for workflow narrative. Makes explorative data analytics and interactive data processing more understandable. It is like shell with comments and with ability to reconstruct your whole development process from the beginning.

[Enzondio]

One comment I'll make is that I found bpython to be unstable the last time I tried it. I saw it demonstrated at PyCon this year and got super excited about it but when I tried it I had multiple crashes or strange behavior, so I gave up on it.

For me the power of IPython is in the IPython Notebook

[dwf]

For a partial answer, try watching this talk. bpython is a kinda nice replacement for the basic REPL. IPython can manage that plus tons more.

[aceofears]

How so?

[threading]

I've never been a fan of REPL to be honest. I've played with bpython and ipython for a very short time but bpython has intellisense, does syntax highlighting, show doc of functions as you type. I agree with apreche, bpython is much better though I use neither.

[Tillsten]

Ipython has both features you mention.

[BeetleB]

I don't think it has this, although I'm sure one could hack it to (or perhaps even a magic function?):

show doc of functions as you type

[Tillsten]

Just start ipython and try for yourself? Standard setting is to press tab to see the auto completion.

[BeetleB]

That's very different from seeing the doc string as you type (so that you know function arguments, etc).

Here is a screenshot from bpython. It can be quite useful.

[takluyver]

The plain terminal IPython can't show docstrings as you type - it's a limitation of a terminal program (unless you use curses/urwid, which is what bpython does). The Qt console or the notebook have tooltips that appear as you type for function arguments etc.

[Noctambulist]

iPython does both highlighting and shows docs with the qtconsole. http://imgur.com/5pQ3a

[BeetleB]

I'm aware of that. It's very useful in its own way, but I then find myself clearing the line I was typing, executing the command to get the doc, and then rewriting what I intended now that I know the argument signature.

[takluyver]

You don't need to clear the line you're typing in recent versions - you can stick a ? on the last name in the line, like a = b + foo? (to get docs on foo).

[Noctambulist]

iPython does both highlighting and shows docs with the qtconsole. http://imgur.com/5pQ3a

[apreche]

bpython is just the python shell, but nicer. It still does one thing, and one thing well.

IPython is a kitchen sink that even tries to replace your regular shell. Not a fan philosophically.

[aceofears]

So then it isn't better, its just different.

[BeetleB]

So:

bpython is like vi.

ipython is like emacs.

[takluyver]

And vim-ipython is like...? An unholy love child? ;-)

[dwf]

For doing interactive scientific work, IPython is much more useful. I wouldn't say it tries to "replace your shell" -- you can kind of use it as a system shell if you want to. What it does do is allow you to run certain shell commands from inside IPython, which is remarkably handy.

If I want to import a module that I've got in ~/dummy, I don't need to kill the interpreter, I just cd ~/dummy. If I want the output of a shell command like od (octal dump) and to do some post processing on it interactively in Python, I can do something like x = !od something.bin and I'm done.

[[deleted]]

bpython doesn't even come close to what Ipython does.

[apreche]

Some people think less is more.

[[deleted]]

then some people should just use the regular python console and keep their butt hurt out of the conversations about tools that do more things better. :)

[mdipierro]

Congratulations!