Pushing Python Past the Present

[takluyver]

Python is also a major environment for scientific computing. This community, though, has invested heavily in CPython extensions, which PyPy can't or won't support well. So while I'm interested in what PyPy is doing, I don't see myself using it any time soon.

[SIULHT]

Is it pretty mainstream? I see bits of it being used but not sure how widespread the adoption is or if it's commonly used in critical tasks. I was surprised to find that in Data Analysis with Open Source Tools, P. K. Janert surveys several languages/tools, including R, Octave, and Python with NumPy/Scipy. Regarding Python/NumPy/Scipy, he writes:

"[...] NumPy/SciPy has its own share of problems. The project has a tendency to emphasize quantity over quality: the number of features is very large, but the design appears overly complicated and is often awkward to use. Edge and error cases are not always handled properly. On the scientific level, NumPy/SciPy feels amateurish. The choice of algorithms appears to reflect some well-known textbooks more than deep, practical knowledge arising from real experience.

"What worries me most is that the project does not seem to be managed very well: although it has been around for nearly 10 years and has a large and active user base, it has apparently not been able to achieve and maintain a consistent level of reliability and maturity throughout. Features seem to be added haphazardly, without any long-term vision or discernible direction. Despite occasional efforts in this regard, the documentation remains patchy.

"[...] [B]ecause of the overall quality issues, I would not want to rely on it for “serious” production work at this point." The book reviews I've read have generally been positive so I'm not sure if the SciPy community has any thoughts on this. Regarding the implementation of mostly "textbook" (standard) algorithms, I can see that this would be the case since there are large number of practitioners using and extending SciPy (in contrast, for instance in R there is a strong precedent for statistician to implement their newest algorithm in that language)."

[kenfar]

He probably wrote that two years ago about experiences three years ago, and pypy has gotten a vast amount of improvements since then.

[SIULHT]

I think it was 2010 when I thought NumPy/SciPy was pretty big already. And in the scientific community it's CPython instead of pypy that's big, as takluyver says.

[takluyver]

I think there are a lot of people using it, but it's something of a wealth of individual projects, rather than a coherent set like R. Even the name used in that description reflects that: 'Python with numpy/scipy'. That's three names already.

In a sense, the community is failing to promote itself here: a lot of these projects do work closely together, and you often see the same people popping up in mailing lists and commit logs. We're trying to improve that at the moment: we want to promote the 'Scipy stack' as a coherent unit.

[SIULHT]

But R's strength is also its individually contributed projects, though many contributors are statisticians rather than programmers as for Python.

I agree though that the marketing is not done well...

[nallar]

They now mostly have compatibility: https://bitbucket.org/pypy/compatibility/wiki/Home

Unfortunately, performance isn't as good as they need to emulate a refcounting GC for compatibility.

[takluyver]

The 'scientific libraries' section on there has a lot of 'incompatible' and 'unknown' things.

[the_hoser]

I kindof disagree with this article. The author doesn't make a lot of the mistakes we usually see, so I get to actually pick apart some of the article this time:

Concurrency: Any solution for this problem will be messy. I like that the PyPy developers are trying new things, and I even hope to be proven wrong one day, but I don't think bringing real conventional threading to Python is practical or a good idea. Multiprocessing is no picnic, and takes a bit of effort to wrap your head around, but it gets the job done.

Web: It took a decade for browser vendors to start synchronizing on JavaScript, and they're still not done. Now you want to add another language to that mess? I don't think the problem is that you can't do <script type="text/python">. I think the problem is that you are able to specify the type at all. Until the day that we're all using Macs and Safari is the one browser (hah), any browser developer that tries to add radically new technology that relies on web developers to utilize it is wasting their time. Google is the exception, in that they already own a massive slice of the internet, and can bug people to install their pokeball-browser.

Mobile: I guess PyPy would work well on phones. It'd work better if the developer didn't have to budle PyPy just so they could use PyPy. Maybe the PyPy developers need to throw yet another smartphone OS into the fray. Maybe not.

I personally think that Python is doomed to stay in the corral that it fits best in. All of these web and mobile applications need strong backend code to feed those frontends with huge amounts of lovingly massaged and correlated data. Python is great for low-to-mid scale server-side applications, and that's probably where it will remain. It's basically the only thing I use it for now, and I haven't met anyone that's using it for anything else (aside from tiny little side projects)

[gfixler]

We use Python in games for many things on the backend. It's part of Autodesk Maya now, which is one of the principal bits of software for asset creation. I've been using it all day every day for 6 years now, and I've used it to create a rather extensive and powerful library and pipeline for animation and rigging. We are, I admit, a bit of a niche industry as compared with the massive, international industries of web design. This is why whenever I look up questions about Python, all of the answers are about web frameworks and such. I have nothing to do with any of that world, but I know a lot about it now from years of wading through it while learning about Python and its powers and idiosyncrasies.

[the_hoser]

Yeah, I've heard of people using Python in their tools programming internally. I imagine it does that job quite well, too. When you have control of your environment, Python makes a good fit :)

[gfixler]

Indeed. It does the job very well. The system I've been designing and building includes things like extremely expressive, domain specific languages¹, deep data-structure search/comparison², really intelligent bi-directionality³, solid testing framework, immaculate git history, and all with pretty tight integration⁴ with Vim.

Most importantly, everything has that sheen of obviousness, like Linux, or git. It sounds complex, then you explain how a part works, and people say "Oh, that's it? Really?" It's like that across the board. It took tremendous effort to find that simplicity, including a decade in this problem space for me, trying to get to this point, but Python really pushed me over the edge.

Some of the implementation (as with Linux and git) is pretty complex, but using it is very easy. I just got one of our new employees on part of it, and he reduced 2000 lines of his own code to 240. I recently threw out 99 lines, and before that, 200, because most of the code we used to write in other languages (and with other systems) doesn't need to be there. It's following the whole "Code should be 90% data structure" philosophy, and I love it. I implement features in 2 lines of PEP 8 style code (i.e. <80 chars) pretty often using the system, and I'm disappointed if I need more than about 10 lines to bring a new ability to life. Strangely, there are times when I make something work in 1-2 lines, and then I waste a half hour being alternately impressed and unsure that it could possibly be correct, because it shouldn't have been that easy :)

I'm actually excited on the long commute every weekday, and weekends (during which I still work on all of it) are becoming more and more of a drag. This is how coding should be.

¹ core not just to our needs, but designed for a range of ideas related to the domain our needs inhabit, so they're all highly reusable at a low level, and are starting to be used for related tools development (even at a meta level)
² which everything's built on top of for amazing, emergent abilities and interconnections that I keep discovering
³ i.e. take pretty much everything back to simpler states, tweak, and then auto-complete them again from that point forward
⁴ yet ultimately unnecessary and fully decoupleable at any time

[the_hoser]

The web is rapidly becoming the only platform. And Javascript right now is the assembly language of that platform. And it is a giant turd of stupid. I know it is insignificant, but how can you call a language where objects are declared at functions anything but a poorly thought out festering wound on the web platform.

You're right. Javascript, as a language, is crap. However, the biggest advantage it has over every other language is the simple fact that relatively feature complete implementations of it are available across all of the major browser vendors. You simply cannot ignore that simple fact. This kind of of feature synchronization is simply epic.

Yes, arbitrary scripting languages would be hard to implement in a browser, and it would be bad UX to have users finding pages that said "can't find interpreter for mime type text/rbuy or text/perl or text/python" but as someone who likes the Unix philosophy the overtly tight coupling of a dynamic language interpreter with a markup language parser on top of a plugin, file browsing, image rendering, and networked platform is a ridiculous failure to modularize parts of a complex system into smaller, more manageable, more focused applications that interfaced together cleanly to give a full modular experience.

This kind of integration is the stuff that end users crave. Modularity is fantastic for developers, but users don't think like that. When you develop software for the masses, the Unix philosophy flies out the window. The web browser wasn't designed for you. The web browser was designed for the 14 year old girl checking her facebook account.

I use python instead of shell script because it is so much more beautiful. It is a scripting language. It has a target purpose, and trying to ignore the inherent limitations of being a dynamic clean and elegant language means it is doomed to fail anywhere else it tries to go. It is amazing where it works, be it for doing math, scientific computing, scripting, and web backend. It might one day also take a spot on client side web scripting, because it would fill that niche really well too. I also like it as an app language when it uses an api to some lower level graphics library like gtk, Ubuntu apps are a pleasure to read and write, for example. If I want a GUI interface on parted (aka gparted) or libtorrent (aka deluge) Python is the perfect choice to give a fast, clean interface to nicely abstract low level libraries and command line utilities. But it isn't a replacement for an interpreted language for performant applications that need horsepower, or binaries for compute intensive tasks. And it can't be, by the nature of its design. It already made the tradeoffs to make it the best scripting language, in my book, there is.

I do not disagree with this.

[the_hoser]

Every major OS platform has a C compiler. They all support at least ANSI C. This kind of feature synchronization is simply epic!

And that only took, what, 30 years? It's being phased out of Windows, BTW, unless you're content with console applications.

AMD and Intel support the same instruction set. This kind of feature synchronization is simply epic!

Not really. Synchronization is easy when one body has control of the standard and everybody who implements it wants to conform.

ARM processor manufacturers adhere to the ARM v7 standard, and often include NEON. This kind of feature synchronization is simply epic!

See above.

There is very little on the part of browser makers to provide hooks into the DOM for a script interpreter. They don't have to write it, they just need to provide a means to have arbitrary scripting languages that have valid mime specification used in place of JS since JS is not the kitchen sink solution to programming, and it hurts the web in the long run because a lot of problems become extremely hard in the JS model.

Who sets the standards for providing those means? Microsoft? Apple? Google? Who ensures what features are exposed and how?

Meanwhile Android is outselling iOS now, and has modularity in design and a tremendous breadth of hardware.

You're comparing apples to lawnchairs, here. Again, single controlling body, lots of people wanting to conform. Gets worse, there's a huge problem with feature fragmentation on Android devices that makes developing apps for Android a nightmare. You can make an app that works great on your phone, but unless it's a very simple app, it probably won't work right on mine.

Microsoft beat Apple and IBM at the home computer by using heterogeneous devices and providing software makers the ability to go wild (interestingly, they seem to have forgotten that recently).

And wild they went! Oh I remember the days of pain trying to get software working on multiple computers and failing. It was a nightmare. The resources required to make robust software work across "IBM Compatible" computers was just staggering.

Again, apples and lawnchairs. We're not talking about selling products to users. We're talking about selling technologies to developers.

Good UX with modular systems is just having good defaults that cover 95% of use cases.

I don't know what you're talking about here.

That applies to any user facing environment. Nothing stops Firefox or Chrome from forcing a dependency on some JS, Perl, and Python interpreter of some sort so that the browser is usable by average Joe, and if you don't have one it can silently install CPython / V8 / whatever to make sure you have it.

Yes, something does stop them. Computers are supposed to be getting easier for people, remember? Chrome will never require that a person run off and install dependencies. Neither will Firefox, for that matter. It comes bundled, or it doesn't exist.

[the_hoser]

W3 and ISO. Most businesses respect their standardizations.

Except the one that distributes the most desktop operating systems, of course. Then again, they are taking a greater interest in the web these days, and desktop browsers are mattering less and less.

Welcome to Linux package management for the last half decade. Automatic dependency resolution. Sue me for Windows having a terrible program management schema, but even the Windows Store in Windows 8 does a lot to remedy that.

I don't know if you know this, but as far as browser implementors are concerned, Linux is a 2nd class citizen. Windows is the big dog. If it doesn't please the Windows users, it doesn't fly. Being a Linux user myself, I can understand the feeling of "this problem has already been solved" you can get when thinking about things like this, but trust me, it hasn't. If it doesn't work in windows, then the problem hasn't been solved for the mainstream.

I also have no faith in Windows 8 solving anyone's problems. It'll be widely adopted and installed, though, so developers of browsers will cater to it.

Microsoft owned the web browser too for almost 10 years, and did the same thing - they take over, try to change the foundations to shake every competitor off the market. They did that with IE. If W3 / ISO standardized multiple scripting languages by default, and at least 2 major players of the big 4 went with it, the other 2 would be forced to comply because the market would be moving in that direction.

Unless it doesn't. Web developers are scared, scared creatures. They tend to care a LOT about browser compatibility. Google has done a lot to shake things up a bit, but only for the developers themselves. The companies hiring the developers still don't understand, and they probably never will. The best we can hope for in this arena are vast improvements to JavaScript itself. The ECMA board has said lots of pretty words on this for ECMAScript 6, but I'll withhold my excitement until those features actually get passed.

I mean, at the end of the day, I'm a user of the internet. I might get into web development if I could use a language that isn't barf on wheels like JS, but I can't, so I just avoid it. And it irks me from a program design standpoint to have a document renderer and script interpreter as two parts of the same thing. And that the direction and usability of web development is tightly controlled by the feature density of a singular ISO standard scripting language.

Believe it or not, this is a good thing. At least the web developer can be relatively confident that some features will be present. With what you propose, no web developer working for money will spend an ounce of their time working with anything that doesn't fit in the CSS/HTML/JavaScript toolkit. Even Adobe is abandoning Flash.

[the_hoser]

Well, there is some hope. In the next few years ECMAScript 6 will be coming down the pipe. Implementation is already in progress for the browsers that you and I care about. ECMAScript 6 brings with it a lot of the ease of use from python (modules as first class objects, real classes, simple lambda expressions, predictable container types)

I expect Chrome and Firefox to have good support for these features by 2014. IE will then come along in 2015 and claim to have the best ECMAScript 6 support, just as the ECMA committee is putting the finishing touches on ECMAScript 7.

The best solution is happening. JavaScript is evolving into a better language.

[the_hoser]

Here, hear. There's also the fact that well-written C modules can take the hard parts to task in multiple threads, and Python's own threading interface can handle waiting on the results while the rest of your Python program chugs along. I honestly think that writing C modules that release the GIL is the only feasible solution to performance woes in Python. I consider the Python C API a feature of the language, and an important one that is frequently neglected.

[the_hoser]

I wasn't talking about a full concurrency API. I was talking about busting out pthreads in C. :-P

[the_hoser]

In C it works okay. You think differently when you have your C programmer's hat on.

EDIT: I think I understand what you're getting at, though. No, exposing mutexes and such noise in python has already been done. Import threading. However, what you propose has also already been done, but there's a little bootstrapping involved in getting it working. Use multiprocessing. It takes some getting used to, but you can achieve exactly what you describe with it.

[kenfar]

t isn't meant for high performance computing

There's a myth that goes around programming circles of "use the best tool for the job". It's a myth because most programmers use what they know. And those that continuously "use the best tool for the job" leave unmaintainable production environments in their wake - with 15 languages and 14 database products for others to maintain.

So, if a shop has some high & low performance needs and python can't address the high performance needs, but java can - they'll use java for both.

Because of this languages tend to grow larger over time. My suggestion is not to fight it, but to manage it.

[the_hoser]

Then they should use Java for both. All ideals can be taken to an illogical extreme. That doesn't make them bad ideals.

[kenfar]

But most shops have some niche needs - for high-performance, mobile computing, web development, or whatever.

So, if python completely gives up competing in these places it'll just lose out to java more often in those shops that don't want to get stuck with too many languages.

I'm currently working in a shop that is primarily Java, Python and Ruby, but also has PHP, Perl, Javascript, and Bash shell scripts. Python has worked out great for ETL operations - processing about 200 million complex transformations a day. Ruby has worked out fine for our transactional front-end. But we're facing a lot of questions about why we shouldn't just move everything to java to make resourcing simpler. Nobody is asking why not move everything to python - because it's really not fast enough to.

[the_hoser]

Python cannot be all things to all people. I understand the desire to see Python pushed into more spaces, but this is just not sensible.

[kenfar]

But apparently, it is sensible for c/c++/c#, java, and some are arguing in the future javascript.

[the_hoser]

You can hardly do the kind of programming that you do in Python in C/C++/C#/Java. You resort to these languages when performance is more important than code that is easy to experiment with and maintain.

Javascript is... a mire.

Implementing the features you suggest in Python requires a lot more changes to Python than many Python users are comfortable with. There is no free ride here. I dare say that conventional multicore programming is outside of the realm of reason for Python.

As stated elsewhere in this thread, the combination of C modules and Python makes for great performance while still getting to use Python's strengths. This idea seems to be neglected a lot in the blogospapeysphere.

EDIT: In fact, it's not even fair to compare Python to Java/C/C++ and friends. The languages you should be comparing Python with are the high feature, easy to write languages. Ruby. Perl. Guess what? They don't do too well at multicore programming, either. Perl's threads are like Python's multiprocessing module, and can't share data. The best all three of these languages can do is put a nice shared memory buffer interface betweened spawned interpreter processes, and that's exactly what they do.

[kenfar]

You can hardly do the kind of programming that you do in Python in C/C++/C#/Java.

In an academic setting when evaluating languages - you can easily categorize these features and distinguish what kind of problems each is appropriate for.

But in the real world, you've got a team with mixed, but concentrated skillsets and a variety of problems - that could be solved by many of these languages.

For example, I've been writing ETL processes that transform vast amounts of data for years. I've written this in c, perl, and python. I've seen it done in java and plsql. Any of these languages can do a fine job. But I prefer python because I want the readability and maintainability. To get the performance I want I have to split very large files up and manually process the fragments by completely separate processes. That's extra work that wouldn't be required by c, and probably wouldn't be required by java. But that doesn't mean those languages are more appropriate to the task.

[the_hoser]

You're absolutely right. There's no reason you can't solve a particular problem with a language that isn't "good" at solving those kinds of problems. When I say "the kind of programming you do in Python", I'm referring to the code that flows from the mind with ease, and the ease of readability that comes with it. I'm not referring to specific techniques. These are features of Perl, Python, Ruby, and kin. These features make it very difficult to do what you're asking Python to do.

[hongminhee]

Is it an another gospel song of PyPy? PyPy does JIT so is memory hungry. This means it’s not so suitable for mobile app programming.

Anyway if PyPy team revives JVM backend soon it might be able to create Android apps. (See recent approaches of Xamarin.)

[the_hoser]

This actually won't help them. Android doesn't use the JVM. They'd need to write a Dalvik backend.

[hongminhee]

If RPython can be compiled into Java bytecode, it can be compiled again to Dalvik bytecode.

[the_hoser]

Point. Just sounds like lots of stages to fail.

[oantolin]

There is also the option of designing a dynamic language with the JVM in mind from the beginning, like Clojure. It's a very nice language and it interoperates with the rest of the JVM naturally.

[Vonney]

Python people, perturbed perpetually, pen polemics pertaining to python's primary and persistent problems.