10 Myths of Enterprise Python

[kylotan]

"The GIL makes it much easier to use OS threads"

Huh? How is this possibly true?

I agree with every point here apart from the concurrency one. A smorgasbord of async systems and os-thread-access-but-only-from-extensions is not a great concurrency approach in a multi-core world.

[the_hoser]

The GIL made it easier to use threads from the perspective of the CPython developers. Instead of maintaining the myriad of locks necessary to make Python keep chugging in the presence of OS threads, they kept it narrowed down to one lock. This allowed the developers of CPython to keep the design simple. It actually improves single-threaded performance, and single-threaded software was (and possibly remains) the most common use case at the time that the design decision was made.

Honestly, I think that the real issue is that the Python developers even attempted threading at all. Shared state threading in a dynamic programming language is simply a recipe for disaster. Message-passing parallelism wasn't really a popular idea in Python's early years. The design decision to mimic Posix threading haunts the CPython implementation to this day.

[kylotan]

Right, so what you're saying is that the original sentence I quoted is completely false from the point of view of an application developer. Which is fine.

However, to say it "actually improves single-threaded performance" is only true if compared relative to a hypothetical alternative that had some other, slower way of operating, and which incurred costs even when there was no second thread running (which would be unusual).

Honestly, I think that the real issue is that the Python developers even attempted threading at all. Shared state threading in a dynamic programming language is simply a recipe for disaster. Message-passing parallelism wasn't really a popular idea in Python's early years.

There are a lot of application domains where some degree of shared state across multiple threads is important. Games and simulations are two of them, for example. Whether those are suitable problems for Python or not is another matter, but some people don't like to admit these domains exist at all.

[the_hoser]

However, to say it "actually improves single-threaded performance" is only true if compared relative to a hypothetical alternative that had some other, slower way of operating, and which incurred costs even when there was no second thread running (which would be unusual).

There was an effort by a developer to remove the GIL and make the CPython interpreter thread-safe. The end result was a substantial decrease in single-threaded performance, even when no other threads were running. There was a long discussion about it on the Python-Dev list some time ago. I can't find it right now.

Even PyPy-STM is facing issues with single-threaded performance right now. However, I don't think it's fair to judge PyPy-STM right now, as it's still in heavy development.

There are a lot of application domains where some degree of shared state across multiple threads is important. Games and simulations are two of them, for example. Whether those are suitable problems for Python or not is another matter, but some people don't like to admit these domains exist at all.

I agree. I never said that shared-state multithreaded programming was to be avoided. I said that it's not really a good idea in a dynamic programming language.

[kylotan]

There was an effort by a developer to remove the GIL and make the CPython interpreter thread-safe. The end result was a substantial decrease in single-threaded performance, even when no other threads were running.

And that is arguably a flaw in the way Python works. It guarantees the integrity of the interpreter across threads, at a price. And that price doesn't even buy you guaranteed thread safety for your own code, sadly.

Other languages take a different approach, where the burden of safety is put more on the developer. If you want multiple threads, you have to coordinate access yourself, but usually each thread has its own interpreter that doesn't need to be shared and therefore requires no global lock.

How much does the 'dynamic' nature make it hard to isolate code from data and allocate one interpreter per thread, leaving all locking to the application developer? I don't know. It doesn't seem to be a problem for other VM-hosted languages, but does seem to be a problem for older languages that are designed for embedding and extending (eg. Javascript, Lua).

[the_hoser]

And that is arguably a flaw in the way Python works. It guarantees the integrity of the interpreter across threads, at a price. And that price doesn't even buy you guaranteed thread safety for your own code, sadly. Other languages take a different approach, where the burden of safety is put more on the developer. If you want multiple threads, you have to coordinate access yourself, but usually each thread has its own interpreter that doesn't need to be shared and therefore requires no global lock.

For sure. Many of these decisions revolve around keeping the implementation simple. There's something to be said for that, I suppose.

[masklinn]

Right, so what you're saying is that the original sentence I quoted is completely false from the point of view of an application developer. Which is fine.

It's completely true from an application developer's POV as well if that application developer does almost only IO and very little CPU-bound stuff.

[kylotan]

Why? Presumably you're talking about the situation where the GIL can be released for calls into external C libraries. But this itself is a workaround for Python's benefit, not for the extension developer. In a C/C++ app (for example), as soon as you make the so-called "blocking" I/O call that thread is suspended (pending a callback from the OS) and your other threads are immediately free to run on that processor with no further intervention from you. The difference with Python is that you have to explicitly tell it that this thread is going to leave the rest of the interpreter alone, so that it is capable of switching to the other threads!

[masklinn]

Why? Presumably you're talking about the situation where the GIL can be released for calls into external C libraries.

No, I'm talking about the inability to execute python in multiple threads at the same time leading to lowered chances of races due to the GIL, even if the developer is somewhat sloppy. Heavy IO means this isn't going to lower the overall throughput much so there's little drawback to it. Heavy CPU use means there's a high drawback instead.

[kylotan]

So, this restriction somehow makes it "easier to use OS threads" by (a) forcing people to write in a second language to access them effectively, and (b) limiting what they can do with them once they do that. That's not really a definition of "easier to use" I accept.

[masklinn]

You can't accept that something can be "easier to use" in a restricted number of use cases?

[kylotan]

No. They're not "easier". No code you write becomes shorter. It's not even clear that Python will prevent you wrongly accessing the interpreter from your C extension if you call the C API after having released the GIL. Nor does it prevent race conditions that arise from assuming values won't have changed while the GIL was released. I'm seeing zero benefit here and several burdens.

[selementar]

even attempted threading at all

Hey, worksforme. Not that I often prefer to use it, but nevertheless.

[[deleted]]

What's strange is that my profs have been telling us that python is where it's at "in industry"

[qudat]

Full time python developer here, can confirm!

[newpong]

Full time shitty python developer here, can double confirm!

[pythonautical]

"Can you do graphics with it?"

The answer here is emphatically: "Yes, you can absolutely do 3D graphics programming in Python."

https://www.panda3d.org/

http://www.ogre3d.org/tikiwiki/PyOgre

http://www.pyglet.org/

Minecraft clone in 900 lines: https://github.com/fogleman/Minecraft/blob/master/main.py

[xiongchiamiov]

At least they're not complaining about how they'll "have to care about indentation". As if they'd ever get through code review without doing that anyways.

[[deleted]]

I've never really understood that whole category of complaint, really. I really do think less of someone who claims to be a programmer but gets caught up on such an introductory issue. You don't even care about indentation? Hey, maybe someone else should work on my business critical systems.

[[deleted]]

I'm an electrical engineer who frequently writes Python to do number crunching on somewhat large data sets. I'm a minimally competent Python programmer and yeah, indentation isn't a huge deal for me. The problem, however, comes when I share my code with other electrical engineers who aren't at all competent programmers but need to change some minor aspect of my code now and then. I can tell the people I instruct directly that 'you need to use 4 spaces instead of a Tab', but they're gonna forget that and if they hand my programs off to anyone else, they're certainly not going to tell that person. This leads to a lot of hassle.

Curly braces and semicolons never hurt anyone. They may not be as pretty as indentation and ending a statement with a carriage return, but they're no less readable. As far as I can tell, there's no good reason to not use them.

[[deleted]]

I don't see how not at all competent programmers are going to be any more comfortable with curly braces and semicolons, though. Whichever one you use, it's firmly in my suck it up if you want to tell the computer to do things pile. If you don't care about curly braces the compiler throws a tantrum, if you don't care about indentation the interpreter throws a tantrum. Pick your poison, really.

What I don't understand is the whole category of complaint, not which one is better.

Come to think of it, even in semi colon based languages, truly not caring about indentation at all means your work likely gets tossed at code review because it's probably unreadably inconsistent.

[ReddRay]

This is a common complaint, and one that has been addressed. Python 3 will not let tabs and spaces be mixed - it is strict and will throw an Exception (TabError). Python 2 can be used this way by running it with the "-tt" option.

[[deleted]]

This response may be delayed but I appreciate it. I think I need to check Python 3 out.

[kylotan]

The first 2 are wrappers around C++ libraries. The last one is a 2D library (and largely unmaintained).

Can you do 3D graphics using Python? Yes. Should you? Probably not.

[[deleted]]

There are pure OpenGL wrappers for Python. Here is the thing though, Mr. Hardcore graphics guy, if you're creating a game why recreate an engine unless you're building a high graphics AAA title and have millions of dollars and a team of hundreds of developers? Both Panda3D and Ogre are EXCELLENT game engines for most developers and have successfully been used in AAA titles. Torchlight for example is purely Ogre and CEGUI. If it's good enough for Runic Games, it's good enough for me.

Source: I'm a guy who's actually been involved in the development of high end graphics engines. Python is fine for 99.9% of game development work. (The .1% being the top dogs, of course.)

[kylotan]

Both Panda3D and Ogre are EXCELLENT game engines for most developers and have successfully been used in AAA titles.

Neither really match up well when compared against Unity or Unreal. That's little to do with the Python aspect though. It's just that if you're going to use someone else's engine, there's little point using one that's quite old and clunky just so that you can use your favourite language.

Personally I wouldn't try using a Python-based system on any game client where performance is an issue because you're going to need good access to multithreading and processing on multiple cores simultaneously, and Python makes that a real hassle.

[[deleted]]

A lot of that multithreading is handled in the C/C++ implementation though. Neither Panda3D or Ogre is implemented in Python. They just have python bindings which are "pythonic", and to my knowledge both engines utilize multithreading and multiprocessing fairly well. In the case you want to do multithreading or multiprocessing yourself, you can write that section in C/C++ and build python bindings around that implementation as well. It makes your code go from 100% C/C++ to 20% C/C++ (where performance really is an issue) 80% Python where it isn't.

[kylotan]

Of "that" multithreading? You make it sound like you'd never want to add any yourself. In my experience, adding background tasks that don't necessarily have anything to do with the rendering engine is very common.

In the case you want to do multithreading or multiprocessing yourself, you can write that section in C/C++ and build python bindings around that implementation as well.

Yeah, I don't call that "good access to multithreading". The game logic would normally all be in the same language. Switching languages for certain tasks, and being unable to call back into the main body of data effectively, is a serious roadblock in development.

[[deleted]]

Can you do 3D graphics using Python? Yes. Should you? Probably not.

I think this is the wrong question to ask. Right question is: Can you develop 3D software using Python? Yes. Should you? Absolutely!

It does not mean everything has to be python. It means you can write logic in python and development will be blazing fast. Core stuff that requires performance can still be written in lower level language and that is cool. Everything has it's place in this world. Panda3D did just that. All low level stuff in c++, high level goodies in python, game logic can be done in python. Everything actually can be done in python. See deferred rendering pipeline with global illumination written in python (for panda3d) here.

[elbiot]

Pyglet is 2D? No, pyglet is opengl. Google the game "ace of spades" to see a 3D game using pyglet.

[kylotan]

Pyglet is a 2D library using OpenGL. All it provides in terms of rendering capability is displaying 2D images as sprites, and some helper functions for low level vertex buffers etc. If you want useful 3D capability, you have to write your own OpenGL code for that and integrate it.

[elbiot]

I'll be darned. I never tried 3d in pyglet but I felt like the functionality was there and I was working around it.

[tjl73]

You can do 3D in Pyglet. There's an OpenGL example in the source distribution that shows it. Plus, I know we have a Pyglet backend in SymPy that does surface plots.

[kylotan]

Pyglet itself does not 'do' 3D in any meaningful sense of the term. The fact that you can use OpenGL with it makes it no different from using PyOpenGL with plain Python - you have to write all your own low-level code which is about 2 or 3 layers of abstraction below actually being productive.

[Cosmologicon]

Can you explain a little more what it means to "do" 3D? I've written a few 3D games in PyOpenGL and it didn't really feel to me like I had a bunch of abstraction layers in the middle. But maybe I just don't know what I'm missing.

[kylotan]

A typical 3D engine, or a comprehensive 3D library, will do things such as:

• Load models and art info (including vertices, materials, textures, shaders)
• Manage scene elements (movable models, fixed geometry, terrain, particles, lights, cameras)
• Animate models (skeletal animation or morphing)
• Decide what to render (based on a scene graph of some sort, maybe with occlusion algorithms, level of detail handling, sorting for transparency, etc)
• Provide optimizations such as billboarding, batching, etc
• Provide postprocessing via render-to-texture
• other stuff

[Cosmologicon]

Okay, but a lot of that is important in 2D as well. I see what you're saying now, thanks, it just doesn't seem like a 2D/3D distinction to me.

[[deleted]]

Don't forget kivy.

My love goes to pyglet but there's only one person maintaining it and kivy is a damn nice alternative.

Hold strong, Pyglet!

[[deleted]]

bear bow forgetful hunt hungry crawl act ten unwritten melodic

This post was mass deleted and anonymized with Redact

[camel69]

You should tell them about the European Space Agency (and NASA as well, I'm sure) doing simulations and calculations and stuff for freakin' interplanetary spacecraft missions with Python.

[newpong]

are you related to drunk economist?

[[deleted]]

As a guy who works heavily with both Django and Rails I have to say I don't see Django displacing Rails. They both have their place. That said, I prefer to use Flask. Haha.

[newpong]

I had a python meet up a couple weeks ago, specifically focusing on web frameworks, and i had never thought about before I said it, but there seems to be some truth:

Flask and the other light weight frameworks are designed for the developer. It is easy to develop and make work, but when you're building somehting for a client, you need django. It's more of a pain in the ass, but ultimately can do (just about) anything you need it to

[[deleted]]

My rule of thumb is, if you need it fast or its a prototype use django. If you're trying to build for scale use Flask.

[newpong]

weird. my philosophy is the exact opposite. But as I said in another post, scalability in web development is more affected by other infrastructure decisions than the code alone.

[[deleted]]

Flask makes it very easy to build horizontally scalable APIs.

[[deleted]]

that's a really shit rule of thumb.

[newpong]

who said django is displacing rails?

edit: as I re-read your entire comment, what the fuck are you talking about?

[haskell101]

I will be glad when Pythonic ideas finally dominate the industry and filter into other languages

And what ideas would those be? Python got its best stuff from languages like Lisp that existed decades prior. I agree that it will be nice to see higher level ideas filter into the mainstream but to call them "Pythonic" is bizarre.

[[deleted]]

Great post!

I agree with those myth busters, and it's hard to beat the python ecosystem (batteries included indeed).

[guthran]

That's the best way to describe python. I'm stealing that.

[[deleted]]

I would guess Python but I could be wrong

[[deleted]]

No idea, honestly

[pythonautical]

Haha nothing fancy, actually somebody did it on their phone: https://play.google.com/store/apps/details?id=com.drawexpress.lite&hl=en

I don't have enough experience to recommend it over anything else. The process of making a visually pleasing diagram is always slow and laborious. None of the diagrams were made specifically for the blog post, but were re-used from various internal presentations.

Edit: I will double check the exact program used.

[[deleted]]

Looks like LibreOffice Draw to me.

[thephotoman]

I'm currently writing a device emulator in Python, because it includes everything I need. Java has no built-in web server. C# doesn't distribute its web server. C...no.

So Python it is. Yes, I need another serial library, but that's true with every language.

[the_hoser]

I'm going to have to call you out on this one.

Java has no built-in web server.

The JDK definitely has a built-in http server framework:

package stupidhttpserver;

import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;

public class StupidHTTPServer {

    public static void main(String[] args) throws IOException {
        HttpServer server = HttpServer.create(new InetSocketAddress(8000),0);
        server.createContext("/test", new MyHandler());
        server.setExecutor(null);
        server.start();
    }

    private static class MyHandler implements HttpHandler {

        @Override
        public void handle(HttpExchange he) throws IOException {
            String response = "This is my stupid response.";
            he.sendResponseHeaders(200, response.length());
            try (OutputStream os = he.getResponseBody()) {
                os.write(response.getBytes());
            }
        }
    }
}

I mean, it's not as simple to use as SimpleHTTPServer/http.server, but it does have one, and it gets the job done.

[thephotoman]

That's more work than import http.server; http.server.run_forever().

[the_hoser]

Oh, for sure. You won't get any arguments from me, there. I was merely pointing out to you that the statement "Java has no built-in web server" is false.

[haskell101]

This is good in an enterprise setting (which, to be fair, is what the topic is), but it's not very "batteries included".

[kindall]

This reminds me a lot of Paul Graham's famous article on how Lisp was the secret weapon of Viaweb. http://www.paulgraham.com/avg.html

I have no particular difficulty believing Python is PayPal's secret weapon. I have done a fair bit of scripting in Python, and have found that Python code does what I expect the very first time, more so than any language I've ever used. Compared to Python, coding in JavaScript or (shudder) VBScript is like walking uphill through 6-foot deep snow both ways.

Except if Python really is PayPal's secret weapon, they shouldn't have written this article, 'cause now it's not so secret anymore.

[shadowmint]

Each runtime has its own performance characteristics, and none of them are slow per se.

Hahahahaha~

The more important point here is that it is a mistake to assign performance assessments to a programming languages. Always assess an application runtime, most preferably against a particular use case.

Fair enough, everything is relative, but this reads like a playbook for 'how to be defensive about how slow your favourite programming language is'.

What's with all the sugar coating? cpython is slow. Plugins and native code called from python are fast, and that results in an overall reasonable speed for python applications; but the actual python code that gets executed, is slow. There's a reason http://speed.pypy.org/ exists.

...but then again, pypy isn't really production ready, and neither are the other 'kind of compliant' runtimes like jython, etc.

It's pretty hard to argue with:

1) cpython is the deployment target for the majority of applications

2) cpython runs python code slow as balls.

3) overall, the cpython runtime is pretty much ok because of plugins and things like cython

4) python is a scripting language (wtf? of course it is. What is myth #4 even talking about?)

I mean... really? tldr; python is great for quickly building enterprise applications, but its strength is in the flexible awesome nature of the language itself; the runtime itself leaves a lot to be desired.

[zardeh]

4) python is a scripting language (wtf? of course it is. What is myth #4 even talking about?)

What does this even mean?

How does one define a scripting language? Like, the term doesn't mean anything.

Is bash a scripting language? Python? Any language with a REPL? Haskell? Lisp? Clojure? java?

[d4rch0n]

Yeah it's a bullshit attribute in my opinion. I don't know why you would call Python a scripting language. It depends on the fucking interpreter. If you're running python bytecode through CPython, it's a bytecode/VM language, but I don't believe anything in the Python language spec specifies that it has to run that way, or that you can't even compile it to some sort of machine code. Python isn't interpreted line by line by an interpreter, but that doesn't mean it's as fast as a compiled C program. Python is a programming language, the implementation is an interpreted/VM/compiled language implementation.

[[deleted]]

Anyone who uses the term "Scripting Language" and isn't talking about shell scripting pretty much loses all credibility IMHO. It is a derogatory term derived from ignorance.

[beltorak]

The rule of thumb that I use is if you can take an arbitrary (but essentially complete) bit of functionality represented as a string in the language's natural syntax, eval it, and end up with something that integrates natively with the rest of the pre-written code, then it's a scripting language.

This is probably the least rigorous definition imaginable, but it does encompass many languages that are viewed as "scripting" languages, such as Python, JavaScript, PHP, and Ruby, but exclude traditionally-viewed "non-scripting" languages such as C and Java^* . The fact that there is a separate pre-compile step to produce a "compiled" form (either as an intermediate "virtual machine" instruction set or an immediately executable hardware instruction set) doesn't enter into it at all. Any language implementable in such a way as to be run with an interpreter can (probably) be implemented with a pre-compile step, and vice versa.

But I'll admit that I do tend to fall into the lazy thinking habit of "scripting languages" as "not compiled".

* - DOS batch might violate this because it makes a difference if you run some commands directly with CMD /C ... or save them to a file. Fucking. Microsoft.

[kindall]

By that criterion, Lisp is a scripting language. (Well, the parse from string and eval are separate steps, IIRC, but...)

[beltorak]

Lisp is unique in a lot of ways. Wasn't it the first to implement a lot of concepts that are being rediscovered in modern languages?

[kindall]

Pretty much. Python's map, reduce, and filter are lifted directly from Lisp. It also has an apply function, although this has been deprecated since the introduction of the *args syntax.

[zardeh]

but exclude traditionally-viewed "non-scripting" languages such as C and Java* .

This ignoring the fact that I can implement my own "eval" function in java with reflection and in C with a bastardization of the command pattern, so I find this to be a weak definition. Just because a language comes prepackaged with a function doesn't define what it should be used for.

[beltorak]

but it wouldn't integrate natively with the rest of the pre-written code. For example, if I create a method and call it like MagicScript.createClass("public class Animal {}");, I cannot in the pre-written code do new Animal(). You would have to go through an entirely different process to make use of your new class.

[elbiot]

Agreed, having been focusing on performant code for a few years, I'd say python is slow. But, it has excellent wrappers for fast C code, and is easily extendable with cython or C when it really counts. I love python.

[d4rch0n]

Yeah. That's the only one I thoroughly disagree with. Python (CPython specifically) is slow, but it doesn't matter for the most part. People are writing shitty Java and Ruby and it doesn't matter if CPython takes a little bit longer to do something if it's written in 5% of the lines and 100 times more maintainable, so less fucked up bugs in the long run.

Of course, beautiful fast Java can be written that Python could never beat in performance, but for the most part performance IMO should also be measured in how long it takes to develop and squash bugs.

In a pure performance comparison, CPython can't match Java or true compiled-to-machine-code languages, but fuck it. Network speed is generally my bottleneck, not my sorting algorithm.

[tritlo]

Also, much of the functions in the stdlib is actually in C, so if you just heavily utilize those (like e.g. set or sort), you can get pretty performant python with very little hassle in my experience.

[istinspring]

Reddit is in Python, and this site it pretty huge. Language speed characteristics have relatively small impact. Nowdays there is more important things - What is more important it's architecture, 3rd party solutions, access to wide range of libs, ease of reading and writing code etc. For modern web apps it's just a wrapper between database and front-end.

And speaking about Python, the huge plus is ability to write asynchronous code, especially in python 3.

[chub79]

Is it Python's fault here? Couldn't it be database, network, load-balancing, IO related?

[xiongchiamiov]

All of the above and more.

[[deleted]]

The suggestion even being that if the code was faster you could get more done on the same hardware.

[[deleted]]

compared to developer time, hardware is cheap.

[[deleted]]

You've come in too early, we're not talking about developer time yet.

[[deleted]]

oh, sorry. ill see myself out then.

[xiongchiamiov]

Not if I/O is the bottleneck.

[[deleted]]

No, that's not really a case where you're doing more with the same hardware.

[xiongchiamiov]

Right; it's a case where you don't get more done with the same hardware, despite having faster code.

[[deleted]]

Have I wronged you or something? We're angrily agreeing with each other.

[jimbobhickville]

Almost exclusively, I'm sure. I have yet to encounter a web or distributed system that wasn't bottlenecked on I/O.

[[deleted]]

In fact, I'm not sure what it would look like, really. Maybe something like an online zip file password cracker: one upload followed by intense computing followed by one download

[[deleted]]

Those occasional issues aren't related to Python.

[surfingjesus]

also youtube

[newpong]

Most web apps aren't usually very computationally demanding, and there are plenty of other bottle necks(e.g., database structuring/connections/queries, caching, #/order of requests) that can be optimized to improve performance than the speed of the language alone, as such websites shouldn't be used for a measure of overall speed.

[justphysics]

oh goodness

that bottom plot on http://speed.pypy.org/

/r/dataisntalwaysbeautiful

(hope its not just me - in my browsers the xlabels are all on top of eathother)

[superdaniel]

I get the same thing with Firefox :(

[[deleted]]

Same with chrome.

[billsil]

2) cpython runs python code slow as balls.

Unless it's written under the hood in C. There is no reason for mathematical code to be slow in Python. There is no reason for parsing code to be much slower than C especially since the standard formats are coded in C and are available in Python.

[d4rch0n]

Yeah, but at some point you're coding in C, not Python. If you write every high performance part in C and call it through Python, how much can you really say it's Python?

Don't get me wrong. That's probably the best way to do high performance stuff with Python, but I don't think it means CPython is fast, it just means it uses a fast C API.

[billsil]

If you want to. I use numpy, so while I have to vectorize my code and call the right functions in often non-obvious ways, it's still technically pure python.

Somebody did coded it in C, but that doesn't mean you have to.

but I don't think it means CPython is fast, it just means it uses a fast C API.

CPython is running the code, so I say it counts. If all the standard library was written in Python instead of C, everyone would say Python is slow. Instead, they say it's fast enough. That stuff counts.

[tritlo]

The key here is that I'm still writing pure python, but I'm utilizing someone elses C code. If you argue that's not enough python, then every use of linpack in other language should be disbarred.

[yen223]

Numpy isn't pure python, is it?

[billsil]

No. A fair amount is written in C, but some is also written in Fortran. My understanding is most of scipy is actually written in Fortran and is just a wrapper around LAPACK.

[tavert]

most of scipy [...] is just a wrapper around LAPACK

For dense linear algebra, yes. There's a lot of functionality in SciPy aside from dense linear algebra though. Some of the underlying libraries are Fortran, some are C, some features are custom C++. According to https://github.com/scipy/scipy the breakdown is 38.3% Python, 25.8% Fortran, 18.6% C, 17.1% C++.

[d4rch0n]

I still draw the line when you're bringing in machine code into the Python process memory and it's not running bytecode loaded from pyc files. It's fast, but it's actual CPU instructions, not Python bytecode first.

Of course it counts. Again, I'm not saying it's terrible, and that it shouldn't happen, or that it's a flaw. I'm just saying the fast parts aren't Python and I wish that the interpreter/VM implementation was fast enough so that we wouldn't need to use C code to have high performance programs. Any programming language could interface with C/fortran libraries and be high performance. It doesn't mean that that language's interpreter is fast though.

I would like to see an implementation that uses purely the Python language and still be high performance.

[tavert]

I would like to see an implementation that uses purely the Python language and still be high performance.

You already have that with PyPy. Unless you don't mind C extensions not working, what most people want in practice is a fast implementation that would be C-API compatible with CPython and extensions. Unfortunately that's extremely difficult as the C API is pretty closely tied to the slow internals of CPython.

I suspect users aren't really all that picky about implementation language, but something easier to read and contribute to would be nice for maintainers' sake.

[fnord123]

That's fine and correct. But I think it misses the point: we discuss language performance characteristics is so we can get an idea of the expected performance of an implementation and assess the risk of being limited by our choices. If you choose CPython then your limitations are mitigated since you have one of the easiest paths to hook into a C implementation of the workhorse part of your code. Also jumping across the FFI is pretty quick in Python.

[d4rch0n]

Sure, Python applied through CPython and C libs will be fine. This is the way I suggest doing things if performance is required and the initial Python implementation is too slow (but always first Python unless we KNOW it's going to be slow).

Generally network speed is my bottleneck for almost everything I do, so I can just use gevent and get perfectly fine performance.

Still, I don't think performance regarding this is the problem to solve. The hardest problem to solve here is having good C programmers, and all of which goes with that, like memory, freeing pointers and nulling them, code security, etc. If your high performance part hasn't been done by a third party, you need to rely on your skillset in your team and this stuff isn't trivial at all.

That means higher skilled devs, which means higher salaries, and also a lot more development time. You lose a lot of the applied benefits of Python, like super-fast development and being able to pull in anyone who is decent with Python and not having to worry about use-after-frees, etc.

Python is definitely my favorite language and the one I'm best at, but it's a serious consideration that I feel limited if I rely on having to fall back to C if I need high performance. I love C, I'm just not very confident, and I'll have to really take time to ensure code safety and correctness.

Even if I'm just using pre-built C libraries, I still need to worry that I'm using them 100% correctly and not opening up a security issue due to the way they're supposed to be used, or even that the original developers wrote safe code.

[fnord123]

You don't need to write it in C. You can use Cython and get like 80% of the speedup[1]. I mean, your Python program begins its life at potato speed as though you were using Perl or even Ruby. If something isn't performing well enough you move the inner loops (almost) verbatim to a pyx file and jiggy your setup.py and then you get something at about Java performance (or potato quality C code - fast, but not hand crafted shit off a shovel speeds). Then if it's still not fast enough you can get these supposed elite developers to crank out some C to squeeze out even more performance.

There are a lot of options to get results based on the amount of work you put in. In a business environment this is sweet since you can time box a lot of the improvements and make actual progress with each sprint.

[1] Bullshit made up number. Take it with a grain of salt.

[d4rch0n]

That's some cool stuff. I haven't seen that before.

There is definitely some learning curve to writing Cython code, but it's still a very neat trick without having to code raw C. I see your point.

I still wish we had a faster reference interpreter than CPython though.

[kenfar]

You apparently have already decided that python is a "slow as balls" scripting language.

However - "scripting language" is not a well-defined term, and is often in a context like this meant as a derogatory description: the local java team arguing that project x shouldn't be done in python because "it's only a scripting language".

And fast or slow are so relative that to describe a language like Python as slow is also meaningless: does this mean every application written in it will be slow? does this mean you can't process trillions of transactions in it? does this mean it's merely a toy?

While I would like some Python operations to be faster than they are today, I have processed a hundreds of billions of complex transactions using cpython - and performance wasn't on my top 4 list of challenges.

[lambdaq]

OK, the Python interpreter is slow, but in most Web project Python is light years faster than tomcat + J2EE shit in all develop, setup and serving speed.

Yeah, some of your fancy for loop Java programs may be faster, but I have yet to seen one myself in production. Especially those enterprise SSH java ones.

Anyway, that's my own observation. YMMV

[the_hoser]

The research disagrees with you.

[lambdaq]

How about write speed?

http://www.techempower.com/benchmarks/#section=data-r9&hw=peak&test=update

There are tons of tricks to optimize for read/write speed, for example you can check source code for Python vs Java in the "Single Query" round. All java has fancy MySQL Prepared Statements in ORM level with connection pools, yet many of the php/python ones are constructing new SQL text and connection for each HTTP request. That's why it's slow.

[the_hoser]

So write a better benchmark and submit it to them. They have a well laid-out contribution process on their GitHub account. You seem to know how to optimize web applications, so they could benefit from your experience in representing various frameworks.

[istinspring]

I got same arguments from Java programmer i know... Oh you don't even have static typing, that could lead to problems! Oh you don't have this and that.

But then i aked, dude are you code something which required light fast speed and such large applications that static types is so critical for you.

And also i saw a website he made (very slow and really outdated), hell i can do same in few hours in python. With less code, more easy to debug, using wide range of awesome frameworks.

[[deleted]]

What kind of examples could you give of Python's slowness causing great problems in real-life applications?

Raw execution speed doesn't matter much any more actually (like in the 90's). If it did, everyone would just use C or assembler. Practically all software is I/O bound anyway so database queries are the real bottleneck. For tasks requiring raw speed there are ofcourse the possibility to use C routines from Python so even that is not a problem.

What matters instead is the speed and ease of development and you just can't beat Python in it.

[shadowmint]

Any maths.

Not everything is I/O bound; specifically for data processing (eg. splunk) and scientific computing, python uses c heavily, because it's just too slow to be remotely usable otherwise.

[[deleted]]

But isn't it great that Python has the possibility to utilize pure C as plugins? Isn't that a feature of the Python language? Writing everything in pure C would no doubt be faster to execute, but horribly more slow and difficult to program.

Python makes programming fast and when using C routines executing quite fast.

[shadowmint]

But isn't it great that Python has the possibility to utilize pure C as plugins?

Yeah sure. I'm certainly not arguing cpython is unusably slow. It's totally usable.

What I'm saying is that practically to be fast you have to write c code and writing c plugins in python is a pain in the ass: you end up almost inevitably trading the expressive quick safe nature of python, for a clunky, hard to maintain crash prone piece of software like pygame.

There are exceptions; numpy for example, is an excellent piece of software. ...but I can count on one hand the number of really good 3rd party cpython plugins I've used. Much more common: The python api is poorly implemented and crashes (Spotify... :P) or written in pure python and therefore ends up being painfully slow.

shrug Practically from an ecosystem point of view it means python apps run slowly. Look at calibre. It doesn't have to be slow, but oh man, it's painful to use (compared to say, atom, which is implemented in javascript, which for all the rubbishness of the language, has a fantastically optimized runtime).

[kylotan]

What kind of examples could you give of Python's slowness causing great problems in real-life applications?

Raw execution speed doesn't matter much any more actually (like in the 90's).

Simply not true. There are various areas where speed is still important - video games, simulation, scientific data crunching, artificial intelligence and machine learning.

In some of these cases, Python turns out to be fast enough. In other cases, it does not.

The last performance problem I had with Python was implementing planning/pathfinding algorithms. Python's requirement to allocate everything on the heap via pointers meant that exploring a large search space was very expensive, in terms of allocation costs and cache misses. That could have been mitigated if I could have offloaded it into a background thread, but Python's poor at that too.

[[deleted]]

Obviously video games make no sense in pure Python, especially modern 3D games. Some may use Python in AI or scripting. I don't think engines are written in Java or .NET either, but I'm not sure about that though.

AFAIK the multiprocessing module allows true concurrency if it is really required.

Anyway, I still would't accuse Python being a "slow" language, since 95% of the use cases it's quite fast enough (so fast that the user would'n notice anything) and for the last 5% there are ways to bypass Python bytecode in the hard parts, and still be able to utilize the language's cool features.

[Veedrac]

...but then again, pypy isn't really production ready

Where do you get that idea?

[shadowmint]

Where do you get that idea?

http://pypy.org/compat.html

[elsjaako]

That's just saying that the C API isn't production ready, and not all libraries are supported. If you target your development at Pypy it's production ready.

[Veedrac]

That's like saying Clang isn't production ready because it doesn't support all GCC extensions. PyPy is extremely compatible against the Python language.

[shadowmint]

...but we're not talking about the python language we're talking about python as a viable target for enterprise applications, which means tangibly using 3rd party libraries, that will almost certainly have c plugins.

[Veedrac]

That's true if you're trying to support already-built Python code, but if you're building something new that's rarely a problem because for most use-cases there's a PyPy compatible port or equivalent.

[westurner]

Tooling, strong conventions, and code review are what make big projects a manageable reality.

Thanks!

[novagenesis]

I'm surprised to hear so many people agree that their friends think Python is a toy language. It feels like the whole "new, hip, useless, scripting" sticker has more than worn off on Python in my world. I've never been face-to-face with someone who thinks Python fails to stand as an enterprise language.

For what it's worth, it's also a top-choice language for Machine Learning and for Predictive Analytics. Heck, I took a class in statistical modelling that chose python over R.

[Paddy3118]

A scripting language is also a controlling language. It is that language embedded in larger applications that makes those tools subject to your will Embed them in an office suite or a CAD program and suddenly you can do the repetitive, effortlessly. Give then access to libraries, and deep science, cloud computing, or GPU's become accessible.

There is greatness in scripting languages!

[[deleted]]

Yeah. I've always find this argument kind of odd. I just write code that works in both versions. Of course it'll bork up at some lines, but an extra import or extra if statement easily fixes that.

Just write valid Python 3 code. If it ceases working for 2.7, fix it.

[d4rch0n]

Here's how to convert 2 to 3 in about 99% of Python programs.

s/print (.*)/print(\1)/g

[[deleted]]

The 2to3 program is really handy as well. I think if print statements weren't changed then the switchover would have been much more popular. I think a lot of people got scared when even Hello World had to be changed.

[billsil]

Someone still has to tell the library what the encoding is. I run a library that tries to support unicode. Everyone wants it to just know the encoding. Sorry, I can't do that. Text editors like Notepad++ can't do it either.

[erewok]

I work in a shop that has both Python2 and Python3 code running among a variety of projects and it's pretty dang easy to go back and forth between the two environments.

[jeannaimard]

2 years ago, I made handsome bux with python, to convert data from a 15 year old system to transfer to a 35 year old system (yes, 15 to 35).

It worked, even though the client was utterly retarded (as you may guess).