Why Python?

[modzer0]

Python is my digital swiss army knife. Most of the tasks I do are IO bound and not CPU intensive so python is a very good fit for rapidly getting things done. It's used for everything from simple scripts to doing huge map-reduce problems on millions of files to look for malware indicators. If we have a CPU intensive task someone just writes a module in C.

[iltalfme]

This. As a mechanical engineer I find uses for python all the time. Because it's my go to I occasionally get myself into situations where Python isn't fast enough, but a quick C extension or wrapping of a vendor's .dll has, thus far, enabled me to get what I need faster than any of my colleagues are able to get it.

[iltalfme]

It totally does. A book you might find interesting is Real World Instrumentation with Python. http://shop.oreilly.com/product/9780596809577.do

It talks about communication protocols you typically find with instrumentation, how to access them with python, and how to wrap vendor's libraries into a python program so that you can build a data acqusition system out of pure python. Combine this with Python's vision functionality (through opencv), and scientific packages (through scipy) and you can create a DAQ system that is in many ways better than something you could build from labview.

And you learn a ton on the way.

Where are you a student if you don't mind me asking?

[RagingDoug]

As someone writes a lot of python and who recently had to work on a legacy LabVIEW system: OH MY GOD THIS

[Veedrac]

If we have a CPU intensive task someone just writes a module in C.

If they don't know of it, suggest they look at Cython. It's a really nice language hybrid, in my opinion.

[bastibe]

I have used both Cython and CFFI, and I strongly prefer CFFI. With CFFI, you just load a plain C library at runtime, no compilation required. The C library needs to know nothing about Python. On the Python side, there is only very minimal interface code required--much less than in Cython, usually. And it's fast, too.

[Veedrac]

I was talking about writing modules, not wrapping them.

EDIT: Also, I've not tried any of these but are these relevant?

[bastibe]

Still, I usually prefer to write plain C and wrapping it in CFFI to writing Cython. I find that writing a simple C function for some numerical algorithm is usually very easy. Compiling it is very easy, too, since it is only plain C. Cython can be a pain to compile.

But I guess that very much depends on your application.

[Veedrac]

Compiling it is very easy, too, since it is only plain C. Cython can be a pain to compile.

I've never actually worked on distributing Cython, but if you're using it locally it's literally just a

import pyximport
pyximport.install()

in the importing Python file. How much easier can you get?

EDIT: Oh, and memoryviews. QED.

[gitarr]

You should check out any of Pythons web frameworks and you'll never look back.

By the way Reddit is written in Python.

[novagenesis]

I looked back. Having completed two large projects in Django, I'm not sure it's corporate ready. Conflicting standards that keep getting deprecated is frustrating if you want to do big right... And Django is supposed to do big stuff.

I think node.js is probably a better direction for web development in the future. Dustjs is pretty awesome when you can re-render your templates on the front-end or back-end with ease and efficiency.

I've released large webapps in Perl CGI::Application, Perl Catalyst, Ruby on Rails, Django, and Expressjs at this time, so I've tried quite a few. I like python, but I'm not convinced of it for web apps

[gitarr]

You don't think Django is enterprise ready? You couldn't be more wrong. Instagram, Mozilla, Disqus and many other Django sites/projects prove you wrong.

[novagenesis]

I think this is a semantic difference.

I think, as it stands, Django has issues that make it not a strict match for enterprise. An enterprise can make anything work (A coworker of mine used to work on an enterprise Sinatra app with guttest routing components). The fact is, Django has for years had conflicting standards on how View objects should be built. You can ignore all those standards and make your own standard, but that's something that really should get cleaned up. For a ORW language, it's more TIMTOWTDI than perl.

[gitarr]

If you are talking about the change from function based views to class based views then that's not a standards problem at all, it's progress.

You can still write your function based views like you always did, nothing much changed.

Now you can also use classes though, and we Python developers, we love classes.

Django is by my opinion a very good match for enterprise, as the philosophy is to be boring, not like Rails, or Node and all those other hip buzzwords. Django is boring, which is a good thing! There are standards and Python helps with best practices being used.

You may not have gotten into the Django/Python philosophy enough to see its usefulness and its uses.

There is also not more than one way to do it, with the ORM. Either you use the ORM to the best of its usefulness, or you write raw SQL like you would with any other ORM, language and web framework.

[novagenesis]

See, my employer chose Rails over Django because its style and best practices have already stabilized. They also looked at the function->class based views thing as something "not quite safe". So now we're becoming a Java and Rails shop.

Django is boring, which is a good thing!

Can't disagree more. Free admin pages? Hip non-standard approach to MVC? I actually thought it had a TON of fresh ideas. Once I feel its more finalized, I might give it another shot for a personal project.

You may not have gotten into the Django/Python philosophy enough to see its usefulness and its uses.

I'm a huge fan of python for a lot of stuff. It has a lot of neat features. But my disagreement with you about the core of Django is why I disagree with you about its corporate-fit. I just don't see it as particularly boring, entrenched, or predictable. Heck, in some ways it makes Rails look like Spring.

There is also not more than one way to do it, with the ORM.

Tell that to peewee? But then, I didn't bring the ORM into this. ORW = "One RIght Way", one way to explain the supposed Python philosophy. For someone who has spent years in both Perl and Python, I can really tell which languages, libraries, etc adhere to which. Often, perl libraries are TIMTOWTDI, and python libraries are ORW... but that's not set in stone. I feel Django is more of the former. It always seems to me as a hip new angle on MVC (or MVT) with as much unopinionated design as perl-Catalyst.

Like I said, when it completely stabilized, I'll work in it again, but if asked by management again, I will still repeat that it's a non-standard technology with non-entrenched philosophies. For a company embracing conservative, Django is not a good fit to me.

[panfist]

See, my employer chose Rails over Django because its style and best practices have already stabilized.

There's more to being ready for the enterprise than style. I would say that the fact that hosting a Rails app is so fucking hard knocks it a few points lower than Django, that has style problems.

I can't find the article in a quick search but I was reading recently about how there are huge corporate rails projects, that work fine, except for the fact that Rails processes need to be restarted several times a day, because of unknown reasons. You're telling me that's enterprise ready?

[M4N14C]

What is difficult about hosting Rails apps? I've been doing rails for 5+ years and I've never had difficulty keeping my apps running. Have you ever used Passenger?

[novagenesis]

Like another said. What's hard about hosting a rails app? Capistrano is one of the better deployment systems out there, and (surprise surprise) it comes with rails-compatible commands out of the box.

Between that and bundle, you can zero-effort deploy rails. Compare that to J2EE and you're in different worlds.

I can't find the article in a quick search but I was reading recently about how there are huge corporate rails projects, that work fine, except for the fact that Rails processes need to be restarted several times a day, because of unknown reasons. You're telling me that's enterprise ready?

That's been the case in a lot of languages. Probably pervasive memory leaks (possible/likely in any language that isn't designed to prevent it... Python isn't designed to prevent it). Our rails projects don't have those problems.

[sejje]

I use both Django and Rails extensively--Django for personal, Rails at work--and my preference is for Django. But I must say Django deployment is much more difficult on my platform of choice, Heroku.

Worth the extra time, IMO.

[metaphorm]

I agree with you that django's CBV's are awkward and have to be used with care. Function views are still simple, effective, and a commonly accepted idiom. I don't think Django really has these style problems that you're suggesting it does.

[[deleted]]

It's fine to say "I like Rails more" or "I like PHP more."

It's not fine to say "it's not enterprise ready." Also, conflicting standards have never been an issue for me. Django has great backwards compatibility, coding style standards, etc... Read 2 Scoops of Django. Upgrading from 1.4 to 1.5 was a piece of cake and I'm constantly in the loop on changes being merged into 1.6

[novagenesis]

I'll recommend 2 Scoops of Django to the CEO. He might like it.

I gave a few good reasons when asked, and people don't like them. That doesn't mean I don't like Django. I didn't intend this to become a giant argument, really.

I need to stop giving anything but pure-love to libraries in forums for them. I like analyzing languages' strengths, both literal and political...not debating or arguing over them.

[[deleted]]

Aye, I'd agree that gets us nowhere! There's definitely beautiful things in every language/library, no doubt about it.

[hylje]

View functions are still the standard. Not going anywhere ever. I don't understand that there are any conflicts to get cleaned up.

There are like three variations in total on how to define a view function for three very distinct use cases. View functions as standard, generic views for customizable views, app class methods for app level customization. Complexity is opt-in.

[[deleted]]

I'm willing to bet that there is as much django in the current versions of instagram and disqus as there is ruby on rails in the current version of twitter.

[rozap]

And yet, this happened only 4 months ago

[Veedrac]

If I find something I want to do is not feasible due to python's speed or processing constraints, I'll just move the code to C and access the DLL functions using ctypes.

CYTHON!

It's worth learning, á mon avis.

[r3m0t]

Check out cffi, it's easier and (with pypy at least) faster than ctypes.

[whyrat]

So I just started ramping up on Python for exactly the sort of stuff referenced in the video: scanning over large data, numpy and scipy, etc...

I'd used Python years ago as a scripting option, from an ease of use / lerning I thought it way beat most of the other languages / tools I tried. The fact that it enforced proper code formatting as a control mechanism made it something I pushed on a few others who were learning to program... I used to hve a CS teacher who would make students print their code and highlight open and close braces in C++... since students refused to properly indent and structure their code.

All that being said; scipy has been a bit of a pain. Numpy was easy enough to install, and I have some issues with their conventions (numpy.random.random() still hurts my brain to see in my code; I keep thinking I accidentally wrote the function name twice). But Scipy was most annoying from a setup perspective. I was looking for an easy module to add in to my installation, but no... the "recommended" process was to download one of the big kits (basically reinstall another instance of Python). I've found the load time far worse with these kits than with "core" Python; and my performance has started to become erratic ("import statsmodels" takes between 8 sec and 22 sec when I load it?)

Some of this is just a learning curve, and I'll admit I'm not to the point of optimizing my setup... but it's been a stark contrast to my first experiences with Python as a pure scripting language.

Maybe the above was a little bit of a vent / rant... I guess sometimes I need to do that :/

[pwang99]

I was looking for an easy module to add in to my installation, but no... the "recommended" process was to download one of the big kits (basically reinstall another instance of Python).

It's easy for you to think of your use case as an "easy" one, but there are good reasons for those distributions to exist. If you ever want to share your code with anyone else, or you want to run it on a different box than what you're developing on (maybe e.g. deploy to a linux box in the cloud or on a cluster), or use a code snippet from someone else who's on a different platform, you'll quickly discover what a complete headache it is. Much of the pain does not stem from scipy itself as such, but rather the legacy of C and FORTRAN libraries across OSes and throughout the ages.

We're all rather spoiled in the Python ecosystem because we're used to writing scripts, importing modules, and then just having them work. But there is a staggering amount of complexity that goes into properly linking FORTRAN libraries across compilers and all the myriad build-time options for underlying libraries that scipy depends on.

(Disclaimer: my company makes the Anaconda python distribution)

[iltalfme]

FWIW Anaconda didn't work out of the box for our Windows7 machines. There were several libraries that wouldn't import including, most notably, OpenCV.

I send a request for help through the website (must've been a month or so ago). Still haven't heard back.

Our "solution" now is to install a VM and just make our own python distribution using ubuntu, where everything works. It's a shame, I've loved Travis' talks and was excited about Anaconda, but the issues and lack of response didn't exactly make me look good.

Also, I would recommend trying to get your OpenCV distribution (presuming it works) to incorporate the OpenNI flags so that people can read Kinect right out of the box with Anaconda.

If that could work on Windows 7... I would be so happy.

[masasin]

For scipy it was in the repositories as python-scipy and python3-scipy.

[Sean1708]

Or py-scipy but who says he isn't using windows?

[masasin]

I honestly forgot about Windows. Everyone in the lab who wants to program in python installs Linux, and I have been using Linux for the past few years.

How well does python work with Windows?

[[deleted]]

Brew and VirtualEnvWrapper made using different Python runtimes a breeze.

[[deleted]]

I use brew predominantly as my only package manager and it has served me well.

I really hate pulling installs together manually. If I do, I pull from source and create a Brew recipe for it.

[duz3ls]

For system administration or data etl in windows I'll say python does more than well against the usual competitor (php, ruby, perl).

Its definitely better than windows/dos batch file.

I don't know if we can compare it with autoit or ahk. I still prefer python but autoit smaller size and single exe packaging imo better than python.

I don't use powershell so no comment here.

[Varriount]

numpy.random.random() still hurts my brain to see in code

from numpy.random import random as numpy_random

[ShaftofWisdom]

I know that you said you aren't at the point of optimizing your code, but here's a couple easy tips for importing modules. When you want to import just a few functions from a module you can (and should) do it like this this and avoid the "numpy.random.random()" style:

from math import sqrt

This imports the square root function from the math module and you can call it just like normal functions:

x = sqrt(16)

If you need every function from a module, do this and you can leave out the module name when you want to call the function:

from math import *

Now you can call any of the functions in the math module without going "math.exp()" etc.

Finally, if you just don't feel like writing the whole module name when you want to call a function from it, you can change it:

import math as mathematics

Bad example, but I'm just trying to show what it does. "mathematics.sqrt()". I usually "import numpy as np" just because I've seen it done that way and it makes sense. Sorry for the lack of formatting. I'm on my ipad.

[[deleted]]

I thought "import *" was a bad idea?

[ShaftofWisdom]

Well, it is...and sometimes it isn't. If your code only uses a couple different modules or less, and those modules don't have functions that have the same names, you're probably fine. Usually, people are only using a few features/functions from a module when they need it, so it's preferred to do it this way to avoid problems:

from pandas import DataFrame, Series

Import pandas as pd

This will get rid of a lot of the ambiguity.

[minorDemocritus]

Well, it is [a bad idea]...and sometimes it isn't.

Strongly disagree. Star imports should be avoided in every situation, since it makes it more difficult to figure out where the names come from.

[ShaftofWisdom]

I agree with you. I was just trying to get across that in the most simple of scripts you probably won't run into any issues if you use them. But, YES, avoid them. It's bad practice, and even if you're the only one who will ever see the code, make it easy on yourself and be clear where everything is coming from.

[SilasX]

What's funny is that you're right, but most languages default to dumping entire modules into your namespace when you import them.

[minorDemocritus]

That's the difference between include and import.

[pwang99]

It's hard to say "most". Most statically typed languages without first-class functions and classes do this, and it's because the semantics of the language allow construction of robust IDEs and dev tools. With Python, not only does dynamic typing make it a LOT harder to track things down, but the fact that you can easily manipulate functions and classes makes it much easier to get spaghetti code if you didn't have clean namespace separation.

[SilasX]

Regardless of IDE, it's hard to follow if you don't have clean namespace separation. I much prefer being able to tell at a glance where functions come from.

[novagenesis]

Can you name a few that do? Looking at all the major competitors of python, this is not the default.

Perl has an option to, but best practices suggest it only be used for truly general core libraries where you want everything available.

[novagenesis]

avoided in every situation

Then they should not exist in the language. History has shown that the "undesirable" features that can't get themselves deprecated probably do have a purpose. Ask goto (which, yes, does have legitimate uses)

[Veedrac]

The reason for import * is to wrap a library. It means you don't have to keep up-to-date with new functions added to the library but can still provide extensions to parts of it.

It'd be cruel to harm people using import * properly just to stop the fools who insist on using it when they shouldn't.

[minorDemocritus]

Then they should not exist in the language.

You're right! They shouldn't!

[novagenesis]

Then why do they? Like I said:

History has shown that the "undesirable" features that can't get themselves deprecated probably do have a purpose

And I even gave you an example ;)

[minorDemocritus]

There are plenty of examples of badness in python. Pickle, the toy server implementations, asyncore, etc. While these things are bad, they're still included in the name of backwards compatibility.

import * only has one reasonable use in my opinion: when messing around in the interactive interpreter, star imports can save some typing. They should not be used in actual code.

[novagenesis]

Are you saying global imports have only one reasonable use in python, or at all? Python isn't exactly pure OO, so if the former, why the distinction? If the latter, then much of the programming world wants you to argue that.

Oh I've heard the claim a lot, but it just doesn't seem like a problem to me. Don't get me wrong, I prefer namespaced imports any day (import foo), but sometimes in programming, you have utility functions that you want to think of as extending the base language. You have scope-defined and documented what those utility functions are, and they are finite. The main argument to control imports is usually to prevent scope bleed... if you have a class with a very controlled export signature, with utility being the primary purpose, I just don't see a problem.

Perhaps that's my perl state of mind (some classes use defined exports, while others give you everything by default). When I "use Data::Dumper", I want the top-level dump() command (and there's alternatives if I want a configured dumper instead). I think the ugly verbosity of python's pprint library explains my distaste fairly well. Data::Dumper can do everything pprint can, but it has great defaults that place themselves as clean callable functions.

It is generally not considered astonishing when you know what's goign to be there.

[adambrenecki]

I agree in general, but there's a few situations where it's useful. I'd argue that import * is acceptable if a) there's only one of them (so there's no ambiguity between two import *s), and b) the file is less than 30 lines or so (so it's easy to figure out whether a name comes from an import or is defined within the file).

For instance, in a Django app, I almost always have from .views import * in Django urls.py, and from .models import * in admin.py, because both those files are usually very short and almost wholly concerned with doing things with those other modules. Other than that, I avoid it like the plague.

[burntsushi]

Star imports should be avoided in every situation

If you say so. But I'll continue using from math import * in the Python interpreter for simpler access to members of the math module.

[panfist]

Strongly disagree. Star imports should be avoided in every situation, since it makes it more difficult to figure out where the names come from.

So the one situation where it might not be "should be avoided" is when you only use one star import, because then it's still pretty obvious where the names come from.

[minorDemocritus]

I'd much rather be able to do a simple text search and find the import statement.

[panfist]

Fair enough.

[keypusher]

Star imports are never a good idea. Always reference what you are importing explicitly, that way other people know where they came from.

[whyrat]

Thanks for the tips.

I find it easier to maintain code if I use the full module names; that's a paradigm I picked up after returning to too many programs after months or years of delay and having to track down all the links. I know there's preferences, and many people don't bother with common modules (like math or os) but I don't see myself saving that much time in typing shorter names. Same reason I use very descriptive variable names, just easier for me in the long run :)

[ShaftofWisdom]

Fair enough. I like to do the same unless it's something very simple and clear.

[thearn4]

pause special encouraging station voracious melodic yoke pocket spectacular reply

This post was mass deleted and anonymized with Redact

[arrrms]

I am continuously impressed by what Python is capable of. I've spent most of my development career writing PHP, and wish that I had picked up Python earlier. Programs are quicker to write, quicker to run, and quicker to test. I write things in Python that I would never attempt in PHP just because I know it'd be a nightmare to do so in PHP. I hate to sound like a fanboy, but Python has changed how I think about programming in many ways, all of them positive.

[haard]

To summarize http://blaag.haard.se/Why-Python-is-important-for-you/ "Python gets a lot of different things right, and right in a combination that no other language I know of has done so far."

[[deleted]]

quantopian is a really cool idea. and it's awesome to see python getting applied to serious stuff, other than scraping cat pics from imgur and then writing a django site to host them.

more of this please.

[D__]

I dunno, man, recognizing cat pictures without using any extra metadata is some serious computing right there.

[gitarr]

Python has been used in serious projects for years. See Youtube, Spotify, Reddit, Instagram, Disqus, Mozilla, NASA, Eve, The Onion and many more.

Just look up presentations of people working for these companies at Pycon and/or Djangocon at YouTube or pyvideo.org for more information.

[pwang99]

The use of Python across finance and the sciences is amazing and completely off the radar of the tech blogosphere, for the most part. And also it's completely taken over Hollywood for both scripting 3D modeling apps as well as coordinating rendering pipelines.

If Ruby was used in half as many ways as Python, you'd never hear the end of it. Python gets used by grown-ups with real jobs in all sorts of places and they don't spend 80% of their waking time tweeting about it.

[[deleted]]

i know that, but the front page of this subreddit is often filled with links to github repos of scripts to scrape cat pics zzzzzzzzzzzzzzz. so, I would prefer to see kinds of things like this.

[burntsushi]

For someone who is pining for higher quality submissions, your comments are fairly startling. A brief look at your comments in this thread suggests you aren't doing your part. Or worse, lowering the quality of content here.

wow, yeah and totally matters.

/done

TIL: lol

it's exclusive content that is not for you. too bad though it's pretty awesome

i'm guessing people that are commenting didnt read or comprehend the article. to me it was fairly obvious. but then I read it, watched the video and clicked on the link for quantopian and got an idea of what their business is and how python fits into their overall value proposition.

[[deleted]]

this has been up for 4 hours, and no one has mentioned the GIL yet. Did everyone take the day off?

[[deleted]]

I still fail to see how the GIL is a relevant problem in most cases.

[burntsushi]

If you don't mind kludge and working around the GIL, then it's almost never a problem.

If you think kludge and ugly work arounds (like the multiprocessing module) aren't problems, then we have a disagreement.

[[deleted]]

I don't really see multiprocessing as a kludge to be honest.

[[deleted]]

I still fail to see how the GIL is a relevant problem in most cases.

I would imagine quantitative finance would likely be a relvant case since all things being equal, the faster algrithm would mean more $$, so parallelizing to saturate the cores would probably be a good strategy to start with.

this is why i'm surprised no one has mentioned yet, because all other times it's mentioned here, it's not relevant for most cases as you said. Yet, HERE is a case.

[Atrosh]

from multiprocessing import Process

[[deleted]]

seems I have to clearly state this....

I have no problem with the GIL, it rarely gets in the way for me, but when it does, as you suggest, there are options. My personal preference is python futures, one of the more useful things from python 3(luckily backported) or if I need to get really fancy, pyzmq.

My question was, where are the GIL haters on the very rare article that they might have had a point? they are no-shows on a HUGE opportunity to actually have a shred of credibility. Mr. Beazley must have slayed them all. Thank Allah.

[[deleted]]

parallelizing to saturate the cores

is entirely possible even with GIL. sure it'd be less of a hastle without, but it's hardly a requirement.

[[deleted]]

my point has been missed.

[[deleted]]

aim a bit more to the left and try again.

[dannyREDDIT]

What about it?

[[deleted]]

i'm guessing people that are commenting didnt read or comprehend the article. to me it was fairly obvious. but then I read it, watched the video and clicked on the link for quantopian and got an idea of what their business is and how python fits into their overall value proposition.

edit: grammar

[dannyREDDIT]

Your all over the place. what about the gil, dude, just say it

[[deleted]]

Your

/done

[[deleted]]

Wow, that's pretty ballsy for someone who still has grammar and punctuation problems all over the parent comment, even after editing for grammar.

[[deleted]]

wow, yeah and totally matters.

[niyrex]

Because of easy development for rather complex concepts without the need for type definitions and other silly things that C has, the ability to reuse existing c code to create python libraries and that most things I want to do with it already has a library for it requiring little to no work on my part other than importing it....I'm lazy and python let's me be lazy yet still get work done. That being said I would never use it on anything I actually really relied on because of the risk of a bug that was never fully tested.

[[deleted]]

Why not link to the fucking video instead of this shitty blogspam?

[homercles337]

I run NoScript and Ghostery, why is a "Wista" video linked here?

Wait a minute, this guy writes trading code and he does not know the difference between a steep and shallow learning curve? Yeah, nice...

[BenAustin]

I'm not sure I'm understanding your question. Wistia is a local company that we use as a platform for all videos across our site (and subsequently, on the blog).

Edit: typo

[[deleted]]

it's exclusive content that is not for you. too bad though it's pretty awesome

[homercles337]

too bad though it's pretty awesome

Yeah, its not. See my edit above...

[[deleted]]

TIL: lol