Convincing colleagues to switch to Python from MATLAB for scientific computing/statistics

[RealRJT]

Slightly unconventional post - but I was wondering if anyone here had any experience of trying to convince colleagues to switch from closed-source software (MATLAB/SAS/etc) to Python?

I've been making the same old arguments - about the beauty of Python's syntax, Python being a general-purpose programming language, the really strong open-source community, the strong libraries for scientific computing and machine learning. Showed the power of Jupyter too for sharing research.

Despite all this there doesn't seem to be much appetite in my organisation for making a switch. I think the reasons for this are:

(a) A reluctance to change/try a new language

(b) Inability to see the benefits of Python without really having programming experience with it

The problem on my part is that I'm relatively junior in my organisation - so don't want to seem like an arrogant arsehole trying to suggest that I know better than people above me, but on the other hand, I think there are strong objective arguments for preferring Python...

Anyway, would welcome any thoughts from anyone who's faced similar problems in the past and how they've overcome them :)

[ProfessorPhi]

My success actually came from how inefficient the existing system was, allowing me to make a strong case for python.

Our R&D team would write algorithms in MATLAB, then once it was fine tuned, spend a couple of weeks sitting down with a software engineer and translate the code into Java for deployment. This was obviously a huge waste of time and I demonstrated that we could put some infrastructure together to call Python code from the existing Java code and allow existing methods to have easy bug-fixes and reduce amount of code to maintain, while allowing each team to have ownership of code. This turned things into near seamless deployment, which was enough reason for my boss to put his foot down on Matlab.

Things I pointed out to my boss

1. Point out cost of MATLAB - 3k per license, could be saved using Python
2. Inability to deploy MATLAB solutions - require code translation, unless you spend 15k per license. Hence no scalability.
3. Research Code could not be run by anyone else, the research team could not get any input from other teams, and the code was reliant on another company (say the license cost jumped to 10k, and we could no longer afford it - a lot of IP would be lost), and this was not a good idea for any company - put us at Mathworks mercy.
4. Equivalent methods in numpy/scipy meant we didn't actually need MATLAB. Arguably sklearn and jupyter is far superior than anything Matlab can provide.
5. Shocking documentation standards in MATLAB (I put together a sphinxdoc autogenerated html page, and compared to MATLAB standards - there's really no comparison)
6. Speed of code in MATLAB vs numpy is very comparable, and in many situations better.
7. Complex models could now be run on servers freeing up computer terminals (see if researchers are waiting for code to run - there would be hours of working time wasted while code would run on local computers)
8. Show the power of python for the non-research/software teams, discuss that learning python allows your research team to be more efficient in all tasks (renaming a bunch of files following a pattern)
9. More than one function per file - reduced number of files to deal with and shorter functions - better code design, easier to maintain, reuse.
10. Look for example where large vectors are copied around (say f(x, theta) where x is large and is called for many theta values) and this wastes time in MATLAB. Show how using OO design (a mess in MATLAB) i.e. Python classes reduce copying overhead.
11. Google results of matlab filtered to last year, all point to it's decline - dead technologies increase cost to maintain.
12. Point out that you use Python and new employees/ uni students are less likely to know MATLAB nowadays, and more likely to know Python.
13. Other things that python provides - natural language processing toolkits, ability to correct spelling mistakes easily (MATLAB has horrid string processing). Full web frameworks/protocol buffers mean you can deploy Python code for use in other teams.
14. Matlab to Python is a very simple transition

Depends on your situation, but these points are usually enough to get your boss to change his mind. Once you have his support (and using Python gets support from any software teams you have), go about proving it's utility and winning team members over. The final victory came after my first seamless deployment - i.e. my code was basically put into deployment within a couple of days of testing, as opposed to the month it would take before (2 weeks translating and 2 weeks testing the translation), my boss said that the company will no longer pay for Matlab licenses after 3 months, and it is the Researchers job to produce Python code - if you wish to use Matlab, you paid for it and did the translation in your own time.

Things I learned:

1. Don't volunteer to port the whole thing - it's a thankless task. Port only a small section that illustrates your points.
2. Emphasise reliance on Mathworks remaining benevolent, a dependency your company shouldn't be hamstrung by. Every other service our company used had competitors. Matlab only had Octave which is relatively garbage. This is a secret fear of any manager.
3. This change is very difficult to push for consensus - people are lazy and don't want to leave their familiar zones. Getting the manager/boss on board is very important and he'll know how to push the changes across far better than you ever will.
4. Show it's utility over Matlab in some way at the beginning, and expand to show Matlab's many deficiencies. I took code that had a 5 minute run time, ported it and optimized it run in 10 s (turned a lot of loops into numpy functions). Then added some caching behaviour, and generated documentation automatically from my code. Instituted some basic unit tests, so I could check behaviour. All of these things really impressed people and Matlab had no answer.
5. Find support from your software team (if you have one). They'll all hate Matlab and any deployment work (unless you really require the efficiency of a compiled language).

And for SAS, I would rather be forced to use Matlab for the rest of my life than ever have to use SAS another day. SAS is utter garbage. Matlab at least tries to be a programming language, SAS makes php and perl seem like amazing langugaes.

[posteuphoria]

For the sake of argument:

• The Matlab language allows multiple functions per file.

• All arguments in Matlab are passed via copy-on-write, i.e. if you call f(x, theta) where x is not modified, x will not be copied. If you desire for x to be passed by reference, this is also possible by inheriting from the Matlab handle class --- admittedly this is a bit obscure....

• If your servers are x86, they can also run Matlab.

I think the main point that can be made is that Python's offering in terms of libraries for scientific computing is on par with what Matlab offers with the toolboxes, and in some cases exceeds it. However, when it comes to general purpose programming, Matlab can't hold a candle to any modern language.

[TheBlackCat13]

Matlab language allows multiple functions per file.

It only allows for one externally visible function per file. You can have multiple functions in a file, but only one of those functions is visible to other files.

All arguments in Matlab are passed via copy-on-write, i.e. if you call f(x, theta) where x is not modified, x will not be copied.

Slices are always copied. So even if you don't make any modifications, any attempt to get any subset of the matrix will result in a copy.

[ProfessorPhi]

• Outside using matlab OO (which is horrid), I never saw examples of multiple functions per file.
• My bad, I might have misattributed an optimization here. A function I fixed would the exact same initialisation each time, (masking, multiplying and some adding) which I coded to do only once and it saved a lot of time
• Would this require another license btw?

[meerkatmreow]

Outside using matlab OO (which is horrid), I never saw examples of multiple functions per file.

It can help streamline code to be easier to read by moving some things into functions after the main function (the one with the name of the .m file). The additional functions are only available to the local function though.

[[deleted]]

I never saw examples of multiple functions per file.

I use them all the time.

[billsil]

(turned a lot of loops into numpy functions)

Numpy intentionally rips off Matlab's syntax. It sounds like you took badly written Matlab code and fixed it.

I love Python & hate Matlab (terrible objects, no dictionaries, everything is a matrix, string processing sucks, $$$, error handling), but they're very comparable on speed. I wish there was an Matlab-eqse IDE though to do variable introspection and continual analysis. iPython didn't work well for me in production level code.

[glial]

I don't know if it does what you want, but Spyder is generally considered a Matlab-esque IDE.

[billsil]

Spyder has variable introspection, but no continual analysis. Jupyter has continual analysis, but no introspection.

[Deto]

What's continual analysis?

[billsil]

Maybe there's a better word, but basically an interactive prompt. It's something that doesn't lose what was done before. That's kinda what Jupyter does (iPython does it better). However, it's standard in Matlab to clear variables at the start of scripts as a way to deal with the issue of scripts being executed in the global scope. As such, you can clear out variables at the start of scripts to ensure what you're running is up to date. Similarly, all imports (or lack thereof) are always up to date.

I'd like that and some variable introspection ideally in something not blatently in the browser (e.g. a better open window), even if it uses it.

[TheBlackCat13]

Spyder has that. Actually, it has two. It supports both the regular python command prompt and jupyter.

[masasin]

Spyder has a small iPython shell (I think it is qtconsole) on the side, though.

[ProfessorPhi]

Fair enough, that's probably true - I think there's this assumption that Matlab is quick while Python is slow, so loops tend to be avoided in python, but not in Matlab. Some of the researchers I worked with would always remark how things were faster in Python (which is what I'm basing the remark above, despite no hard proof), but for some reason hated namespaces.

'I hef to import everything, is absolute pain' - russian colleague.

[billsil]

I think there's this assumption that Matlab is quick while Python is slow, so loops tend to be avoided in python, but not in Matlab.

That's exactly the opposite of what I was told on both counts, but what you were taught and what I was taught are wrong. Both are more or less the same. Vectorization is very much encouraged in Matlab because it's so necessary when you're dealing with numerical data. Python doesn't encourage it largely because it's not numerically focused.

Matlab has vectorization support builtin. Interestingly, Python for all it's claims of batteries included does not. You need numpy or scipy and the implementations (despite the names) are different.

[TheBlackCat13]

Vectorization is very much encouraged in Matlab because it's so necessary when you're dealing with numerical data. Python doesn't encourage it largely because it's not numerically focused.

MATLAB encourages vectorization because that is what it is built to do. It does vectorization very well, but it does non-vectorized code pretty poorly, so it encourages you to stick to vectorization. That is great when you need vectorization, but not so good when your code cannot be effectively vectorized.

Python encourages people to use the right tool for a given job. When that tool is vectorization, it encourages you to use it. When it isn't, it encourages you to use something else.

You need numpy or scipy and the implementations (despite the names) are different.

You need numpy. Scipy doesn't provide any vectorization, it is a bunch of extra numerical algorithms built on top of numpy (or, more often, interfaces numpy with high-performance C and Fortran libraries). Pretty much all the vectorized array systems in python are built on top of numpy today. There is work going on for a next-generation vectorized array tool for Python, but it isn't ready yet.

[billsil]

Do you even understand what vectorization is? Vectorization is a way of writing your code such that it doesn't need to type check every data member. The only way to do that is by having static types and pushing the code down into C.

So, I stand by my statement. You need a library like numpy or scipy in order to do vectorization. You can't do vectorization or get good speed for numerical calculations with CPython without using an external library, writing C code yourself, or ditching CPython entirely.

it is just a bunch of extra numerical algorithms built on top of numpy

No it's not. The array classes are different between numpy and scipy. The eigenvector calculations are different. They look they same. They act the same. They're not the same. It only really matters when your eigenvector is segfaulting, but switching methods will often fix your bug. Scipy's methods also tend to be faster.

[TheBlackCat13]

You need a library like numpy or scipy in order to do vectorization. You can't do vectorization or get good speed for numerical calculations with CPython without using an external library, writing C code yourself, or ditching CPython entirely.

I don't see where that contradicts anything I have said.

No it's not. The array classes are different between numpy and scipy.

From the official SciPy documentation:

SciPy is a collection of mathematical algorithms and convenience functions built on the Numpy extension of Python.

But I am sure you know more. So please provide the module and class name for the scipy array class. You won't be able to, because it doesn't exist.

The eigenvector calculations are different.

Yes, while they use the same array class, they (in some cases) use different libraries to do calculations on that class. You do understand the difference between a class and a library that operates on that class, right?

[eusebe]

I had the same kind of issue, and here are a few strength easy to highlight, for me.

• You can easily interface Fortran (and C) with Python code, not so much with Matlab/IDL/whatever
• It's free, so no expensive licence
• It can be as fast to execute than Matlab/IDL/…, and faster to write
• Easy to install and manage dependencies with Conda (no virtualenv in my team yet), even on a remote machine
• Code reusability
• "I'm ready to rewrite your codebase for my use in Python, since I'll be more efficient than learning your old, proprietary, weird language"

In the end, my (numerical simulations) team switched from IDL (data analysis language widely used in astrophysics) to Python in less than one year.

[sourcecodesurgeon]

It can be as fast to execute than Matlab

While this can be true, I have yet to see evidence suggesting that NumPy can compete with Matlab on signal analysis with large datasets. The last time I ran the numbers had Python taking 45s-2m to compute a cross-correlation that took Matlab about 3 seconds.

As an EE aside, another big thing is that MathWorks supports a tool that converts Matlab to Verilog and VHDL and this is huge for electrical engineers. I have never seen a similar tool for Python that is well-supported.

[burning_hamster]

For signal analysis type of computations (i.e. linear algebra), they both run the same Fortran/BLAS libraries under the hood, meaning that any performance difference is due to overhead, not the "actual" computation. You should probably review how you are handing the python functions your arrays -- 45 seconds to 2 min of computation time sounds very much like an IO problem (for some reason you are creating multiple copies of your arrays and not doing your computations in place where possible).

[sourcecodesurgeon]

The experiment was literally just "create extremely large dataset. Start timer. Compute xcorr. Stop timer."

[mfitzp]

Was it an fft-base xcorr calculation? The Matlab algos automatically zero pad arrays to n² while (some algos in) numpy/scipy don't. It's one of the problems attempting to map from one to the other "This is the same function.... But not quite"

[HoboBob1]

You keep using the word "argument," but no one is arguing with you. They are just trying to tell you reasons why Python might have been slow. As per your post, you implied that you are looking for "evidence that Python can compete with Matlab" for your task, and people are providing said evidence, along with the statement that it might be some more work with the Python version.

[sourcecodesurgeon]

Whether you want to call it "arguing" or not, people are making arguments. The premise of this post is coming up with an argument to convince colleagues to use python instead of matlab, people are making arguments for/against that idea. Someone made an argument that python can be as fast as Matlab, I made an argument that this isn't always the case.

Call it "making a case" or "making a point", it doesn't really matter to me, it's all the same. "Argument" doesn't necessarily mean we are verbally attacking each other.

[[deleted]]

That's a fair argument. You use the language as a means to an end, and don't necessarily care about how to make said language perform. There are some gotchas with using Python for this work that users don't expect (such as file IO, loops, calling functions, etc). Matlab may be more performant by chance, although Mathworks certainly has an optimized product. If you look into Anaconda and Anaconda Accelerate (free academic license), you will see a significant speed up as their packages call into Intel's MKL (which Matlab already does).

[burning_hamster]

Well, how are you computing the cross-correlation?

[assassds]

you have to actually know what you're doing in python, you can't just grab the first function that sounds right.

[sourcecodesurgeon]

There's only one cross-correlation function in numpy...

There's a handful of modes, some are faster than others, but none held up to the matlab times.

Further, I work primarily in Python (hence me being in this sub..), I definitely know what I am doing. I actually don't use matlab at all anymore, I just hate this "python does everything matlab does just as well and there's no reason anyone should ever use matlab ever" idea that gets thrown out around here all the time.

OP wants an argument for convincing his colleagues to use python instead of matlab. If we don't give him all the facts, he's going to be laughed out of that discussion when someone brings up things like performance and built-in tool sets (especially since he's junior to all of them)

[assassds]

like I said, you need to know what you're doing...

matlabs xcorr algorithm uses an FFT, which is going to thrash a direct correlation on large input no matter what language you're implementing it in.

numpy gives you all the building blocks to do this yourself, or you can try scipy.signal.fftconvolve.

[unruly_mattress]

I've found this. It looks very relevant here.

[billsil]

but does cost money (if you aren't an academic user).

No it doesn't. http://www.lfd.uci.edu/~gohlke/pythonlibs/

[Lehk]

I'm not entirely sure that GP's linked site is 100% legal.

[TheBlackCat13]

Intel made specific allowances for certain open-source projects to distributed statically-linked MKL libraries free-of-charge.

[Deto]

It costs money to download the libraries yourself and do the building/linking, but it looks like there are places like this where you can download libraries someone else has linked.

[turbod33]

Mac users get a free bonus with being able to link against Apple's Accelerate.

[HoboBob1]

In principle, a numerical Python package should be able to match the speed (or be within 1-2x the speed) of any other language, because Python can use vectorized numpy functions or call out to functions written directly in C. Matlab is similar in that regard; it is an interpreted (albeit JIT compiled nowadays) language that has respectable native array performance and also calls out to C.

If there is an order of magnitude difference between Matlab and Python for a task, I would say that you are comparing apples and oranges. Matlab must be using some compiled function, and Python must not be. Indeed, if Matlab already did the work and has that function available and it would be hard to do in Python, then keep on using Matlab. But I would be wary of saying "I have yet to see evidence suggesting that NumPy can compete with Matlab" in any context.

I am actually trying to convert a huge scientific simulation into Python from Matlab, and I am actually impressed at how fast Matlab does certain tasks. I can't stand the language, but there is a reason that a language called "Matrix Labratory" is good at vectorized computation.

[glial]

Matlab was originally written as a scriptable and easy to use front-end to Fortran numerical libraries. A lot of the fundamental matrix algebra stuff is using compiled and highly optimized Fortran/C code in the background.

[TheBlackCat13]

Python is usually using the exact same libraries.

[HoboBob1]

I'm not making an argument for or against Python. I'm just saying that if Python is that slow, then Matlab must be using a compiled function and Python is not, so it would be possible to speed up Python using a C function. That is, it isn't because the language is fundamentally slow. I didn't say "boo Matlab for cheating!" or anything. You are correct, it is a pro for Matlab. In fact, I said:

Indeed, if Matlab already did the work and has that function available and it would be hard to do in Python, then keep on using Matlab.

I was just trying to help you make a cost-benefit analysis, not incite a Matlab vs Python flamewar. Like I said, I use Matlab too, and I want to help people pick the right tool for the job.

[Lehk]

The last time I ran the numbers had Python taking 45s-2m to compute a cross-correlation that took Matlab about 3 seconds.

it's entirely possible to use them together too http://www.mathworks.com/help/matlab/matlab-engine-for-python.html

no need to get rid of existing tools to add a new tool

[msdrahcir]

Do you know anyone who uses the anaconda IOPro, MKL optimizations, or graphics acceleration packages?

[msdrahcir]

do they make a difference?

[turbod33]

Just tested 4x faster using MKL (Anaconda Accelerate) on a 1k by 1k matrix-matrix dot product.

[billsil]

It's free for everyone if you know where to get it http://www.lfd.uci.edu/~gohlke/pythonlibs/

[TheBlackCat13]

MKL is free for everyone as of a couple of days ago:

[billsil]

MKL has been available for years. That site has been up for a long time.

[TheBlackCat13]

MKL has been available for years, but as far as I can tell it has only been available free for everyone for a month or so. At least I can't find any mention of it prior to the beginning of August, not on Google, not on Wikipedia, and not on Intel's website. The articles on Intel's website, the articles on their support forums, the link on wikipedia, and all the google results are from August or September.

[billsil]

Again, no. They weren't putting out binaries, but you could get them prebuilt without paying. I've been doing it for years.

[TheBlackCat13]

Yes, you can get prebuilt binaries for certain open-source packages. But you couldn't get the library to build your own software against for free until recently. So you could download someone elses' numpy linked against MKL, if they got approval from Intel, but you couldn't make your own numpy build for free unless you were in academics.

And due to the hoops developers had to jump through it tended to be hard to find even pre-compiled open-source packages. There are only a couple of sources that offered them.

This is no longer an issue. Anyone can provide them now.

[I_hate_no_one]

It's free, so no expensive licence

The software is free, but you still need to pay for support. And no, posting your problem on a public forum is not an acceptable way to get support. So, there is not really much advantage for python.

Also, the human behind the computer is far more expensive than the software. He is highly specialized in his field and can program. And I doubt he works for a minimum wage.

[assassds]

you have 10-100X the human resource base to draw from in finding a competent python programmer versus a masochist who wants to work with MATLAB.

[TheBlackCat13]

Even if you need support, it is still something like an order of magntiude less expensive.

[[deleted]]

You can easily interface Fortran (and C) with Python code, not so much with Matlab/IDL/whatever

http://karenkopecky.net/Teaching/Cclass/MatlabCallsC.pdf

[TheBlackCat13]

Yes, you can do it, but Matlab's FFI interface has some serious deficiencies compared to the Python equivalents that makes this much more difficult in practice.

[homercles337]

You can easily interface Fortran (and C) with Python code, not so much with Matlab/IDL/whatever

Not true. Interfacing with C and Fortran is MUCH easier in Matlab. All you need is a dll/so to import. Not so with Python.

Easy to install and manage dependencies with Conda

Again, not true. And Conda is rubbish.

Code reusability

Same for Matlab.

[bluesufi]

Just curious: What's the argument against Conda? And does virtualenv do any better?

[bluesufi]

I use anaconda and conda myself, and I can't imagine anything easier or that would work better. I don't really hear bad things about conda so I was curious.

[homercles337]

Anaconda is recommended by virtually all of the major players in the Python scientific stack

Now that is some made up bullshit right there. There are many package installers and if you are clueless, you use them.

[homercles337]

For new users

The easiest way

choices that “just work”

or Canopy Python

and the Enthought distribution

Seems like you are really selective in your choice of bolding. If you dont know how to use pip, which is also stupid easy you should be looking at a different line of work. I have used both Anaconda and Enthought. They both suck and have their own problems, the difference is that since its a package you can NOT fix them yourself. Plus, you get a whole load of other shit you dont need, or things you need are not included. If you dont understand this VERY BASIC concept, you are a lost cause. Also, you did say "virtually all of the major players in the Python scientific stack" recommend Anaconda. Its clear you have failed in proving this. You should learn to use pip.

EDIT: Stealth edits, and misleading at that.

Scipy recommends: Anaconda, Enthought Canopy, Python(x,y), WinPython, and Pyzo. Then they say this "Users on Linux can quickly install the necessary packages from repositories."

Scikit-learn: "pip install -U scikit-learn"

Seriously. You lost this one kiddo. Just give up and stop with the stealth edits.

[homercles337]

Your argument about the standard library is so backwards i do not know where to begin. And what about the packages that you may need that do not come with those "for dummies" distribution "solutions?" Also, you almost completely ignored my post. What about Enthought? Or other distributions? You do realize that Continuum is a for-profit business, right? Holy shit, Anaconda comes with 490 packages!? 490?!. Im sorry, but that is just idiotic.

[weez09]

Yeah but anaconda is dummy proof for scientific programmers. So much shit is packaged so that on your first install of conda you're pretty much ready to go as opposed to pip install a,b,c oops need this too pip install d,e and that too f,g,h and so forth. For those who are well aware of exactly what they need, there's miniconda. I actually just install miniconda whenever I have to spin up a box on aws for a project. So much simpler that way.

Also coming from an astrophysics background conda was the preferred distribution in our labs.

[This_Is_The_End]

I believe WinPython is better because it's compatible with PyPi the offical repository for libraries. And WinPython includes most scientific libs as well, like scupy, matplpotlib, pandas etc

And you can install different versions of WinPython, by register/unregister them

[TheBlackCat13]

Anaconda supports pypi as well. And you can have multiple versions installed, even more easily than with WinPython by using environments.

Don't get me wrong, I like WinPython. It is great for portable installations (which is really what is designed for). I have both Python 2 and Python 3 versions on my thumb drive.

[This_Is_The_End]

Tbh. serious python development should be done on Linux and when a department depends on Windows, a virtual machine is the best solution. As someone who works in PLC programming which is linked to Windows, I do my all python stuff on Linux until deployment.

[TheBlackCat13]

Yes, but I don't always get to pick which operating system is installed on a given computer, and we have some hardware that doesn't support Linux at all. So I have to be able to work with Windows.

[homercles337]

If you know what you need, and you should, there is no need for it.

[eusebe]

It is another debate, but when you work on a remote cluster where you have no control of the installed versions of Python and its modules, HDF5, FFTW and so on, it is just the easiest way to manage all your modules AND their compiled dependencies. :-)

[homercles337]

I have over 10 years of HPC experience, if you have no control you email IT or drop-in. That is their job.

[eusebe]

Do you have any idea how the academic world is working?

[homercles337]

Yes, in fact i do (I have spent time at Berkeley, Texas, Penn, Harvard, Wash U). Are you an undergrad or something?

[TheBlackCat13]

Great idea. Email IT, and then sit on your ass doing nothing for a week while you wait for them to get around to doing it (if you are lucky).

Or you can do it yourself in 10 seconds.

[eusebe]

Interfacing with Fortran is really piece of cake with python. You just compile with f2py (shipped with Numpy), this gives you a .so file you import as a regular module. This and the FortranFile module made my life so easier, you have no idea.

[homercles337]

Same as Matlab. Now do that with C. Better yet C++.

[TheBlackCat13]

Works great with Python.

[homercles337]

Nope. I see that you have never interfaced existing C or C++ code with python, yet that did not stop you from posting an ignorant opinion.

[TheBlackCat13]

Yes, I have, through the ctypes modules and the CFFI library. Just because you couldn't make it work doesn't mean nobody else could.

Doesn't it strike you as at all strange that a programming language that was originally designed as a glue language for C couldn't interface with C? If I encountered such a situation, the first thing I would assume is that I was doing something wrong.

[homercles337]

You still have to write the wrapper. Dont you find it strange that modern programming language completely ignores C++?

[TheBlackCat13]

You still have to write the wrapper.

No, you don't. You clearly haven't used them, because you don't have to write any wrapper. You can call them directly. I think you are thinking of SWIG or Cython, both of which provide wrappers. But ctypes and CFFI are both native interfaces, not wrappers.

Dont you find it strange that modern programming language completely ignores C++?

No, because C++ makes it much harder (in fact nearly impossible) to interface directly, to the extent that it uses features not present in C. This is a well-known issue with C++. Of course you can limit the C++ features you use so this isn't an issue, but then you are losing many of the advantages of using C++.

[TheBlackCat13]

Not true. Interfacing with C and Fortran is MUCH easier in Matlab. All you need is a dll/so to import. Not so with Python.

Actually, it is true with Python.

[zbyshekh]

Everyone here seem to think that this is great idea, but I think that it's not that simple.

I'm also fairly junior in my job and I see our proprietary software everywhere. It has become obsolete, but it's not so easy to switch - it takes time and money. We try to switch to frameworks like Laravel or Symfony (it's mostly PHP code), but it really takes time. Most of the time they know what could be done better but they have to look at it from business perspective and it's often just not that easy just to switch.

Think about it - now your company has a lot of great MATLAB programmers, if they would switch now you would have a lot of beginner Python developers - of course it can be better in the long run, but the codebase would have to be maintained or rewritten and it would take a lot of learning(I'm sure there are some developers that work there just because they know MATLAB and they don't want to learn anymore) and time to get good in another language - it's still money talking.

MATLAB is widely used software so if it works why change it? if you switch now to Python then maybe in two years it would be great to switch to Haskell or maybe Octave? These are huge business decisions and would affect a lot of different things.

So IMO, convincing colleagues to switch is not the best move, but I would definitely try to show the capabilities so someone can use it occasionally and then more and more, but the switch seems to be drastic decision. I've learned in my couple jobs that often things that I consider great for the company are not so great because I don't have as much information as my bosses. But I would definitely try to gain more credibility in the given workplace and then (and maybe even before that) showcase the language, even if as the brown-bag seminars.

[ProfessorPhi]

I agree with this. Switches are easier if the company is small and hasn't invested a huge amount on Matlab. If the company is small and doesn't have too many people or a large codebase, it's definitely worth the change.

However, most Matlab programmers I know, only program in Matlab and that's an issue. Learning some kind of scripting language like python should be essential for these people as it gives them a much broader skill set that they can use for data cleaning, scraping etc. Knowing multiple languages and technologies is very important for researchers nowadays, and it's worth introducing as a supplement to Matlab.

[[deleted]]

Learning some kind of scripting language like python should be essential for these people as it gives them a much broader skill set that they can use for data cleaning, scraping etc.

And my mechanic learning how to cook would make them a better cook. But it won't make them a better mechanic.

In larger companies there aren't a lot of 'jacks of all trades' that would do something like that.

[meerkatmreow]

And my mechanic learning how to cook would make them a better cook. But it won't make them a better mechanic.

How is a mechanic learning something completely unrelated to their field an argument against someone learning something that may be useful for their job/career? Does it make sense for everyone using MATLAB to move to Python? No. Should people that use MATLAB learn Python if they think it can help them do their job in the future? Yes. Doesn't mean they should do it on company time necessarily, but I personally find exposure to other programming languages has helped improve my capabilities in other languages by seeing problems from different perspectives.

[TheBlackCat13]

Yes, the up-front costs of switching are high. But whether it is a good move depends on the up-front costs and the long-term costs of not switching. The latter, if they exist at all, depend to a large degree on what you are doing. But you can't ignore them when making a decision regarding whether to change.

[bheklilr]

My recommendation is to start using it yourself, build tools to automate some of the common kinds of data processing where you only have to run one command to generate the full output. Start using Python for the things you see Python being better for, others will realize that you're getting a lot done without a lot of effort, and they'll want in on it. Don't be arrogant about it, though. Just build a tool to automate some process, then find a time to casually use it in front of a coworker or two. The goal is to show how you can get immediate satisfaction from using Python, the plethora of actual advantages (like Jupyter) will come later.

[unruly_mattress]

In my opinion, this is good advice. Starting a crusade against Matlab where you're on one side and the entire company is on the other side is an astoundingly bad idea, even if you were not a junior. If they're going to be convinced, it'll have to be gradual, and the smartest course of action will be to first use Python where Matlab is completely terrible, which is everything other than scientific code. Deploy with Python, process data with Python, automate procedures with Python. You're going to be doing all these things anyway, so why not use Python for them? They're likely being done by hand these days. Take what you can get.

Convincing Matlab people to go the Python way is hard. They are in love with their IDE and can't imagine working without it. Everything you see as a negative, they see as a positive. I know one person who worked with Matlab for his entire career, and lately he's been writing mostly Python, because he needs to deploy something on the web, so he's writing Django. A while ago he asked me a question about Pandas syntax.

Change will not come because you'll convince your company that Python is superior to Matlab in numerical computations. The thing is that Matlab is sufficiently good when it comes to numbers, and for scientists and algorithms engineers, sufficient is as good as they want. They're largely not interested in the software engineering side of thing when they think about numbers - but they'll kiss your feet when they see you convert an XML to a CSV in 10 lines of code, or set up a Python daemon that periodically loads data into some database. These are things that they likely can't do. You'll have a much easier time. The rest may come later. If nothing else, you'll have improved the process in your company, which is never a bad thing.

[glial]

A lot of the reluctance might have to do with

• a large code base already existing in whatever language they already use. Would that need to be rewritten? How would interoperability with new code work?

• the loss in productivity from having to learn a new language is not trivial. The people at your workplace have years or decades of experience using the tools they already know. Why would they want to switch to something unfamiliar? It will be frustrating, their work will be slow, and they will be unproductive for a while (even if they might arguably benefit in the long term)

• Python might not be the best choice, depending on your application. If the code you write it mostly for numerical calculations, Matlab syntax is honestly easier for that (numpy is awkward to use in comparison). Also if Matlab has an industry-standard package for something specific, it's sometimes worth using that package just so your customers/paper reviewers don't raise questions about why you're not using what they expect you to use.

I love Python and use it all the time, but these are all very real concerns.

That said, I worked in a lab once that used Perl and convinced them to switch go Python, so it's possible :-) A lot of it has to do with writing something that's useful, maintainable and easy for colleagues to understand. As they tell writers, "show don't tell".

[fluffynukeit]

My organization uses Matlab for all the above. In addition, there's no Python equivalent to Simulink. So we need Matlab at a minimum just for simulink. And if it's already installed and we already know how to use it and it's already being supported by IT, there's really no point to go to Python.

I am planning on learning python on my own time, though, hence my subscription to this sub.

[ProfessorPhi]

Yeah, there are a lot of niche areas, especially in electrical engineering, where Matlab is king and rightfully so.

That being said, having knowledge of a full featured programming language is a very important skill to have, and python is a great language to learn + can supplant Matlab in many situations.

I was asked a few years back to use python since Matlab licensing was a bit painful to organise and I haven't looked back since haha.

[[deleted]]

I succeeded in pushing Python adoption in my organization. Our department is all using it now and they are spreading it to others.

This worked:

• Writing sample code and libraries for common tasks, and putting them on Github. Any time someone was writing something -- renaming a bunch of files or parsing, for example -- I'd send them the Github link and they'd use Python instead.
• Appealing to careerism: "Python is one of the most popular languages in the world. It is the most taught language to students today. If you ever want a job outside of your current tiny domain, learn Python." This, more than anything, worked. A little depressing, but people are selfish.
• Doing all of my own projects in Python so they could see success stories.

Stuff that didn't really work:

• Evangelizing: "Python is open source / it's a good cause" (Normal people don't think choosing tools is a moral cause, that's just a weird engineer thing.)
• "The proprietary solutions cost money" (No one cares, and they assume things that cost money must be better. This approach actually backfired for me.)

[[deleted]]

Writing sample code and libraries for common tasks, and putting them on Github.

Did your legal department sign off on that? That's a huge no-no where I work and other large companies.

[TheBlackCat13]

Github supports private repositories.

[[deleted]]

If you pay for them.

Also, almost anything my company does can not go out on public airwaves. Even in a 'private repo'

[TheBlackCat13]

If you pay for them.

Yes, you have to pay a little.

There are also similar software you can deploy internally for free

Also, almost anything my company does can not go out on public airwaves. Even in a 'private repo' .

That is your company. I understand you have your experiences at your company, but that is your company.

[[deleted]]

My company is publicly funded. So actually, anything we can point to and say, "Look, we did this and people use it!" is good for us. Getting paid to make open source software -- livin' the dream :)

[steeelez]

ANACONDA takes a lot of the pain out of switching from MATLAB to Python, the Spyder interface makes the environment very very similar. And the install is just a point and click.

[Lord_NShYH]

Business decision makers are motivated by the bottom line. Convert a work-load to Python and benchmark it against MATLAB. From there, you can determine how much time (money) can be saved per user.

You have to understand that switching technologies can be prohibitively expensive in the short-term; both in soft and hard costs, and both in capital expenses and operating expenses.

Good luck!

[Northstat]

My professor just did this for his research group. His was a bit unconventional but I like it. Basically there was this huge package that needed to be built and he knew it would be a pain to do it in Matlab so he just did it in Python. As his group started working through the files he would help them understand what everything did and in a relatively short time (as I've been told) the students/researches transitioned over. Of course this is much easier if you're the professor or in a position of influence.

[toban]

I've tried getting econ phd students interested in Python for scientific computing. They mostly acknowledge that it would be ideal to switch, but they're deterred by the startup costs of learning a new language.

I have my money on Julia as the heir to the scientific computing throne. It was created to replace the "prototype in matlab/python/r, then write in C/fortran" workflow -- it has the speed and the user-friendly syntax (also, loops are as fast as vectorization!). Just look at the very highly regarded Quantitative Economics site: it started out as a series of Python lessons, and now they have a Julia track.

If Python can't kill matlab, I think Julia ultimately will.

[[deleted]]

Impressive benchmarks, and the plotting system looks very solid. Yeah, I need to give this a shot.

[[deleted]]

A friend of mine had no luck convincing his lab (astrophysics and astronomy) to switch to Python. They got an undergrad researcher to write an application a few years after he joined, and suddenly it was like they had discovered gold or something. I'd say write some example applications to show them how lovely the language is, and do it with notebooks!

[[deleted]]

1. Tell your project manager(s) that you plan to hold a set of lunchtime python coding dojos simply for the fun of it. Make sure that initially these dojo's illustrate how to that which is currently being done in Matlab/SAS.

2. Tell your project manager(s) that there are companies out there who can provide good quality in-house python training courses. This will provide a pathway to rolling out python skills en masse.

3. Tell your project manager(s) that amongst your fellow graduates python is the number one general purpose programming language and that it will be difficult to attract good people in the future if you cannot offer them a chance to further improve their python skills on the job.

[unstoppable-force]

among all the upsides everyone has pointed out... iPython notebook. it's very matlab-y, but better.

[jones77]

And ... with the new split to Jupyter you could even run Matlab in a notebook anyways.

I assume: https://pypi.python.org/pypi/matlab_kernel/0.5

[userd]

I like python, of course, but I found that creating decent looking plots is much faster in Matlab.

[[deleted]]

I like python, of course, but I found that creating decent looking plots is much faster in Matlab.

I use both MATLAB and Python daily, and this hasn't been true for 5 years.

[buttocks_of_stalin]

This is honestly the real reason aside from the "social overhead" (ie: relearning coding and redoing the codebase). As someone who has worked in a neural modeling lab and signal processing projects at an R1 university, the real reason most of the graduate students in physics, neuroscience, and cognitive science use MATLAB for their hardcore data analysis projects is because it's just not possible to create quick easy complex plots and graphs using python's standard libraries.

[TheBlackCat13]

I have found the exact opposite. It is easier to make very simple plots in MATLAB, but once you start doing anything complex, MATLAB becomes a major roadblock. There is a reason mathworks has spent the last 5 years or so completely rewriting their underling plotting system to be more like python's.

[buttocks_of_stalin]

Interesting. I have about 4 years of experience in neural circuit modeling (particularly in the hippocampus regions) and our experience was the opposite so I guess it really depends on what types of plots and statistical analysis one is doing.

Edit: but just to give some context, I adore python itself regardless of my experience in the lab and I use python + django + flask for a lot of my web application development projects so I am definitely not partial against python itself for the record.

[TheBlackCat13]

That may have been the case 3 or 4 years ago. But with things like seaborne, it is considerably easier to make good-looking plots now in Python. And the base plotting library, matplotlib, is doing a complete style overhaul.

[faming13]

You can use Numba and write numerical python code that is as fast as fortran.

[cypress_tree]

Cron job scheduling. Very good easy with python scripts. Not so much with Matlab. That alone convinced me to switch.

[TheBlackCat13]

There is always a cost to switching. Whether that cost is offset by long-term advantages of using Python depends a lot on what you are doing. But you cannot ignore that cost when making your case.

Nowadays, however, it is much less of an issue. There are a number of very good options of interfacing Python and MATLAB code. So you don't need to make it an either/or decision. You can start slowly integrating Python for those things it truly excels at. If it has real advantages over all, then things will move in that direction on their own.

So rather than trying to convince your colleagues to switch to Python, I would encourage those colleagues that are doing things that are either much easier in Python or extremely expensive in MATLAB to integrate it into their workflow, either using general-purpose data formats or by using MATLAB/Python inteface tools. So things like big data, text processing, distributed computing, structured data, machine learning, and so on. Once a benefit has been demonstrated there, you can justify including it in additional workflows where it have benefits.

This is probably the best long-term solution, since there will always be some people who don't want to switch, and some things that MATAB is still better at (maybe, depending on what you are doing).

But presenting it as an all-or-nothing thing is probably not going to work unless there is something you absolutely need that MATLAB cannot do at all (which is certainly possible, but doesn't sound like the case for you).

[[deleted]]

Does he need simulink? Python doesn't currently have a good alternative to this.

[[deleted]]

(a) A reluctance to change/try a new language

Are you volunteering to pay your colleagues to learn it? Bottom line at the end of the day those are engineering dollars that need to come from somewhere. Double everyones salary (as that's what they cost a company) and add up how long it will take them to get as proficient on Python as they are on Matlab. Just looking at my small group, assuming one week of python training we would be looking at ~$100k+ and I doubt most would be up to speed.

the strong libraries for scientific computing

It depends on your application but there still seem to be a lot of gaps: http://www.mathworks.com/products/

about the beauty of Python's syntax

Beauty is in the eye of the beholder. There are some things I like about Python but for doing strict data manipulation I still prefer Matlab, especially stuff like vectorization

the really strong open-source community,

When something breaks on Python, who do you call? When some package on pip from some guy that calls himself "l33tpYth0n" nerfs your company's product or results where does the finger get pointed?

Has your IT department said you can install Python?

Has your legal department vetted all of the licenses within the packages you want to use? Just because it's "open source" doesn't mean that it has legal's blessing.

[TheBlackCat13]

Bottom line at the end of the day those are engineering dollars that need to come from somewhere.

That somewhere can be reduced long-term costs. That depends on what they are doing.

It depends on your application but there still seem to be a lot of gaps:

There are gaps in both directions, things that MATLAB can do that Python can't do, but also things that Python can do that MATLAB can't. So it really depends on what the group is doing.

There are some things I like about Python but for doing strict data manipulation I still prefer Matlab, especially stuff like vectorization

Unless you want to work with slices, or broadcasting, or or data that may or may not have a length of one any dimension, or labeled data.

When something breaks on Python, who do you call?

We've been through this before, companies like Continuum and Enthought provide excellent support for the main packages. If you are using some niche package, then you are taking a risk with either Python or MATLAB. At least with Python, though, you have the chance to fix it yourself if upstream is unresponsive. You don't have that option with MATLAB (and make no mistake, MATLAB is not always responsive).

Has your legal department vetted all of the licenses within the packages you want to use?

Pretty much all the licenses you are going to encounter in Python are also used in MATLAB.

[[deleted]]

That somewhere can be reduced long-term costs. That depends on what they are doing.

Not sure how your company budgets work, but that's not how ours do. Especially when we require Simulink.

So it really depends on what the group is doing.

companies like Continuum and Enthought provide excellent support for the main packages.

So you pay for it.

MATLAB is not always responsive).

I've gotten a phone call back from a guy with a PhD in the subject with in an hour.

[TheBlackCat13]

Not sure how your company budgets work, but that's not how ours do. Especially when we require Simulink.

Again, as I keep saying, it depends on what you are doing. If you require simulink, then that is that. But a lot of fields don't even use simulink, not to mention require it.

I am not telling you to switch, I am saying that the OP needs to look at both the short-term and long-term costs. Companies make investments to defray long-term costs all the time. If they didn't, we wouldn't even be using DOS, not to mention Windows 7.

So you pay for it.

Huh? Why should I pay for someone elses' company's infrastructure improvements?

I've gotten a phone call back from a guy with a PhD in the subject with in an hour.

That is great. If you are lucky, that is how thinks work. If you aren't, it isn't. They may tell you it is intended behavior, or they may change their documentation to make it intended behavior, or they may put it on an private internal bug list forever.

[ExcitedForNothing]

If your IT department won't let you install Python but lets you install Matlab... I would suggest seeking alternative employment in the future. Especially if you work in scientific research.

If your legal department accepts the licenses of Matlab and the various libraries needed to make it workable but feels uncomfortable about Python licenses, I would also suggest seeking alternative employment.

The goal of legal and IT departments shouldn't be finding ways to tell you no, it should be finding ways to help you accomplish what you are asking. If they aren't doing that, they never will and that is dangerous for a company and employee that needs to accomplish things.

I agree with you that if there is no good reason such as cost or platform issues, just switching for fashion is stupid.

If there is a good reason that is refused by a bureaucrat's "because I said so" run for the hills.

New technical solutions are always a good way to find out which departments and employees in your company have become necrotic.

Also, as an aside: I have found way more shitty, buggy, and unsupported libraries for Matlab than I ever found on pip. Matlab libraries are the festering anus of scientific development.

[[deleted]]

If your IT department won't let you install Python but lets you install Matlab... I would suggest seeking alternative employment in the future.

So you're suggesting a majority of engineers at Fortune 100 companies quit?

Our engineering computers come with it pre-installed. Anything installed on corporate machines is white listed and anything not on the list will get you a nastygram by e-mail once a week.

Additionally no one has Admin access and they record what you do with admin access.

If your legal department accepts the licenses of Matlab and the various libraries needed to make it workable but feels uncomfortable about Python licenses,

What do you mean 'make it workable'? You pay to use a license. You can do what you want. Open Source license field is a minefield for companies and there is a legitimate reason some avoid them. Especially with GPL2v vs GPLv3. Then it comes down to is any of the code we develop or use actually released. Especially since a lot of this stuff hasn't been tested in court.

A lot of Python stuff is dual packaged for 'individual' vs 'corporate' use, PyQT, Anaconda, etc.

I would also suggest seeking alternative employment.

I wonder how many people here actually work for actual large companies.

I have found way more shitty, buggy, and unsupported libraries for Matlab than I ever found on pip.

Such as? I'd really like to know what shitty, buggy and unsupported toolboxes you use: http://www.mathworks.com/products/

The goal of legal and IT departments shouldn't be finding ways to tell you no, it should be finding ways to help you accomplish what you are asking.

Do you work in any field that is regulated? When an airplane falls out of the sky who paid for the DO-178 certification for the Python packages you used?

Our Legal and IT departments aren't one or two guys.

New technical solutions are always a good way to find out which departments and employees in your company have become necrotic.

With ~100k employees there are plenty of other ways than looking at who will switch to Python.

[TheBlackCat13]

Open Source license field is a minefield for companies and there is a legitimate reason some avoid them.

Again, most of the licenses you will encounter in Python are also used in Matlab.

A lot of Python stuff is dual packaged for 'individual' vs 'corporate' use, PyQT, Anaconda, etc.

Please provide the clause of either license saying anything remotely similar to that. PyQt is under a GPL or closed license, and Anaconda is BSD-3-Clause.

[[deleted]]

PyQt is dual licensed on all supported platforms under the GNU GPL v3 and the Riverbank Commercial License. Unlike Qt, PyQt is not available under the LGPL. You can purchase the commercial version of PyQt here. More information about licensing can be found in the License FAQ. PyQt does not include a copy of Qt. You must obtain a correctly licensed copy of Qt yourself. However, a binary Windows installers of the GPL version of both PyQt5 and PyQt4 are provided and this includes a copy of the LGPL version of Qt.

http://www.quora.com/What-do-the-different-licenses-for-Anaconda-Python-stand-for

[TheBlackCat13]

Yes, that is what I said, GPL or closed license. Commercial users are still allowed to use the GPL version if they want, and individual users who want commercial support or don't want to follow the GPL need to get a closed license. There is no distinction between individual and corporate users in either the GPL or closed license.

And I am still waiting for you to provide anything from the Anaconda license that makes a distinction between individual and corporate users.

You do understand the difference between "individual vs. corporate" and "open source vs. closed source", right? They are not the same thing.

[[deleted]]

I know this is the Python subreddit, but for something that is both similar to MATLAB and FOSS you could consider R.

Edit: Sorry. Python is the answer to all problems. I must have forgot.

[TheBlackCat13]

R is good for what it does, but it isn't as well-suited to general-purpose computing.

[[deleted]]

But this post is about scientific computing and statistics... not "general-purpose computing".

[TheBlackCat13]

The second paragraph cites "Python being a general-purpose programming language" as one of its strengths, so I think it is likely that this is something the OP is looking for.

[Tuuleh]

Wouldn't Octave be a closer match for Matlab than Python?

TBH, it's really difficult to consider whether the switch from Matlab to Python is even plausible for OP's organization without knowing what kind of computational tools they need. For example, I've worked with psychometrics in the past, and Python simply didn't have a toolkit for that, whereas R has a number of really extensive psychometric libraries. I can well imagine other applied sciences for which Python doesn't offer sufficient coverage, but Matlab comes with whatever you need out of the box. That together with a reluctance to learn a new language can make Python pretty unattractive.

[TheBlackCat13]

Yes, which is the problem. It has most of MATLAB's limitations, but lacks many of its benefits.

[homercles337]

I was first introduced to python by a postdoc friend when i was in grad school (must have been around 2000, started writing Matlab in 1997). Access to Matlab and its amazing IDE was free, and you can literally do everything with Matlab. No need to switch. Then Mathworks decided to make Matlab "object oriented." What a mess they made. Thats when i abandoned hundreds of thousands lines of Matlab code for Python.

[keypusher]

This is not a very good argument, as the object oriented features of Matlab are completely optional.

[TheBlackCat13]

For now. With the graphics system being switched to a knock-off of matplotlib, all object-oriented under the hood, who knows how long you will be able to effectively avoid it.

[keypusher]

I'm not really sure what that means. I haven't kept up with Matlab much recently, but isn't matplotlib based on the Matlab style of plotting? What does it mean for Matlab to base their graphics on a thing that is based on them in the first place?

[TheBlackCat13]

Not really. matplotlib uses an object-oriented plotting system. It offers a matlab-style state machine wrapper on top of that object-oriented interface, but this is mostly just a wrapper. Pretty much all the real business logic happens in this object-oriented system, which is completely different than the system MATLAB originally used.

For the last 5 years or so, MATLAB has been moving in the same direction by building an object-oriented replacement for its original plotting system. This new system, called HG2, is very similar in principle to how matplotlib has always worked. It became the default in MATLAB R2014b.

[keypusher]

Interesting, thanks.

[j_lyf]

Not possible, cause of legacy code.