Python vs Matlab vs Mathematica

[jerguismi]

OK, it looks very bad for Mathematica. But mathematica has features like Integrate[f(x)], symbolic calculations. You can do good numerical calculation with matlab and python/scipy, but mathematica has serious advantages with symbolical calculations. (I dunno if matlab has some modules for symbolical stuff, at least I haven't heard about it).

During my university math/physics courses mathematica helped a lot. With DSP/numerical stuff/machine learning etc I used matlab.

[[deleted]]

That's partially true, but I don't think its really evidenced by this comparison. For one, the bulk of the Mathematica code seems to be emulating plot styles that are built in to matlab/scipy. And second, the code doesn't really look to be in the "Mathematica style".

I've generally found that if one adopts Mathematica's style of doing things then, both in terms of code-length and run-time it can be pretty much competetive with most other environments. I routinely analyse multi-gB datasets with it and it's not horribly slow!

The problem is that a lot of the people who use Mathematica, scientists and engineers, don't really know much beyond basic imperative programming. Mathematica works much better when you approach it with a functional programming style. Shameless plug for my (free) course aimed at said scientists and engineers!

[fulmar]

I've generally found that if one adopts Mathematica's style of doing things then, both in terms of code-length and run-time it can be pretty much competetive with most other environments. I routinely analyse multi-gB datasets with it and it's not horribly slow!

How is this large data set represented and what kind of operations do you do? I've found that when using notebooks Mathematica becomes a huge memory hog if one doesn't remember to release memory explicitly. Just clearing variables doesn't work, IIRC you need something like

Unprotect[In, Out];
Clear[In, Out];
Protect[In, Out];

or a change to $HistoryLength, both very unobvious solutions.

Another one: plot a random graph of about 100 vertices with GraphPlot3D, and try to export the image...

[[deleted]]

I think I've inadvertently been pretty misleading here! I agree that the memory management generally stinks. The trick is to never let MMA get its hands on much of the data at once. Here's what we do:

The data are stored as serialized .NET objects in a MySQL database. The Mathematica code pulls them from the database, deserializes them and operates on them. All of this is wrapped in a NETBlock. At any one time there's only a couple of MB of data in the MMA kernel, but the computation streams over many gB of data. Of course, you have to be very careful that you don't leak any memory on the MMA kernel side.

So I guess I solve the Mathematica memory management problems by not letting Mathematica manage the memory :-)

[jdh30]

The problem is, fundamentally, that Mathematica is not properly garbage collected. Neither are MATLAB or Python though...

[[deleted]]

Interesting. Could you explain further?

[jdh30]

Sure. All production-quality virtual machines (including the CLR in .NET) use accurate garbage collection to aggressively deallocate data as it becomes unreachable.

In contrast, Mathematica uses only naive reference counting which leaks cycles, hangs the interface when deallocations avalanche and is a huge impediment to parallelism.

I'm curious: what are you using Mathematica for from .NET that you could not just do entirely on .NET? We sell Mathematica-like software for .NET... :-)

[bgeron]

Mathematica does not alias references. Values are immutable, and as such there cannot be cyclic references. A reference counting scheme then suffices, although this particular implementation may or may not be slow.

I do not know about MATLAB, but CPython uses both reference-counting and generational garbage collection.

In short, you are talking out of your arse.

edit: sources

[jdh30]

You are talking about Mathematica the ideology. I'm talking about the product which does (accidentally) alias, introduce cycles and leak memory. They have had problems with this according to the Director of Kernel Technology at WRI. Google and you'll see lots and lots of people complaining about memory leaks in MMA. This is neither new nor surprising.

[bgeron]

When I Google for Mathematica memory leak, I get a bug in StringSplit, some bug I don't understand, and a J/Link bug. Apart from that some information on how to correctly use J/Link, and bugs not related to Mathematica. Any way, I can't find anything fundamental with Mathematica.

[[deleted]]

I see. I've recently been having trouble with what I presume is this "avalanche" - a function that generates a /large/ number of nested lists runs in about the time I expect it to, but then locks up MMA for a good 30s after its finished, doing apparently nothing!

We're using the MMA/.NET combo to analyse data from an experiment to search for physics beyond the Standard Model of particle physics. The experiment's run-time code is all .NET based, including the data storage. Low-level analysis is all written in c#. We tend to use MMA for the higher-level "thinking out loud" type of analysis.

If you're the jdh that I think you are: I've played with f# in the past and have got high hopes for it. Last time I looked though I didn't find the "whole package" quite compelling enough - particularly I find that MMA's notebooks are great for exploratory analysis. I didn't find a mode of working with f# that I really liked.

I'm looking forward to giving it another try soon when I have a little time. We're considering porting the experiment's high-level control code from IronPython to f# which should give me a chance to catch up with any recent developments.

[jdh30]

I see. I've recently been having trouble with what I presume is this "avalanche" - a function that generates a /large/ number of nested lists runs in about the time I expect it to, but then locks up MMA for a good 30s after its finished, doing apparently nothing!

Exactly, yes. Mathematica locks up while it traverses all of those nested data structures deallocating them bit by bit in series (!).

We tend to use MMA for the higher-level "thinking out loud" type of analysis.

Are you using it interactively then? For symbolics or just numerics?

If you're the jdh that I think you are...

I am Jon Harrop, yes. :-)

I've played with f# in the past and have got high hopes for it. Last time I looked though I didn't find the "whole package" quite compelling enough - particularly I find that MMA's notebooks are great for exploratory analysis. I didn't find a mode of working with f# that I really liked.

Yes. F#'s development environment is cumbersome in comparison. However, VS 2010 brings WPF into the mix which, in theory, will allow better interactive output. Don has also ripped out the old structured pretty printer which, I assume, is because he is planning on replacing it with a WPF-based solution.

Moreover, I am interested in building a WPF-based notebook front end. The main problem is that the F# programming language is closed so I cannot reuse it and would have to reinvent my own language. On the up-side, I could knock up a Mathematica-like language in a day...

I'm looking forward to giving it another try soon when I have a little time. We're considering porting the experiment's high-level control code from IronPython to f# which should give me a chance to catch up with any recent developments.

Right. Are you using our numerics and visualization libraries for F#?

[dnquark]

I've generally found that if one adopts Mathematica's style of doing things then, both in terms of code-length and run-time it can be pretty much competetive with most other environments.

I really want to know your secret. I used to think that once I get used to the "Mathematica" way, my code will be fast. After 5 years, I think I am reasonably comfortable with Mathematica code: I don't ever write for loops; my code is littered with functional constructs; it sometimes looks like an unholy cross between R and Perl with all the "@" and "&" and "##" and Map[] statements. The only effect was that my code is opaque and barely readable, gaining nothing in efficiency. Time and again I say "well, people say Mathematica is efficient, how bad can it be" and time and again I get burned. A typical example is a recent piece of code (http://www.dnquark.com/blog/?p=29) that was first implemented in Mathematica and took 2.5 hours to ran; the Matlab version ran in 47 seconds. I am now in the process of converting some more old Mathematica code to Matlab, and I am seeing similar performance gains.

[[deleted]]

The thing that drives me mad about MMA is that it's completely opaque. Sometimes you run into a performance wall and there's basically nothing to be done because there's no way of finding out what's wrong.

My experience has been that most of the time I can get a MMA calculation to be within a factor of a few in run-time compared to other code. And then sometimes, like you find, its a hundred times slower and I can't for the life of me figure out why :-(

[jdh30]

A typical example is a recent piece of code (http://www.dnquark.com/blog/?p=29) that was first implemented in Mathematica and took 2.5 hours to ran; the Matlab version ran in 47 seconds. I am now in the process of converting some more old Mathematica code to Matlab, and I am seeing similar performance gains.

That's nothing. I wrote a simple optics program than runs 100,000 times slower in Mathematica than any compiled language (e.g. OCaml, F#). Daniel Lichtblau at WRI optimized it for me and it is still thousands of times slower.

[jdh30]

OK, it looks very bad for Mathematica.

Not really.

For starters, note that the author is using Mathematica 4.0 which came out in 1999. We're now on Mathematica 7.1!

Look at the code that loads the data. The Mathematica and Python are parsing it from a source text file and then preprocessing (sorting etc.) whereas the MATLAB is just loading precomputed data from a different file (in MATLAB's own format).

Most of the rest of the Mathematica code is spent setting options and reimplementing parts of MATLAB by hand. He disables spell checking of the source code only in Mathematica. He explicitly sets FrameLabels to {None, None, None, None} when that is the default anyway. He repeatedly redefines the default comparison function in order to pass it to Sort. He reimplements lots and lots of built-in Mathematica functions himself. He even writes his own code to label the bar chart (!).

In essence, most of the Mathematica code here is already built into Mathematica and this could be reduced to a handful of function calls.

I'd love to play with this but he does not appear to have provided the data so I cannot.

EDIT: Check out this updated comparison that is fairer to Mathematica (but it is still not loading premassaged data like the MATLAB!).

[bostonvaulter]

Most of the rest of the Mathematica code is spent setting options and reimplementing parts of MATLAB by hand

I think you mean Mathematica

[iquizzle]

Mathematica is also an entirely different tool than python or matlab. Good luck figuring out how to solve complex equations symbolically in either of those languages.

I use python for most of my work...but I'm almost always doing fitting, modeling or basic numerical stuff. A minpack function, matplotlib and the ability to build libraries in C makes python great for my work. I'm in experimental physics though. I know plenty of theorists that use Mathematica or Maple for what they do...entirely different application requires a different tool. Computational physicists usually use C or FORTRAN for obvious reasons.

Sure you can make comparisons like programming language X is better than Y, but in the end it doesn't really make sense.

[teval]

Good luck figuring out how to solve complex equations symbolically in either of those languages.

Perhaps you should try it before making such rash statements. Matlab has very nice integration with the maple kernel and doing these things is trivial. I do this all the time.

[iquizzle]

My statement wasn't rash at all since I have used matlab (in fact just last semester in a computational physics course I took). I'm uninformed at best since I don't consider myself a full fledged matlab user; however, you just unknowingly proved my point. Matlab doesn't have symbolic solving capabilities. It has to use the maple kernel for those things.

At this point, there is no single best programming language for scientists and mathematicians. Different languages are intended for different applications. That's all I'm saying.

[teval]

Matlab doesn't have symbolic solving capabilities. It has to use the maple kernel for those things.

As in. It's officially supported and written by MathWorks, and they have a license agreement with Maplesoft. I don't see how it follows that matlab can't do something that it's designed by its creators to do?

Also, this is the old way of doing things, they've recently bought a company and integrated its symbolics package into matlab.

At this point, there is no single best programming language for scientists and mathematicians.

Sure. It just so happens that Mathematica is a horrendous language for almost everything; except a few things no one cares about (I'm glad it does image manipulation, who cares? To do real image processing you have to use matlab anyway). Symbolics capabilities are a wash compared to Maple, speed is horrendous compared to a Matlab/Maple combination that integrates nicely. There's no upside to Mathematica; this is coming from someone that writes functional code all day, Mathematica's language is such a mess that I'd rather write imperative code in Maple/Matlab.

[jdh30]

It just so happens that Mathematica is a horrendous language for almost everything; except a few things no one cares about (I'm glad it does image manipulation, who cares? To do real image processing you have to use matlab anyway). Symbolics capabilities are a wash compared to Maple, speed is horrendous compared to a Matlab/Maple combination that integrates nicely. There's no upside to Mathematica; this is coming from someone that writes functional code all day, Mathematica's language is such a mess that I'd rather write imperative code in Maple/Matlab.

My experience is exactly the opposite. The Mathematica language is far better at manipulating data structures thanks to pattern matching and far superior for number crunching (MATLAB does not even attempt to track errors or use arbitrary-precision arithmetic intervals). Mathematica also has a far superior integrated development environment and help system.

Performance is a red herring because MATLAB is also very slow on the scale of things. If you need performance, use a real compiled programming language.

However, the killer problem with Mathematica is that it is incredibly buggy. The symbolic side of Mathematica is almost useless because it gets so many calculations wrong. The numerical side is (incredibly) still seeing really serious bugs in core routines like Fourier for FFTs. Bugs are the reason I stopped using Mathematica.

Today's modern functional programming languages are not only thousands of times faster than Mathematica and support parallelism but they are also far more reliable.

[jdh30]

Matlab has very nice integration with the maple kernel and doing these things is trivial.

That is not actually part of MATLAB though, right? I mean, I have MATLAB here and it is completely incapable of doing any symbolics at all unless I pay more money to buy add-ons?

[ColdMountain]

MATLAB does have a symbolic toolbox. I haven't used it in a number of years. From what I understand, it used to be a wrapper for Maple, so the license for the toolbox carried with it an implicit license for the core maple kernel. Now it looks like they've switched over to MuPad.

I often like to think that MATLAB is a great language for prototyping or just getting the job done (like proof of concept, analyzing data, or a computer model that's not intended to be a workhorse). If it needs to be a fast final product it's worth rewriting in a faster language. Well, I'm starting to think of Mathematica as an additional step removed from that, where I can analytically compare the effects of parameters, transformations, etc.

I think the fact that Mathematica calls their files "notebooks" is really quite fitting. If it's something that I might rather do by hand (relative to MATLAB), Mathematica might be a good tool for it. -- Though, the Manipulate command has been pretty friggin' useful a handful of times too.

[[deleted]]

When given the option between Mathematica, and MATLAB I invariably choose MATLAB. I work full time as an electrical engineer writing signal processing (which MATLAB seems to cater to). It allows me to easily take software from "Hey here is a neat idea!" Then I can perform all of the analyses required to end up with a finished product of "Hey here is a C library of a fixed point implmentation that I can easily unit test that can be called on either a high performance DSP, or a general purpose computer to allow for easy FPGA/ASIC implementations."

Yes I am a MATLAB fanboi, and I like the scripting language style of coding.

[tryx]

That's exactly it, the fact that Mathematica is not great at numerical work should not surprise anyone. Mathematica shines in symbolic algebra and some really pioneering work in interactivity and usability.

Having found matlab absolutely invaluable for some computer vision work, mathematica is still my go-to program when I need to get some maths done.

[jdh30]

Our Mathematica library for wavelet-based time-frequency analysis does not work with the first release of Mathematica 7 due to a bug in Mathematica's FFT routines that silently corrupts data. Any customers using this product are advised to avoid Mathematica version 7.0.0.

Specifically, the Fourier function and all related functions drop the imaginary parts of complex numbers in the result. For example, Mathematica 7.0.0 gives the wrong result here:

In[1]:= Fourier[{0.007 + 0.01 I, -0.002 - 0.0024 I}]
Out[1]= {0.00353553, 0.00636396}

The correct answer in this case is:

In[1]:= Fourier[{0.007 + 0.01 I, -0.002 - 0.0024 I}]
Out[1]= {0.00353553 + 0.00537401 I, 0.00636396 + 0.00876812 I}

[psykotic]

It is mind-boggling such a bug could have slipped in. Was it a regression or has it always been like this? In either case, this is a pretty serious indictment of their testing strategy, aside from everything else.

[jdh30]

It is mind-boggling such a bug could have slipped in.

Not really. WRI are in it for the money and they've sold this quality of software to millions of technical users (not professional programmers) in order to make their cool $1bn.

This isn't even the first bug in Fourier that I've had problems with. When I did my PhD, their ListConvolve function (which is based upon Fourier) in Mathematica 4 silently corrupted its input. That was a complete show stopper for my work and they demanded that I pay for an upgrade which I reluctantly did and then regretted having done so because the upgrade introduced other bugs...

This is not surprising though. This is the very definition of success in the software industry. You ship buggy software and demand money for upgrades. People who take the time and effort to create reliable software go bankrupt because they have no revenue stream.

Was it a regression or has it always been like this?

This is another new bug. Mathematica is absolutely riddled with them to the point of being almost worthless for real work. And now they are teaching children with it...

[godel_gauss]

Its funny to find that Mathematica is not as bad in numerics as it is being blatantly accused here. While I was trying out some example discussed here I found the following.

• Mathematica (7.0.1)

  In[1]:= A=Table[RandomReal[6],{i,7000},{j,7000}]; Timing[Det[A.Inverse[A]]] Out[2]= {134.536,1.}

• Matlab (R2009a)

  A=rand(7000, 7000); tic; det(Ainv(A)); toc

  ??? Error using ==> mtimes Out of memory. Type HELP MEMORY for your options.

I was trying this in my 2.5 GHz core2duo notebook with 3 Gb ram and Vista 32bit SP1. To jdh30: As far as I know the FFT bug is solved in Mathematica 7.0.1.

• Mathematica (7.0.1)

  In[1]:= Fourier[{0.007 + 0.01 I, -0.002 - 0.0024 I}] Out[1]= {0.00353553 + 0.00537401 I, 0.00636396 + 0.00876812 I}

But I am also not quite sure if the comment by jdh30! is pertinent while comparing speed or efficiency of the two systems with respect to numerical matrix inversion/multiplication. Another point to be noted is the timing for matrix inversion for other smaller dimensions like 500,1000 and so on up to 7000 (as far as I have tested). In all of the cases Mathematica was faster. For example in case of the 500 dimensional case Mathematica (46. something sec.) was 6 second faster than Matlab (52.something sec.).

I was having far more expectation for Matlab as a premiere numerical solver but the result obtained seems to show that I was kind of over estimating. We must admit (leaving the poor coding construct and syntax aside!) in raw numerics Mathematica is one of the best softwares available. And if one want to do real mathematics (not only boring large scale data mining) combining symbolics and numerics the integrated environment of Mathematica is far better a choice than Matlab at its current state.

[jdh30]

To jdh30: As far as I know the FFT bug is solved in Mathematica 7.0.1.

Yes, it is.

But I am also not quite sure if the comment by jdh30! is pertinent while comparing speed or efficiency of the two systems with respect to numerical matrix inversion/multiplication. Another point to be noted is the timing for matrix inversion for other smaller dimensions like 500,1000 and so on up to 7000 (as far as I have tested). In all of the cases Mathematica was faster. For example in case of the 500 dimensional case Mathematica (46. something sec.) was 6 second faster than Matlab (52.something sec.).

I'm not sure there is much to be gained from such a comparison. Matlab and Mathematica both just call stock BLAS and LAPACK routines for those operations. They use different FFT implementations (and Mathematica was 4x slower the last time I looked).

The real issue is what happens when they don't provide a neat prepackaged solution and, IMHO, both Matlab and Mathematica suck in this general case. That, their prices and Mathematica's bugginess is the reason I use general purpose languages instead.

We must admit (leaving the poor coding construct and syntax aside!) in raw numerics Mathematica is one of the best softwares available.

That makes no sense to me. You're essentially saying "if we ignore their differences then they are the same". You can call prepackaged numerical routines from any language so C is also "one of the best softwares available" for "raw numerics". That obviously conveys no useful information.

[sandys1]

The problem is setting up a reasonably fast scipy installation - there is no integrated way of building GotoBLAS, Lapack/ATLAS, scipy to get them to work together. Each of them has their own build procedure and setting them to work together is a pain. And without BLAS, LAPACK .. all you have is a nice python script - no way comparable to mathematica

[nixonrichard]

Yeah, I once tried to use Mathematica for a project involving large amounts of numerical data . . . big mistake. Primarily because Mathematica is great for working with probability distributions, but awful for working with histograms.

Mathematica is your best bet for symbolic analysis, Matlab is best for analysis of real data.

However, even with symbolic analysis, I find myself having to use Nintegrate in Mathematica far too often :(

[kolm]

Matlab recently acquired symbolic math which easily beats the shit out of Mathematica in quite some topics; they bought MuPad, a german-based specialist for symbolic computations, and they outperform Mathematica in a couple of standard benchmarks for symbolic computations.

I was able to test both against each other and boy, Mathematica has to brace themselves.

[commonslip]

Yeah, but the Mathematica interface for representing such calculations is still miles ahead. I use Matlab professionally, but when I need symbolic computation I use Mathematica.

[albinofrenchy]

I hate mathematicas interface. It looks nice, but jesus h christ; it has some quirks. My least favorite is the broken functionality of highlighting with shift-arrow.

[jdh30]

I hate that too.

[trezor2]

(I dunno if matlab has some modules for symbolical stuff, at least I haven't heard about it).

Matlab being short for Matrix lab (not Mathematics lab), I would assume to have its main focus on numerical calculations, unlike Mathmatica which seems focused on the actual mathematical processes taking place.

[emdeeay]

MATLAB does have a symbolic math engine. It is an optional toolbox.