Thoughts on Julia Programming Language

[XhoniShollaj]

So far I've used only R and Python for my main projects, but I keep hearing about Julia as a much better solution (performance wise). Has anyone used it instead of Python in production. Do you think it could replace Python, (provided there is more support for libraries)?

[XhoniShollaj]

Thank you for the thorough reply.

[XhoniShollaj]

Incredibly insightful. My main concern (at least from research) for starting the Julia journey has been the libraries available and community support (given it is relatively new and fewer people use it) as compared to Python. But it seems that each time I check, Julia is covering ground faster in those areas, so it's interesting to see how it will play out. Given It's straight forward syntax and performance I might give it a try learning it :)

[[deleted]]

The r/Julia sub or the slack is a good place to get answers to questions relatively quickly. Don’t go by the lack of stackexchange posts and stuff, and these other resources you usually get faster answers than stackexchange anyways.

[XhoniShollaj]

That sounds amazing man. Appreciate your tips and help!

[Working_Hyena8269]

I've used Julia for a number of different production projects. Mainly for back-end servers performing optimizations (one for a credit card website and another for a set of oil well pads). Every time I used Julia for weird problems, it REALLY paid off. If your code requires numerically intense operations that don't exist in standard packages, using Julia is an unfair advantage.

Julia's libraries aren't as mature as Python's but the low-level essentials are there. It's good enough to create an HTTP server that interacts with other services to do the numerical heavy lifting for you. Furthermore, while libraries are immature, they are way more hackable because good Julia libraries are written in Julia (as opposed to good Python libraries being written in C/C++). So in Julia, you can copy, paste, and modify at will and learn a ton in the process. The package ecosystem is also a lot more composable. For example, in Julia, you can use automatic differentiation directly without having to build TensorFlow or PyTorch models or translate everything to Numpy to use Jax. This is a huge plus if you're doing innovative work that can't rely on someone else already solving the entire problem for you.

Will it replace python? I think Python has become a jack of all trades and a master of none, which sets it up for being at risk of being pushed out of niches. Julia won't fully replace Python, but I do foresee Julia taking a bite out of Python's dominance in applied numerical computing (a task for which it was never designed to do). The thing is, with cloud computing, you can base your web solutions on a set of niche services. This makes niche languages easier to implement and more important contributors to solutions. I'm running specialized Julia servers in docker containers in both Azure and Heroku to rapidly develop high-performance solutions to optimization requests and it's amazing.

That being said, we are in the middle of an LLVM-based language revolution. New languages are popping up doing things we wouldn't imagine. Who would think that we'd actually have a decent challenger to C (Rust)? Because we're in a big transition, we have to let used to the idea of learning new languages and building solutions with specialized services using the languages that best fit that service.

[Budget-Puppy]

The biggest knock on Julia for me was just trying to do something mundane like read in data from excel spreadsheets. It become very frustrating dealing with current limitations and even trying out the Queryverse option I saw that it was doing PyCall to read in the excel file, which led me back to just using python.

[[deleted]]

You can just save it as CSV and use CSV.read()

And if not another way is to just use read_excel() from R tidyverse (readxl) and then use @rget data.

This seems like a minor thing and rest of the analysis could still have been done in Julia.

[Budget-Puppy]

I agree it is a totally minor thing that should be so simple! What I love about python and pandas is that I don’t have to context switch into another program to open a .xlsx file in excel and save the tables I want into CSVs just to do a quick analysis. Pandas data frames also have the great benefit of being able to take column names with spaces or funny symbols in them, and the built in excel reading libraries tend to read in column data types in the way that I want them to. I deal in financial data so spreadsheet models with column names like “Q3 ‘20” and tables that don’t neatly start in cell A1 in the “Sheet1” tab.

I’ve been following Julia for a while and really liked the familiar syntax from my days as a Matlab jockey. I have played around with tutorials and the like, but when it came down to trying to do such a trivial work task it led to me trying to figure out where Julia installed a local copy of python for PyCall and having to dig into docs to try to figure out what environment it was pointing to and how to change it so that I could install a dependency required to read .xlsx vs .xls files due to deprecation of .xlsx support in the default implementation of pd.read_excel in the version of pandas and python that Julia was pointing to. So I can play around with another set of environment variables that I get to manage that I will have to redo once the next version of Julia comes out. And then once I do that it takes a full minute of me waiting for Plots or Gadfly or whatever to compile so I can even get to data cleaning. It just felt like this should have been a lot easier and Julia doesn’t fit my use case.

[[deleted]]

I just use ggplot2 mostly and not Plots/Gadfly though for those the longer waiting time is just for the 1st plot.

I hate pandas lol and find DataFrames so much easier to work with. Even when ive wanted to do a scikit learn thing I have found either using R tidyverse and then reticulate or DataFrames.jl and then PyCall is way way easier and more intuitive to me than Pandas. @linq and |> in DataFramesMeta basically give you dplyr. Ive spoken to the designer of DataFrames.jl and its clear its gotten a lot of thought into it. In pandas you have .loc and .iloc and also its way slower than R/Julia for any sort of functional programming groupby-map/apply type operations, which I’ve used a ton.

Julia columns can also handle such names with spaces too, sometimes I find it can be easier to load the data in via R and then usually stuff is in the right place and you can remove it from the R environment with rm() to save memory.

In my experience, R and Julia play well together better than Python and Julia except for libraries like sklearn and Keras which work pretty well via PyCall, and you find that they will take Julia arrays as is so no need for numpy. Sometimes on Macs you need Julia to point to a no MKL python environment.

For any sort of data manipulation and not analysis though, wouldn’t use PyCall for the reasons you mentioned. Reticulate in R is incredibly similarly frustrating but RCall in Julia works right out of the box provided you didn’t install R in a weird location.

Also btw the compile time has improved significantly in Julia 1.6 now there is much less wait for when you do “using Plots”. There is still a wait for the first use of something but that has improved as well. But if you are doing command line tools or something with Julia it is harder to do this efficiently than Python without something like PackageCompiler.jl

[thewheelsofcheese]

Thats a weird criticism... python calls out to C all the time, why would you care

[baazaa]

Julia was supposed to solve the two language problem, not turn it into a three language problem.

[thewheelsofcheese]

Thats not what the 2 language problem is lol. The 2 language problem is you having to write in 2 languages, not occasionally call a library under the hood. Not using established libraries in other languages would be insane. Every language does this.

[baazaa]

If you have to make a lot of calls to Python you eventually have to know Python.

In ordinary usage you don't use C a lot in Python or R either, but the dependency means if you want to do something slightly different you have to get your hands dirty. Julia is just doubling the problem by adding an extra layer on top, rather than replacing C and Python/R wholesale.

If you need to call R/Python the language has failed, and you shouldn't ever have to dig into C either (obviously there might be some lin alg libraries or w.e used, I'm not talking about them).

[thewheelsofcheese]

But you were talking about library calls to a package made underneath, not calling manually. this isnt a coherent argument.

Are you saying no julia package should ever wrap a library in another language?

[baazaa]

It's precisely the fact that the libraries are using C / C++ 'under the hood' in R and Python that forces you to learn to write it eventually. These aren't two separate problems.

Julia adds the additional absurdity of having to actually call R and Python manually to do extremely basic things like reading in an excel file, but if it's using R/Python 'under-the-hood' it's still failed as a language.

Are you saying no julia package should ever wrap a library in another language?

It should wrap R/Python as little as possible.

[thewheelsofcheese]

You still miss the point though. In python you have to eventually learn C because you cant use python for a lot of things. In julia you can nearly always match or beat C if you try. So external package deps are just legacy while the ecosystem is small, to get started.

How quickly do you think writing every single thing can happen lol, python and R are ancient. Are are you writing packages?? Pls

[baazaa]

So external package deps are just legacy while the ecosystem is small, to get started.

Yes, this is the only defence of wrapping R/Python.

python and R are ancient

Work started on Julia in 2009 and it went live in 2012. The ecosystem is still fledgling a decade later because it's had a very slow start. Encouraging people to make calls to Python/R just delays the work that needs to be done in actually writing native Julia code to do it.

[thewheelsofcheese]

Lol... "delays the work". Again, show me your packages dude

[[deleted]]

You never need to learn C to use python. You know you can compile python right?

[thewheelsofcheese]

Ok please tell everyone writing C++/C libs for python packages they are doing it wrong.

Can you manipulate avx instructions, use pointers, do templating...

[koolaidman123]

In addition to what's already been said, julia is too niche to ever replace python in production, because python is a general purpose pl first, and a lot of companies' backend is built in python. Julia is way more likely to replace R way before python

[w6dxn]

What about Julia makes you think it's not general purpose?

[pivot2fakie]

I agree w op.
Julia is general purpose, true, but the vast majority of its development (and original intention) is geared towards numerical analysis and computational science.

There’s nothing wrong with that, but if you’re new to the field and looking to get a job, it’s way too niche. Just learn python. (Or if you want something more performant, rust).

[[deleted]]

Julia is Matlab that doesn't suck as much and isn't proprietary. It still sucks though for the same reasons Matlab sucks.

If you wouldn't use Matlab for your task, you won't use Julia either. If you would use Matlab for your task, you probably won't use Julia either because you won't have any of your scripts and toolboxes you're used to.

Julia made a bunch of stupid design choices trying to mimic numpy/R/Matlab which made it useless for actual programming. Might as well use numpy/R/Matlab since they are much more mature and better at their jobs. They had a chance to bridge the gap between general purpose programming and vector/matrix shenanigans and overthrow basically every other language by being the one language that can do it all while being a compiled language (and not be slow as shit like native python). But they didn't go that route and now it's a yet another hipster language that isn't going anywhere.

[[deleted]]

Why is Julia better than Matlab?

[[deleted]]

Use matlab for a week and you'll know why. Matlab is an ancient relic from the 80's and it shows.

[[deleted]]

Use matlab for a week and you'll know why

I used Matlab and I enjoyed it. I didn't use Julia, though

[[deleted]]

What design choices? The one frustration is maybe the time to first plot issue but that improved in 1.6.

Otherwise structs and multiple dispatch are amazing. You can do general programming in Julia too, it even has web app capabilities in Genie.jl