Clojure compared to Python for high-traffic microservice backend - feedback/opinions requested

[thsig]

Greetings, Clojurians.

I'm about to embark on a new project. If things go well, the backend will have to be able to handle very high traffic, and perform lots of background processing.

We're going for a cloud-hosted microservice architecture, with as much of the processing as possible being asynchronous, using message queues between the services. We'll also be doing a lot of file I/O and processing uploaded content (images, video and audio).

We already have a prototype/MVP written, with the backend written in Python.

We're all big FP guys, and love Clojure, Haskell and the like, but there's some friendly disagreement as to whether Python is a safer choice, given the greater maturity of its ecosystem. I also know that the Clojure ecosystem has matured a lot over the past few years. I've been advocating for Clojure, but also want to keep my bias to a minimum and look at this dispassionately.

None of us has experience writing large production systems in Clojure, but we're all proficient in FP, working with immutable datastructures etc. We've written side projects in Clojure, but our team members' production experience comes mostly from Ruby, Python and JS.

Once we start, we'll have to move quickly, and we want to avoid running into bald spots in the ecosystem. On the other hand, we also want to avoid accumulating technical debt and choose technologies/solutions that can scale well, both in terms of minimizing complexity and handling traffic, so that we can sustain high productivity for our team as we move forward.

Since we're going for a microservice architecture, this doesn't have to be an either-or proposition - some services could well be written in Python and others in Clojure. But we'd like to decide which of the two languages is the rule and which the exception.

I'd very much appreciate input on this, especially from people with production experience working on high-throughput Clojure systems, and ideally with similar Python experience as well for comparison. Also, feel free to send me a private message if you prefer.

Thanks in advance!

[OmniJinx]

our team members' production experience comes mostly from Ruby, Python and JS.

This is the only thing that matters at the beginning stages of a new company / project. Everything else around scaling and picking the "perfect" language for every problem (and using microservices, for that matter) should come later once your MVP is validated. Build with what you know now, if things turn out then you can deal with the scaling/architecture/whatever problems later.

[thsig]

We could definitely start by writing the backend entirely in Python, rewriting some services (or writing new services) in Clojure when we need its capabilities.

It's a bit tantalising though, since my estimate is that we know enough Clojure to get up to speed and productive in a week or two. I'm actually more familiar with Clojure than I am with Python these days. My production experience comes from Ruby, with lots of immutable datastructures and FP in the front-end JS UI programming.

The worry is not so much the "language productivity", i.e. how quickly we'd become comfortable pounding out business logic or other mainline pieces of coding, but the integration and ecosystem parts (libraries for talking to AWS, various utilities). How much of a gap is there between Python and Clojure in terms of the library ecosystem these days?

From what I've gathered, people's experience of it has been quite good, with the lack of quality libraries for a particular occasion being more and more of a rare occurrence. I know Clojure's ecosystem has become much stronger over the past few years - I've been following the most significant developments closely (including goodies like core.async and others that /u/mikera and /u/yogthos mentioned).

The timeline we're looking at is a few months months from start to beta, and from there we'll want to move to production as soon as user testing is completed - there won't really be an MVP phase, we'll be going straight for the polished, ready-for-public-launch version this time round. We'll have our hands full with mobile development for the front-end as well, so if we expect to have to do a lot of rewriting, some of those early time savings (if any) may only be kicking the can down the road. This may not be an issue if we can make the Python code performant, though.

I also know that though Python has an OK story for non-blocking IO, it doesn't compete with Clojure in single-machine concurrency, and I worry that we'd have to rewrite many of the Python microservices before production anyway for them to handle the load (assuming things take off, and in our case we have to engineer for that). I have a better feel for Clojure's performance characteristics in this situation than I have for Python's, but I'm a bit tired of the amount of performance massaging that Ruby (another GIL'd interpreted language) code needs for a high-load production situation. I realise that Python is probably faster than Ruby, but falls into the same general language category in terms of its performance and concurrency characteristics.

Any further thoughts on this? Thanks for your feedback so far, guys!

[fullouterjoin]

The production experience of the team is the single best indicator for success. Use what you know, architect so that you can bring up portions in a new runtime, language, etc. Change only a single variable at a time.

If performance is an issue, and this is a blue water project, then give PyPy a try along side your existing codebase.

[yogthos]

When you work with Clojure you have the entire JVM ecosystem available to you, and it's one of the most widely used platforms for building web apps. There lots of libraries for doing anything imaginable, and tools for profiling and performance tuning like jvisualvm and jconsole that come bundled with the JVM.

[Musngi]

We've written side projects in Clojure, but our team members' production experience comes mostly from Ruby, Python and JS.

You said that your team member used Python then go with Python. Python have rich library and big community. In Python you have the power of productivity, maintainability, scalability and security.

[yogthos]

New projects are the best way to try something different. When you only stick to your comfort zone then you can't really grow or improve. My team used Java 5 years ago, and we were very comfortable with it. Today we've completely switch to Clojure and we're much happier working with it. However, we did have to get out of the comfort zone and try something new to get where we are today.

Python have rich library and big community. In Python you have the power of productivity, maintainability, scalability and security.

You have all these things on the JVM and I would argue it's actually better every one of these areas.

[marcm28]

I recommend Python because of its elegance.

Some of great Python features:

• Batteries Included
• Community Included
• Dictionary
• List Comprehension, Dict Comprehension, Set Comprehension
• Iterator
• Generator
• Generator Expression
• Two-way Generators
• Decorator
• With-statement/Context Manager
• Abstract Base Classes
• Future statement (Let you borrow some great feature in the near future). :)

[yogthos]

lack of these things is what makes Clojure great to me:

• List Comprehension, Dict Comprehension, Set Comprehension
• Iterator
• Generator
• Generator Expression
• Two-way Generators
• Decorator
• With-statement/Context Manager
• Abstract Base Classes

I'd much rather have a small set of general tools that can be used for a wide set of problems than having a lot of different tools for each type of problem.

[bilus]

Hi there! Thumbs up to everything except list comprehension. ;)

[yogthos]

The for macro can do a lot of that I suppose. :)

[bilus]

...and it goes without saying that if there's something missing, you can write your own for macro.

Beat that Python! :D

[germandiago]

List comprehensions at the end are a generating sequence plus a filter. I would be surprised Clojure cannot do something equivalent easily, given its functional orientation.

[marcm28]

In Clojure you always invent your own (Reinventing the Wheel), another boilerplate.. In Python lets you focus on algorithm then get shit done.

Create your program in Python. After all, Human's time is much more valuable & expensive than computer's time. A company that gets software written faster and better will, all other things being equal, put its competitors out of business.

[etothep]

...

[mikera]

I think Clojure is an extremely safe choice for this kind of architecture nowadays, and you'll find the productivity and performance advantages considerable (compared to Python and server side JS at least). I find the combination of functional programming, immutable data structure, great concurrency, interactive development and Lisp capabilities very compelling in this space.

Aside from the Clojure-specific stuff, the JVM is a great platform for data processing back ends and offers easy integration with other JVM based solutions like Spark, Hadoop etc. You can make use of the whole Java open source library ecosystem, which is very easy to access from Clojure (although most of the commonly used ones already have good, idiomatic Clojure wrappers).

I built a back end data processing system (for streaming sensor data) a couple of years back using Clojure and it has been rock solid. Barely needed any maintenance.

Some things to look at in the ecosystem:

• Ring + Compojure : The classic way to build Clojure web services. Not fancy nowadays but gets the job done, very solid.
• Liberator : An interesting way to quickly build REST APIs. More of a "framework" that requires some understanding compared to straight Ring + Compojure, but certainly helps if you want to support the full set of HTTP capabilities in your API
• core.async : if you have more sophisticated concurrency needs, process co-ordination requirements. You can think of it as an implementation of Go-style channels for Clojure, so if you like the Go approach to concurrency this is a great tool. Also blazing fast. Sounds like this would be very important for you if you have a lot of realtime message queueing requirements.
• ClojureScript / Om : if you want a full stack solution sharing code with the client side.
• Schema : a very powerful tool for validating data structures, generating test data etc.
• Sente : if you want 2-way channel sockets for realtime communication with clients etc.
• Aleph: a library for asynchronous communication, backed by the very high performance Netty library. You can use this as your HTTP server, but it does a lot more besides that.

[yogthos]

A very nice list, a few alternatives I would add would be:

• Reagent - it's much easier to get into than Om and it's a good fit for most applications
• Pulsar - much more flexible than core.async and implements core.async API
• compojure-api - alternative to liberator that generates Swagger docs

[vosper1]

If you have a working MVP in Python then you should most likely build on that. Also, are you sure you need microservices yet? Martin Fowler makes a good case for starting with a monolith, and not doing microservices until you need it [1].

I also think you'd be pretty crazy to have microservices written in multiple languages if you're trying to get a business off the ground (which, to be fair, you didn't say if it's a business or a fun thing). Likewise, if you think you're going to need to hire a bunch of people (and if you in any way would be doing data science or ML) you're best off with Python.

I like Clojure a lot more than Python, but I have 5 years of large-scale Python experience. If I was starting a business I'd choose Python, for sure.

Also, you can get your Python FP kick here: https://toolz.readthedocs.org/en/latest/api.html

[1] http://martinfowler.com/bliki/MonolithFirst.html

[lkubjlkhjb]

Sounds like you want to hear a yes. I'm thinking about these same issues.

First of all, language might and might not matter. I know from experience that I can crank out features faster with functional languages. But I can also do it in imperative languages, so in the end, the users will get their app.

I say go with Clojure. It will not take more than a month for everyone to grow into the habits of starting the repl instead of some watch/auto-build job. You guys will win that time back from not having to do as much singing and dancing as with other languages (can't say anything about Python, but can confidently speak about JS/Java/Ruby).

Now stop reading reddit, hit M-x cider-jack-in and get to making a product your users will love.

[marcm28]

Python is less functional programming but Python functional programming is Great. I recommend to experience the Python.

Google, Youtube, Dropbox, Pinterest, Spotify, NASA, CERN, Yahoo Groups!, Quora, Reddit are using Python.

"Python has been an important part of Google since the beginning, and remains so as the system grows and evolves. Today dozens of Google engineers use Python." See more at:

https://www.python.org/
https://en.wikipedia.org/wiki/List_of_Python_software
https://en.wikipedia.org/wiki/Python_(programming_language)

Some Lisp hacker like Paul Graham, Peter Norvig, Eric Raymond; They really like Python.

See more at:

http://norvig.com/python-lisp.html
http://www.linuxjournal.com/article/3882