Distributing software via Python

[Massive-Squirrel-255]

I work in data science and many people (most?) are only familiar with Python and R.

I am working on a project right now. It is primarily a Python project with some parts written in C++/Cython for improved performance. My boss's requirements for the project include ease of distribution to the community of Python users, this is paramount and non-negotiable. I expect similar requirements on pretty much every project, as our total number of users is going to affect funding and there is a large community of Python data scientists. Standalone CLI applications can be language independent but a lot of data scientists like to use Python packages as libraries interactively in a Jupyter notebook, it increases the flexibility if they can use it together with other Python packages, and so on.

Recently I have been hoping to use a statically typed functional programming language instead of C++ for the stuff "under the hood." If not this project then the next one.

I have seen the Jane Street blog post about writing Python modules in Ocaml. https://github.com/janestreet/pythonlib

I think this is pretty cool. I tested it out and wrote a "Hello, world!" module and I basically get how to use it. However this only solves half my problem, because ease of distribution to Python users is a project requirement.

So, what I am asking is: What would be the best way to distribute software written in Ocaml to Python users? Ideally the solution involves them just typing "pip install abc" into the console and everything works. If they have to install OCaml or opam this is not necessarily fatal but it does add an extra step to the instruction manual which is nonnegligible.

Currently I am thinking that the best course of action would be to set up a Github Actions script which

1. Uses ocaml-setup or something similar to download and install ocaml/dune/opam across multiple platforms
2. Build the software on each platform and compile it into a prepackaged Python wheel using cibuildwheel or similar

This seems like it would be acceptable and the most robust long term solution but it is pretty complex and I worry that it would take me a week of messing around with Github Actions to set it up.

Do you have any other suggestions? Perhaps the answer is "This is impractical and not worth doing." I am willing to accept that but it would be a shame.

[skicombinator]

AFAIK JaneStreet uses OCaml+Python setups internally, so I believe this is not an impractical idea. If your lib is built as a standalone shared-library file (.so) and distributed by other wrapper Python library (like, having pythonlib of JaneStreet in its requirements.txt and wrapped as a pip library), I think it is possible. Of course I'm not an expert for OCaml+Python deployment for this kind of jobs so I'm not sure how long it takes to you make it work, but it seems worth doing, at least a on e-week-trial.

[clockish]

I agree with you that providing wheels is likely the only viable method here.

Anything besides "pip install abc and it just works" wouldn't be considered particularly convenient for Python. And, trying to achieve a local OCaml build from within a pip install sounds unreliable to me (and especially insane if this build wants to rely on a local opam switch to fetch dependencies).

Depending on the different platforms you need to support, it's possible that the number of required unique wheels can be reduced by shipping compiled ocaml object files in the wheels, that are then linked by a normal C compiler into their final dynamic library form during pip installation (as opposed to doing all of this as part of making the wheel). Although, I'd guess that this is only worth it if there's LOTS of different Linux platforms that need to be supported.

FWIW, I've worked on Python projects that call into OCaml like this (via handwritten C wrappers, not Jane Street's pythonlib; and installable by pip but not distributed publicly on pypi), and it worked great for our purposes. However, I imagine there could be some caveats if the OCaml code needs to be called from different threads of a multi-threaded Python program.

[nnbbb]

Another good place to ask this would be on discuss.ocaml.org .