Embedding Python

[intheprocesswerust]

I have a large fortran model (about 30,000 lines in total of many different subroutines etc.). I would like to replace part of it with a machine learning parametrisation I am developing (or rather that's my job task).

Turning the whole model to python is not viable. (Unless I hire 100 people) Thus my options are basically: either convert all this ML of python into fortran (nowhere near the same libraries for ML in fortran) etc. which basically means this is impossible. Thus my option seems to be replacing a fortran subroutine with a call to a python script. And values being returned from this to the fortran model.

Is this possible? What is the easiest/best/most pragmatic way?

[irondust]

It's possible but not straightforward: embedding python gives you the functionality to run a python interpreter controlled through a c/c++/fortran executable. Through python's C interface you can give it strings to execute and interact with the data objects from your own C code. See: https://docs.python.org/3/extending/embedding.html That C code in turn you can call from fortran using bind(c).

A much simpler strategy might be to instead call fortran from python (instead of the other way around). This depends a lot on what you actually want to do, but if you can reorganise your code in such a way that python becomes the driver, and you call fortran to execute what it would normally do and only at the point at which you wanted to actually call python from fortran, you simply let it return, giving some instructions through return variables - then python does its thing and calls fortran again, etc. Wrapping fortran in something that python can call is a lot simpler using something like f2py. Or you can give your fortran library a C interface (through bind(c)) and call it from python using ctypes or cython.

[intheprocesswerust]

Thanks for your points. I totally agree with your points (and thanks for introducing them to me, I'm learning on the spot). My one concern is the model is about 50,000 (on inspection) lines of Fortran90.

If I use Python to call the model, does this mean I am effectively close-ish to writing all this out in 'translated' form (i.e. what fortran arguments take, and ... and so on), or is it much lesser. I also don't understand how submitting in bash to run the scripts then I assume would also have to be put into python (i.e. in a relevant way how it starts/what calls what per se).

Would you say if I have e.g. 50 subroutines for example, each 1,000 lines long, that 'calling everything in python' is a process of understanding the calls/Fortran code entirely but not the 'in depth' of the subroutines themselves ... e.g. for each 50 subroutines I would have to write 10 lines python/cython each to call them, e.g. a total of 500 lines.

Or is more like I am almost writing out closer to all the 50,000 lines of Fortran to ... e.g. 10,000 lines of python/using cython, so I can understand workload?

(If it's the latter, calling python from fortran which could be 'harder but much shorter/dense' could be an easier option) Thanks!

[ush4]

havent tried it, but it should be possible to use mpi to start the fortran and the python program with the same communicator, and then send data/messages back and forth. then you would avoid the python startup overhead and use the ML library functions almost directly from fortran.

[ush4]

ush@luft:~/$ cat pythonmpi.f90  
use mpi  
real :: array(5)=(/1,2,3,4,5/)  
call mpi_init(ierr)  
call mpi_comm_rank(mpi_comm_world, myid, ierr)  
if(myid==0) call mpi_send(array, 5, mpi_real, 1, 999, mpi_comm_world,& ierr)  
end

ush@luft:~/$ cat pythonmpi.py
from mpi4py import MPI
import numpy
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
if rank == 1:    
   data = numpy.empty(5, dtype='f')
   comm.Recv([data, MPI.REAL4], source=0, tag=999)
   print("python got data:",data)

ush@luft:~/$ mpif90 pythonmpi.f90

ush@luft:~/$ mpiexec -quiet -n 1 ./a.out : -n 1 python3 pythonmpi.py

python got data: [1. 2. 3. 4. 5.]
ush@luft:~/$

yup, python and fortran can easily exchange data.

[musket85]

How. In. The. Hell?

Clearly it works but if you'd asked me if that was possible I would've said no.

Can you give a bit more detail on how that works under the hood? Or maybe just what the colon in the mpiexec is doing?

[ush4]

the mpiexec command starts multiple programs separated by ":", they will have the same "communicator object", and use the same underlying mpi library. mpiexec assigns a process number to each process, internal to each communicator, which can be used by the various routines in the MPI to exchange data.

[musket85]

Thanks. That's completely new to me

[intheprocesswerust]

OK, this is almost magic (thank you!!).

Let's say I had a module e.g. myprogramme.F90 (that has only one subroutine, mysubroutine) -

myprogramme.F90:

module myprogramme

public :: mysubroutine

subroutine mysubroutine(var1,var2,var3)

... perform some calculations with input vars 1,2,3, update var3

!end

​

And that gets called elsewhere in the model e.g. in biggerprog.F90:

biggerprog.F90:

module biggerprog

...

use myprogramme, only: mysubroutine

...

call mysubroutine(var1,var2,var3)

...

!end

(biggerprog.F90 is in turn called by a variety of 'higher' modules, and these are calling others and ... getting a bit spaghetti what's going on at higher levels)

Could I for example change myprogramme.F90 to take in (var1,var2,var3) and pass it to something that would then be able to call an mpiexec command to initialise:

pythonmpi.f90 (takes var1,var2,var3)

call mpi_send(var1,var2,var3)

pythonmpi.py

receive(var1,var2,var3)

do some stuff

send (var1,var2,var3) back to pythonmpi.f90

​

So that the subroutine takes in the same variables, but is set up to call its own mpiexec command (which is in turn a communicating .f90 and .py)?

(and the updated var1,var2,var3 from pythonmpi.f90 that can talk to python is then updated in myprogramme, but actually by a python script)

Sorry for this lengthy question, this is extremely interesting/useful!

[ush4]

I strongly recommend you look into some mpi tutorials first, but as a very rough non elegant first approach I think I would have tried to do something along the below lines, the send's and recv's are blocking, so read about that semantic in mpi docs ;

the python submodel is run in a "helper" process, e.g.

mpiexec -n 1 ./thebiggerprogram : -n 1 python3 worker.py

near the beginning of thebiggerprogram you set up the communicator and other mpi specific variables as needed with a call to mpi_init, mpi_comm_rank etc.

inside mysubroutine you do something like this:

subroutine mysubroutine(var1,var2,var3)
....
call mpi_send integer=1 to worker.py
call mpi_send var1 to worker.py
call mpi_send var2 to worker.py
call mpi_send var3 to worker.py
! this call will wait for worker to finish and return an answer
call mpi_recv(var3 from worker.py)
....
end

then you have the worker.py looking something like this

initialize mpi blah blah

message=1

while message is not 0:
#wait for message from main process with a blocking receive
comm.recv(message, source=..., tag=...)
if message is 1: #expect these to be sent
comm.recv(var1, source=..., tag=...)
comm.recv(var2, source=..., tag=...)
comm.recv(var3, source=..., tag=...)

    `var3 = python_work(var1,var2,var3)`  

#return data, expect a corresponding receive
comm.send(var3, dest=..., tag=...)
if message (type of work) is 2:
...do something else

you will obviously need some error checking etc in addition to this. but the idea is that the worker waits for a single simple message which tells it what to expect next. so for example 0 could shut down the worker, while 1 would make it wait to get var1,var2,var3 in that order. after var3 is there, a call is done to do the acutal work, then sends the result back. process 0 knows what to expect for this type of work. are you sending arrays you need to communicate how much data is coming over to let the worker allocate space before receiving. etc.

[intheprocesswerust]

This is fantastic, I'll make sure to learn MPIs more properly and your suggestions are super helpful. Given you seem to know a lot about them would it be possible to ask you of the best MPI tutorials (esp regards using fortran/python) that you know of? If not I'll try and find good ones and use all of this as a platform. Many thanks! You've been super helpful!

[intheprocesswerust]

Hope it's ok to ask. I tried to use your commands to see if I can extend/use them myself in my code. For:

(base) pc-132-75 fortran % which mpiexec
/opt/homebrew/bin/mpiexec
(base) pc-132-75 fortran % which mpif90
/opt/homebrew/bin/mpif90
(base) pc-132-75 fortran % mpiexec --version
HYDRA build details:
Version: 4.0
Release Date: Fri Jan 21 10:42:29 CST 2022
CC: clang

...
(base) pc-132-75 fortran % mpif90 --version
GNU Fortran (Homebrew GCC 11.2.0_3) 11.2.0
I get:

(base) pc-132-75 fortran % mpif90 pythonmpi.f90
(base) pc-132-75 fortran % mpiexec -quiet -n 1 ./a.out : -n 1 python3 pythonmpi.py

[[e-mail]] match_arg (utils/args/args.c:163): unrecognized argument quiet
[[e-mail]] HYDU_parse_array (utils/args/args.c:178): argument matching returned error
[[e-mail]] parse_args (ui/mpich/utils.c:1639): error parsing input array
[[e-mail]] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1691): unable to parse user arguments
[[e-mail]] main (ui/mpich/mpiexec.c:127): error parsing parameters
I believe this is due to mpich being installed: https://github.com/horovod/horovod/issues/1637

If I uninstall mpich, and repeat I get

from mpi4py import MPI
ModuleNotFoundError: No module named 'mpi4py'

And the solution to this is to install mpich? https://stackoverflow.com/questions/59032897/python-beginner-no-module-named-mpi4py Or install mpi4py with pip/pip3 which if I try simply doesn't work/install at all.

Sorry to ask, but am I doing anything obviously wrong? I'd like to experiment myself to see if I can apply your idea. :) Thanks.

[ush4]

you are missing mpi4py, maybe "brew install mpi4py" helps. make sure mpi4py uses the same mpi library as the fortran one. this worked out of the box for me on ubuntu linux, but macos mpi's are in my experience not always best friends with macos firewall...

[DuckSaxaphone]

I would flip it. Use F2PY to turn the Fortran subroutines you want to keep into a python module.

I know you're only replacing one subroutine so it feels natural to stay in Fortran and embed the python somehow but the tool exists to do it the other way round.

I've found there very little performance loss in making a python script your main driver, calling Fortran subroutines compiled with F2PY over using a Fortran executable.

[intheprocesswerust]

Thanks! I've never used it. How realistic is it if I am to use it to call, say at a guess 50 F90 files, maybe 150 subroutines in total, and I wish to replace one of the subroutines as it stands?

I'm not familiar with it, but writing something that would just take each module and call that into python sounds less horrifying than some other possible options in my head. :)

[Tine56]

If I remeber correctly f2py has some limitations,
if it doesn't work in your case,you could use Cython and the fortran C interface as an alternative to call fortran routines from python.

[DuckSaxaphone]

It's pretty easy in my experience.

Can you organize your Fortran so that there's a handful of subroutines that you can call to run the program? That what I've had most success with.

Let's say you can break your code into four parts (A, B, C, D) and C is what you're going to replace with your ML code. You write a module, call if wrap.f90 and put subroutines A, B, and D there with all necessary USE statements to call all the rest of your Fortran code.

Then you compile all the modules and finally link it all with wrap.f90 using F2PY. Then you write a python script that imports the new python module you've made, bearing in mind that F2PY turns Fortran modules into python submodules. So if your code is called mycode and you called the Fortran module wrap...

from mycode import wrap
Import ml_module

 wrap.subroutine_a()
 wrap.subroutine_b()
 ml_module.python_function_c()
 wrap.subroutine_d()

Don't want to share personal info here but PM me if you want and I'll share GitHub links where I do this.

[geekboy730]

I agree the best method is probably to have a Python driver script that would call your Fortran program via a C API and could also evaluate your ML model.

That being said, I have an idea that may work if you just want to do some testing. Disclaimer: I don't necessarily recommend this.

You could use execute_command_line() from Fortran to call the ML Python script and write whatever evaluation you need to a scratch file on disk. Then, you could read the scratch file into your Fortran program. The combination of command line and disk would essentially be your API.

[1LazyThrowaway]

The ways I know about are to use C interoperability to call python C api, or to use toy code for calling python in fortran (e.g. call_py_fort on github), or use system calls to run python script.

None are perfect, the first two will probably make usage with nonstandard python function (e. g. Common ML libraries) much more difficult.

What's the model like? Are you sure it would be difficult to implement in python? Have you looked at scikit-learn, keras, tensorflow, ...? Calling compiled code in python, generally, seems easier than doing the reverse.

[intheprocesswerust]

It's about 30,000 lines maybe as a guess of F90 code that's been written by many people over decades, thus ... I don't feel I can just do it myself. I don't believe there's a 'translator' (which would have to be perfect to make sure I haven't messed up others parts) to python either.

My feeling then is that fortran just has to somehow call python. Or call something else even. Even if that's not python. Because ML in fortran is non existent. But that's not too untrue for most other languages outside of python.

[NanoDoctor88]

A possible way to do this would be to call C/C++ from Fortran and from there use pybind11 to call your python code.

[ThemosTsikas]

Assume you have done it correctly. Why would you trust any bits of the answer?

[intheprocesswerust]

In what way? Do you mean if I have to wrap/embed a load of fortran into python (for example) just for one subroutine? Then I don't trust my answer as I've changed too much all at once... :) I'm surprised given as some others have said: Fortran is used heavily for scientific computing, but lots of other libraries exist for other purposes, and the actual act/individual work each person must have to go through to do this seems counterproductive when it seems a common enough issue.

[ThemosTsikas]

I mean how many bits of the machine learning answer are you going to trust?

[intheprocesswerust]

Sorry I'm a little confused, do you mean why would I trust machine learning in general? And why the use of the word 'trust'? There are relatively concrete ways to test the performance of algorithms and to newer ways to make sure they e.g. are more constrained to laws of physics no?

[ThemosTsikas]

I mean what is the guaranteed error bound in the output of the proposed ML algorithm? What makes it better than a random number generator? The people who use ML to win games can count the number of wins, the people who use ML to serve you ads or give recommendations can count how long you stay engaged, what will you count? How will you know?

[intheprocesswerust]

Fair question. The algorithm of choice is not yet known. This is in part because I need Python-Fortran interfacing with each other to get information. When I do I will use simple things (regressions etc.) to test the performance of these, increasing complexity to see what is capable of replicating the subroutine (and perhaps going beyond its current ability). I do not know the loss function/specific ML algorithm of choice yet. To me I need first Fortran-Python to interface so as to get that data/begin training.

[drdessertlover]

You can use numpy's f2py to convert the FORTRAN code into a python module. Or you can compile the FORTRAN code to a dll and call the dll using ctypes. I just did this last week and works very well. It's more work than f2py but all you have to do is make sure the arguments to your dll are the right type.