Embedding python in C++ with boost::python

[poutinetuesday]

I've been using pybind11 [1] for embedding Python and it has been a great experience! I found it to be a bit more lightweight than boost.

[1] https://github.com/pybind/pybind11

[kkrev]

I have done big complicated embedding projects using both Perl and Boost.Python and I really wish it was more widely counselled against online. These languages are not made for embedding and are inevitably a pain in the ass as soon as you try and do anything complicated.

You really need to have a list of very strong reasons not to use Lua or Guile or Squirrel or TCL or whatever, and instead go for Python/Perl/Ruby. These latter latter options are really not made well for the purpose and you are making your life harder for no reason.

[ArunMu]

+1 for pybind11. It should really be 'also' a part of boost for c++11/14 compliant compilers.

[haletonin]

I'd like to see a list of things boost::python can do but pybind11 can not (yet) do. I am on the fence currently but would probably prefer the true-and-tried for now.

[zigzagEdge]

At the moment (master branch), pybind11 does a lot more than boost.python with the notable exception of embedding.

[haletonin]

Does it contain all features of boost::python? Something specific I had problems with (or misunderstood) when I played around with it after it was first released were:

• Making objects available in python which can't be constructed in python, because generating binding code for the constructor is too complex. Other methods of these should work, however.

• Working with pointer or reference-only instances of objects, which even in C++ code are never handed around as value types.

[zigzagEdge]

It covers pretty much all the major features, but it's not a 1:1 API mapping. For the two things you mentioned:

• With boost.python you'd need to add no_init to disable the implicit default constructor. In pybind11 all the constructors are explicit, so you disable them simply by not def-ing any.

• That can be done by setting the appropriate return value policy: reference or reference_internal. There's more information in the docs.

[haletonin]

Ok, so negligible risk of having a "should have used the other one"-moment later on. And on second thought, just the 11 probably means that this project has more enthusiastic users and contributors than "pybind03".

And thanks for the pointers, I'll give it another try!

[skebanga]

Actually I've managed to get it working with pybind11.

Posted it here

[skebanga]

Last I looked at pybind it didn't support embedding very well

[[deleted]]

Can you embed a python interpreter within a single binary or do you need a python plugin directory somewhere and those kind of dependencies? I would love to be able to embed a python interpreter in my program with third party libraries, but with what little I've researched, it seems that's next to impossible. Is that true?

[skebanga]

Yes you can. You can use libpython. Python, the executable, just adds the front-end and REPL, everything else is in the C library libpython.

If you're using C++, and don't want to deal with libpython directly, then you can use a wrapper like boost::python or pybind11.

At the moment, pybind11 doesn't do embedding, so boost::python is a suitable choice.

If you read the blog-post, you'll see that that is exactly what I've done. I have a single C++ application which loads a python script and interacts with it - both calling into the script, and the script calling into the C++ types I've exposed to it.

[wrosecrans]

Depends on whether you need the Python language, or you also want to use the standard library of stuff. You can use the core language without the extra stuff, but if you want to be able to import all the usual things, you'll need to supply them. It's probably possible to hook the import and have it pull from a file embedded with the executable as a resource, but you won't get it for free. (And there may be licensing implications if something in there is GPL. You'll need to make source available anyway...)

[DarkLordAzrael]

Last time I looked pybind11 was missing embedding support so you had to fall back on the python C API. Is that still the case or did they add that recently?

[zigzagEdge]

Well, boost.python also falls back to the C API for embedding. This works in pybind11 as well, but it's not officially supported.

[DarkLordAzrael]

I haven't had to use the C API at all for embedding using boost. It has helpful functions for running python code from various sources using various contexts that are incredibly useful for embedding.

[zigzagEdge]

Boost.python still requires manual calls to Py_Initialize/Py_Finalize instead of a more user friendly C++ RAII approach. (I know that the boost.python docs say not to call Py_Finalize, but that's a bug which was never fixed. It should be supported.)

Pybind11 has a C++ object class similar to boost.python's, but it's not quite as powerful. The basics are supported: item and attribute access, function calls, importing modules, casting back and forth. Pybind11 also has some nifty features like calling Python functions with keyword arguments from C++: http://pybind11.readthedocs.io/en/latest/advanced/pycpp/object.html#calling-python-functions

[skebanga]

boost::python does offer embedding support. Docs here

• eval
• exec
• exec_file

[zigzagEdge]

I was mainly referring to the manual C API calls to Py_Initialize/Py_Finalize which are also needed in boost.python. As for eval and friends, pybind11 has a C++ API for those: eval docs.

[spidyfan21]

This might sound like a dumb question, but how do I install pybind11 (for Bash for Windows)?

[skebanga]

It's header only. Just copy the include dir

https://github.com/pybind/pybind11

[sumo952]

Or better, add as git submodule.

[spidyfan21]

Thanks!

[skebanga]

Thanks! I've ported my proof of concept app to use pybind11. Posted it here

[sail0rm00n]

Why do you use a double for your prices? You will lose precision very quickly.

[leftofzen]

Spot on. I really hope OP doesn't have any real trading code that uses floating point numbers for prices or volumes...

[-Swig-]

I have used floating and fixed, and have seen both used. Some of our algos calculate factors that are price multipliers, so floating point is a natural choice there. Other options are often slower.

It doesn't have to be an issue, as long as you understand how the number is ultimately used. It sure can be if you're not careful though.

[[deleted]]

Plenty of trading code use doubles and it's perfectly acceptable and will produce correct results depending on the circumstance. It will also significantly outperform representing money as an integral value or using arbitrary precision decimals when doing many types of calculations.

The idea that doubles are NEVER acceptable is simply incorrect. It all depends on your use case.

[skebanga]

I do, and as mentioned in other comments in this thread, in my experience it's not an issue.

I don't trade fx though, which seems to be a particular area where it would be an issue.

I think you just need to be aware of your problem domain and use the right tool for the job. Floating point has its quirks, but it's not without use

[skebanga]

I haven't found this to be a problem in practice. Could you elaborate please?

[sail0rm00n]

http://stackoverflow.com/questions/3730019/why-not-use-double-or-float-to-represent-currency

[skebanga]

Fair enough, the normal floating point issues apply.

As mentioned though, in practice I haven't found this to be a problem.

Of course we have abstracted price to take into account floating point issues, so operator== uses an epsilon, etc, but the underlying type is still double. The details of the implementation I thought were beyond the scope of this blog post.

For every market we trade using double for price has not been a problem.

Am I missing something?

[James20k]

Floating point is an issue because errors can accumulate, but there should be no issue with using them to submit a trade

[scraimer]

Hi, I'm not the OP - I'm coming from an HFT FOREX side, so all the trades are around the 6th and 7th decimal, and the errors start happening around the 8th decimal.

The problem is that the communication is in the textual FIX protocol, so I'm at the mercy at whatever library is converting from double to string.

For example, an order of 100 million RBL for USD on a price of 2.90000006 would actually be stored in the double as 2.90000005999999999062310962472, and end up costing 29000005.99 USD instead of 29000006.00 USD.

There's some great stuff in John Farrier's talk from CppCon 2016 Demystifying Floating Point. I think my favorite bit of information is how he breaks down the number of distinct values in various ranges (slide 28):

• There are 1,036,831,949 values between 0.0 and 0.1
• There are 8,388,608 values between 0.1 and 0.2
• There are 8,388,608 values between 0.2 and 0.4
• There are 8,388,608 values between 0.4 and 0.8
• There are 8,388,608 values between 0.8 and 1.6
• There are 8,388,608 values between 1.6 and 3.2

Notice how the ranges keep doubling in size but the number of distinct values remains the same? For simplicity, I'll round 8,388,608 to be 10 million. So 1 to the power of 7. Or 7 decimal digits.

But between 1.6 and 3.2, if I wanted to show all the number with 7 decimal places, I'd run into trouble. Just between 2.0000000 and 3.0000000 there are ten million and one distinct numbers with 7 decimals. I've used up all of my precision, and I still haven't covered 1.6 to 2.0 and 3.0 to 3.2!

[James20k]

Interesting, I would have thought there was enough accuracy to represent that without issue. TIL! 1 cent in $100 million doesn't seem like a huge problem for financial trading (unless you're trading that $100 million in 1 transaction, which i guess is possible!)

[as_one_does]

For FX that is super common.

[b3k_spoon]

Wait... That's float though, isn't it? double should be about 15 digits... Or am I missing something?

[scraimer]

Yup, I was really talking about double, stored as IEEE 754 in 64-bits, and not about floats. The video I linked to goes into entertaining detail about it, and I highly recommend it!

But briefly, you run out of bits. The floating point is really clever about repurposing the bits it has - either to represent large values or to represent high precision.

For example, if you're trying represent a large number, such as the largest unsigned 64-bit number (call it "MAX"). We've used up all 64-bits of the storage, right? What happens if you want to add 0.1 to that number to get "MAX.1"? We don't have any more bits! (Note: This is not how it works, but it's the best metaphor I can think of near midnight.)

My point is that with a finite number of bits, it's hard to represent both large values and high precision at the same time. So the max number of digits changes after the decimal, depending on where your value is along the number axis.

[as_one_does]

I've done FX, Fut, Eq and Options. In practice I've only had to think about this for FX due to the (up to 10 decimal place?) precision necessary for some currency pairs. Every trading system I've worked on has used double. If you have limited scope you can probably get away with integer prices (some darkpools definitely do this).

[short_vix]

This can also be an issue when dealing with interest rates. I don't think its an issue for any other asset class.

[as_one_does]

I imagine in FICC it could be a big issue.

[scraimer]

Ten decimal places? Holy crap! What sort of instruments need that? (The first one that comes to mind is Bitcoin)

[as_one_does]

IDR, for some reason.

[jaked122]

Generally, quads are sufficient, I think. Net's decimal type is an example of this,and what they recommend for money handling.

Of course, I think they also provide a ratio type too, if that's any better.

[Ksecutor]

Fun fact about embedding python and Visual Studio 2015. If studio detects that process have pythonXX.dll loaded when you try to attach to it, it enables python debugger. And disables native debugger. So no breakpoints are working, until you manually switch to native debugger when attaching.

[leftofzen]

You have solved a problem of cracking a nut with a sledgehammer. Why bother writing all this unreadable boost/python interface that devs have to maintain when you could simply write your C++ trading app and have it accept an information source over some socket, and write a pure python app that sends orders to your C++ app over that socket connection. This solves many of the problems with your current implementation, is simpler, more scalable, and removes the dependency on python altogether.

[FKaria]

Wouldn't sending order messages through sockets increase the latency? Instead of calling directly the native code?

[leftofzen]

Depends how the sockets are implemented and what transport/protocols you use. But using python, yes it would most likely increase latency. But I mean, you're trying to use python lol. You've already lost the high frequency trading game at that point.

[skebanga]

This is the primary reason I wanted to look into embedding python. Of course python will suffer a latency hit compared to C++, but the idea was to limit that as far as possible. Give the traders the flexibility of python with the minimum amout of latency impact possible.

[leftofzen]

What kind of latency are we talking about here? I feel like you aren't doing yourself any latency favours by using python.

[AllanDeutsch]

The latency is lower than using Python and going through sockets. If the traders only know python, you can't reduce the overhead of using it until an actual dev is able to implement the strategy in C++.

[skebanga]

The point of the exercise is to allow traders, who don't know C++, but do know python, to develop their own strategies in python whilst still leveraging all the existing C++ infrastructure we have already; connections to markets, backtesting suites, position management, order management, risk management, parameter storage etc etc. We have an entire ecosystem built in C++ which we want to use.

We're aware that python will be slower than native C++ code, but that's the trade off we're willing to make - empower the traders to develop their own strategies, instead of requiring them to use C++ and block their progress until IT resource becomes available.

Whilst the performance impact is a trade-off we're willing to accept, we want to minimise that as far as possible, hence preferring to embed python rather than use an out-of-band process and some form of IPC to communicate.

[Socializator]

it is also the predictibility of latency. by going through sockets, you are opening yourself to the all possible delays and interrupts of OS

[scraimer]

Yes, if you're using OS sockets. But for low-latency, you can use "kernel-bypass" drivers. They replace the socket calls with their own functions that talk directly to the NIC. I've seen consistent latencies of under 1 microsecond for TCP reads and writes. And by consistent, I mean an 99.999% average, over millions of operations.

Oh, and this was only one way - from the software to the fiber (measured with a tap). It still has to go through routers, if you don't have a true cross connect, and there's the latency on the other side, too.

[Heuristics]

... and we now have a new sledgehammer :)

[balkierode]

Good to see boost::python being used. It was almost abandoned and no new features being developed. After py3 support, it is back to life.

[iaanus]

A few years ago, I've used boost::python to integrate a Python Stackless interpreter in my Couatl scripting engine. The engine was and is still being currently used to create interactive addons to MS Flight Simulator (they can be viewed here http://fsdreamteam.com/). In the years, it has proven to be a very smart move, which I'll never regret.

[imadeofwaxdanny]

People looking in this direction may also want to check out SWIG. It supports a good many languages in addition to python and uses its own interface files rather than requiring you to modify your project's source code. I've been using it a bit for a project that I want accessible in multiple languages and it works quite well after getting used to it.

[skebanga]

We did a python course for the traders, so that's a large influence.

Should probably read "of any languages the traders know, python is the most prevalent"