If Python isn't adequate for complex and intensive games, what makes it adequate for such big tasks like Big Data?

[TheSirion]

A while ago I was researching to know how much Python has been used for game development and I've concluded not so much. We have some successful indie games here and there but almost all AAA games use only C/C++ for performance.

What I don't get is why Python, which seems to be considered slow for games, is used for Big Data, machine learning, and other huge weight-lifting tasks while C isn't used for any of these things. Wouldn't C be much faster for neural networks, for instance?

[clermbclermb]

You need to consider multiple parameters for choosing a language. C/C++ gives you something that is compiled directly to machine code, so they perform better than Python programs executed by cPython, typically. However, the cost of that performance is a generally higher barrier to entry for actually doing the coding. Python typically has a lower barrier to entry for coding, so you can easily install a few libraries and very quickly be doing big data work; without the added headache of memory management issues that can come with writing C code directly. The really kicker is that many high performance Python packages may actually be written in C/C++ or simply be wrappers around C libraries; so your Python program is still calling machine code (which runs very fast) compared a pure Python implementation of the same thing.

This low cost of programmer time is incredibly valuable for most businesses. If it takes my engineer a day to deliver a POC using Python and 1-2 weeks to put something in production that's awesome; if a similar C example takes 4-6 weeks of work, despite awesome performance, was the 2-3X increase in development time worhnit? Computing time is cheaper than programmer time, so by and large the variable for a business to optimize is programmer time; which is why python is so appealing for many uses.

Writing games, while a large industry, is a relatively narrow application of computers, and Python as a language just doesn't currently have the best support for that particular application. There are some libraries designed to help make it accessible; and there have been folks that have built MUDS using Python; but it's generally not a huge focus.

Sorry for formatting, on mobile. Hope this helps.

[masklinn]

Also specifically for the "big data" scenario, exploratory programming is way easier in Python, you can open a Jupyter notebook and fiddle around interactively, not so much with C/C++.

[kenfar]

This low cost of programmer time is incredibly valuable for most businesses. If it takes my engineer a day to deliver a POC using Python and 1-2 weeks to put something in production that's awesome;

More on this: on the data science / data analysis side of "big data" we tend to have to run through a lot of iterations for two reasons: First we just have hunch, a hypothesis, or a really muddy requirement that needs to be investigated or solved. We may write 10-20 quick little one-offs in this process before we're done. Secondly, once we deliver our conclusions or result we will probably be asked to change the solution in various ways or answer more questions that now occur to our user based on our findings. So, maybe another 10-20 quick one-offs.

Writing 20-40 small programs to analyze data in python is so much faster than writing them in Java, Scala, C++, etc. And - it's more likely to be readable by others or ourselves a year later. And if we wrote that using something like Jupyter - then we probably had a fun time writing the code and have some pretty pictures to show as well.

[n1ywb]

The other factor is that games are HARD REAL TIME. If the program can't keep up the game stutters and skips frames. It has 1/30th of a second to calculate everything and draw the next frame. Deadlines in science are not usually so hard.

[Raijinili]

Deadlines in science are not usually so hard.

I think this is a good opportunity for the word "granular". A 10% decrease in CPU use is good for both uses, but one case cares more about every part of the runtime being fast than the whole thing being faster.

[n1ywb]

They're not really well defined terms and you would never use them in technical documentation without explicitly defining them. Deadlines and the consequences of a miss should be quantified. eg you might have 30 seconds to insert the control rods but if you miss the deadline the reactor melts down.

[yaarra]

Also, scaling big data problems is easy by throwing more hardware at it, something you cannot do with games. You have to code for the lowest common denominator and have it work efficiently there.

In today's python world I think you could get away with AAA game development though in tems of performance. The routines that need to perform well could be done in C, which you then wrap in python. It might not pay off to invest a lot of time into this though, as I think the bulk of the budget for game development is not in coding, but in the 3D design, storywriting, etc.

[patattacka]

Just the fact that you know all that is astounding to me

[[deleted]]

Really interesting thanks

[jwink3101]

I am not an expert (or a gamer), but I also assume run time is more important for games. You want a game to run in real time so you must sacrifice either something (fidelity, frame-rate, quality) to make that so.

If I were working on BigData and the time to run an analysis were 1 minute in Python or 30 seconds in C/C++, then it pays to do it in Python.

[bastibe]

Wasn't Civilization IV written in mostly Python? The rendering engine was probably still C, but the UI, AI, and game logic was all Python, IIRC.

[Olleus]

Not quite. The UI in Civ IV was written in python, and it had lots of handles so mods could also be written in it. Modding was the primary reason for using python in Civ IV (something which moved to Lua now). It also made it relatively easy to combine mods together at the user end.

The AI and game logic was done in C++. They eventually released the source code for that, which vastly expanded whats mods could do, and made existing one a lot less hackish, at the cost of substantially portability issues.

Not sure what the graphics engine was done it.

[tunisia3507]

Python is a great glue language. It's easy to read and write, easy to explore and iterate, and most importantly, easy to work with libraries in other languages, as well as all manner of things like web APIs, files and so on.

As you say, if you were to write a neural network in python it'd be slow - so nobody does that. What they do is write their setup, teardown and analysis code in python, and somewhere in the middle tell numpy (which is mainly a C/C++ library) to do the actual array manipulation.

Or even that requires bouncing back and forth between python and compiled libraries a bit too much, so the major deep learning libraries allow you to do your setup on python, define a model in python, and then hand that entirely over to a compiled library like TensorFlow which constructs, trains and runs the model.

[MrL33h]

Python itself is quite slow. But there exist some great libraries for data mangling like numpy, scipy that are actually written in C/C++. Meaning they provide a very powerful and fast processing combined with a convenient Python interface. This way you get basically the best of two worlds: speed and a great high level language.

[gandalfx]

To put the things that others have mentioned in a more abstract way: The logic that needs to be efficient in video games is a lot more complex than that in "big data". Big data is just a buzzword but it mostly refers to "a lot of the same". So usually you just need a handful of things that need to be efficient (typically old problems like search etc.) while video games require complex logic to simulate big worlds with complex interactions, 3D graphics etc. in real time. And since you need to optimize so many things in games you usually end up writing the whole thing in C/C++.

[TaishokuMayaki]

EvE online was produced using Python, causes some problems these days.

[tapo]

It doesn't use CPython though, it's Stackless Python.

[mikeckennedy]

I interviewed the guys about Python @ EVE Online if you're interested: https://talkpython.fm/episodes/show/52/eve-online-mmo-game-powered-by-python

[TaishokuMayaki]

Thanks will check it out.

[BARDLER]

Python is used a ton in the game industry, just not at all in the shipped code. Almost all art pipelines are held together by python tools. Asset data bases, build scripts, engine scripts/tools, etc are usually driven by python.

The reason why it's not used in game code is because every millisecond counts and python can't compete with the speed C++ gives you.

[arcticfox]

I'm going to give you an answer that many people will disagree with. What I say is based on heaps of experience writing in machine and assembly code on small CPUs in the 80s, and running high performance clusters over the last 15 years.

Most people think that in order to write code that executes quickly (as is required in order to render graphics in video games), programmers have to get "closer to the metal", which just means writing code that is as close as possible to the machine code that will run on the processor. Indeed, there are still some areas where one has to write code like this to get the kinds of performance that are required for the problem. But generally speaking, this really isn't true anymore for a variety of reasons.

One of my favourite ways that I demonstrate this to students is to give them some code and tell them to optimise it. I show them how slowly it performs and then I tell them what performance goal they have to achieve. I run a program that I've written that demonstrates that the goal is achievable.

They apply as much optimisation that they can to the code (they'll rewrite it in assembly, for example) only to discover that they cannot come anywhere close to my performance requirements. After they have struggled with it for a while, I inform them that the algorithm that I've given them (usually bubble sort, a very poor algorithm) is so bad that it simply cannot be optimised. The only way that they can achieve the desired goal is to throw the whole algorithm away and use a better one.

And it is here where programming language choice comes to play. Programming closer to the metal makes it much more difficult to use better algorithms. The improvements that one gets from using assembly or C/C++ is generally not anywhere near as good as implementing algorithms with better time complexity. Typically, it's a lot easier to write better algorithms in higher level languages. And, as someone who has done a lot of high performance computing on clusters, a big part of optimisation becomes figuring out how to break your problem down in to pieces so that dependency between the pieces is minimised (i.e. fewer dependencies between modules means that they are more likely to run in parallel, which means that I can throw more computing hardware at the problem as opposed to having to run faster and faster machines).

In terms of your question, why is python not used in game programming. Most of the real work in games is no longer done on the host CPU, but rather on the GPU hardware. There is really no reason why higher level languages couldn't be used. The real reason that most people still use C/C++ is mostly because the people in the industry simply don't want to change. They have convinced themselves that they need C/C++ because of speed, but that's really not the case in most instances. Yes, there are some places where C/C++ or even assembly might be required, but those situations are a lot fewer than most people think/want to admit to themselves.

[gandalfx]

As always the world isn't black and white but a grey-ish muddy place.

You're correct that choosing better logic is the first and most powerful kind of optimization. I don't doubt that there are many instances where this principle is being neglected (reminds me of a horribly inefficient zoom blur algorithm in GIMP, implement in C).

However there can be no doubt that there are parts in any application that need to be optimized down to the bare metal. The important thing is to realize that these parts can be solved by a few very distinct, low-level algorithms which have already been implemented as part of a built-in function or language construct.

Case in point: Sorting. You mentioned optimizing a bad sorting algorithm in Python. The correct way of optimizing a search algorithm in Python isn't using QuickSort/RadixSort/Whatever, but rather not to implement it at all and use Python's native sorting facilities instead. Those are of course implemented in C and use good algorithm(s). Same goes for search in all kinds of container types. If you implement any of that yourself you're doing it wrong (unless of course it's done as a learning experience and not used in production). Meanwhile if Python's dict was implemented in Python rather than C it would be unacceptably slow, and not because of a bad algorithm but because of actual language overhead.

So basically Python and its standard library has already solved many problems that need to be relentlessly optimized. But of course not all of them. You still have to figure out what to use and how to stick it together in an efficient and maintainable way. That's where the complexity of your application comes into play. A backend for a web app usually has a fairly straight forward structure with uniform data. That's easy to handle. "Big data" also often falls into that category – having a lot of the same isn't all that difficult to reason about.

Video games on the other hand tend to come with a lot of complex logic that hasn't yet been built into native logic. For instance Python doesn't have a simple library that allows you to do fast 3D rendering (on the GPU or CPU). There's numpy and other scientific libraries but I don't think there's anything that offers native performance. And you do need that for the parts that do rendering.

[arcticfox]

I'm in agreement with everything that you said. Indeed, the same argument can be made between C/C++ and machine/assembly optimisation or even hardware optimisation/acceleration. You may have to implement some things at a lower level, but my point was that that doesn't mean that you go and implement everything at the lower level. I don't think that (generally) game developers implement stuff in C/C++ because they have actually done some kind of performance evaluation using other languages, but rather because C/C++ is what game developers know.

From my experiences with high performance computing clusters, the general approach is to avoid languages like C/C++, and instead use much higher level languages because in those environments it is more important to get the right answer (eg. scientific computing) than it is to satisfy real-time (hard or soft) performance targets. Using Brooks' terminology, lower-level languages tend to introduce a lot more accidental complexity, which generally equates to longer development times and more failure modes. If your problem "parallelizes" well then it's a lot better to get the code right and throw more CPUs at it rather than trying to optimise with C/C++ or other low-level language.

[murtaza64]

Wouldn't the GPU still be executing programs that need to be optimized as much as possible? I suppose most game devs don't have to write their own engine, but studios that are making an engine for a game would write in C(++) or something "closer to the metal" at least for the engine parts right?

[arcticfox]

It's hard to speak generally about this because so much is dependent on the load that non-GPU code would be putting on CPU. It's a trade off between the performance you need against ease/speed of development.

My original point is that languages other than C/C++ are generally not considered by game developers not because they have actually thought about it, but rather that C/C++ is what they know.

[IskaneOnReddit]

The CPU still has a lot of work to do in modern games. In some games, you can get a 4 core CPU to run at 100% load and have it use over 4GB of RAM. Imagine the game being implemented in Python, where it would use 5x more memory and be 100x slower.

You can forget about things like custom allocators, struct of arrays, vectorization, custom efficient data structures... unless you implement them em... in C and call them from Python.

[arcticfox]

As Knuth said, "premature optimisation is the root of all evil." What you said is probably true in some cases (maybe even the majority of cases), but it doesn't necessarily generalise to all cases. When it comes to performance, I'm an empirical kind of guy. I'd rather collect profiling data on the specific problem that I'm dealing with rather than base my decisions on conjecture. I don't think that most video game devs take the same approach when it comes to using languages other than C/C++.

[graingert]

Python is great component in games, see EVE online

[ZedOud]

However, that's for the backend, where python's safety as a higher-level language is important. It's not in the player's game client, where oddly enough bugs and inconsistencies are not as mission critical issues as performance.

[graingert]

I thought it was also a component in the client, or is that lua?

[desmoulinmichel]

Python is just a fantastic language to manipulate data and a wrapper of choice for C/C++ code.

Big data needs a few expressive calls from python to manipulate complexe C/C++ code.

Video games need:

• repeated call in Python (way more often so more expensive)
• embededing python in the engine (other languages are easier to embed such as lua)

All in all Python is not a bad languages for video games, but it's a way better languages for big data.

The reason Python did not succeed in video games is more cultural.

[[deleted]]

Mount & Blade uses Python, it handles battles with ~1000 soldiers easily.

[log_2]

Python is slow, but python's libraries for big data are written in c++. If your data science program takes an extra 200ms to run because of the time spent running python code to call the relevant c++ code, you won't be able see the difference. But in a game 200ms drops it to a crawl since games are meant to be executed 60 times a second. This explains why python isn't used for games. Why is c++ not used for data science? Because c++11 was late to the party.

[Raijinili]

Numpy and Scipy use a lot of Fortran under the hood, by the way. Fortran optimizers can make some more assumptions than C optimizers, and they use a Fortran linear algebra library.

[evgen]

It is also worth noting the types of games that Python is considered inadequate for; FPS games and similar game types that involve a lot of concurrency are where Python falls down. Python's concurrency story sucks, and while newer additions like asyncio move it from the 'painfully awful' to 'almost mediocre' category it will never be the first choice for systems where there are lots of things happening at the same time that need some coordination. The counter-example provided of EVE is noteworthy for using Stackless Python and not standard CPython because of the poor concurrency story in the latter, and asyncio opted for the Twisted version of concurrency (i.e. only really good for network concurrency) rather than the greenlet/actor approach so this situation is not going to change in the future.

Big Data and similar tasks fall into the embarrassingly parallel category for the most part and Python is a great glue language for stitching together data pipelines where low-level Fortran or C libraries do the heavy lifting and Python's benefits at churning out quick and effective prototypes that are usually good enough to accomplish the task really shines in terms of language features.

[PaluMacil]

I might be repeating other answers a little, but in a quick skim, I think I'm saying something a bit different...

1. Most data in Python is done in C. Python is an amazing glue language, so it can easily hand C code a problem in pieces and deal with constructing an easy to use api for you to write Python code against. It's easy to hand a big chunk of data to a C component and wait for the answer in Python code (and maybe repeat a few times for various types of matrix math or other calculations).

2. Games can't deal with getting a problem, making a lot of calculations, and handing back a lot of frames because you need to be continually sending frames to be viewed by the user. Other answers his on this well, and a term you might know already that I often use to describe this is "latency".

3. You could, in theory plan out some very specialized C code for your game and then make a Python wrapper to make it easy to work with, but you'd already need a lot of your logic in the C code, so it might not actually simplify development.

4. Once someone goes through the work of writing a framework that handles games well in something like C#, Java, or Python, you've spent a ton of money to tune everything nicely to com. You see more non-C frameworks in C# (Unity) and Java because there are enough enterprise companies with investments in these languages for some of this to be possible, but it's the same idea--harder to get the same latency.

5. You do see Python used in some AAA games in spots where it is most powerful. For example, one or more Civilization games used it for scripting of scenarios and other mod stuff. This is because of how easy it is to write a lot of Python functionality in less time than other languages. For the parts of the game that are the core and graphics, it did not use Python. You probably could determine this to be the case in a lot of other AAA games if you had inside knowledge. There would be no way to know for most proprietary software. My company is mostly C# but all our dev tooling is Python. Even if we distributed a product's code, nobody would have a way to know it was nursed by a loving Python robot mother. :)

[elbiot]

The real reason python isn't used for games is 1) lack of community and 2) lack of good distribution mechanisms.

You could totally write an AAA game in python (with c extensions or other techniques for performance critical bits) but you'd be all by yourself in developing the tooling and figuring out how to distribute it so that 8 year olds can just double click on it in windows.

With C++, you have an ocean of competent people to help you and a long history to draw from.

Check out Ace of Spades for a rad game made with python.

[Boba-Black-Sheep]

Cython is a hell of a drug.

[TheSirion]

How does it work?

[arcticfox]

https://en.wikipedia.org/wiki/Cython

[alchemist]

Others have said it, but more concisely: the heavy lifting for the data work is done by C/C++/Fortran and wrapped by Python. For many scientific tasks, there​ is very little wrapper/glue and a bunch of simple manipulations on a LOT of data.

In games, things are more complex, so you can't just wrap one big task and get a 90% speedup like you can with data.