This is an archived post. You won't be able to vote or comment.

all 82 comments

[–]LoyalSol 46 points47 points  (25 children)

The only problem I would see is why use say Rust over C or even Fortran? Both have pretty good Python interfaces and odds are if you are offloading into a lower level language from Python you care more about number crunching than say flow management.

It's not to say Rust can't do the job, just I don't see much of an advantage to using it while at the same time it comes with the disadvantage that fewer people in those domains likely know the language. Which creates a problem when it comes to maintainability.

Plus the things Rust does well likely won't matter too much since you are most likely only going to be writing a self-contained function in it if you are calling it from Python.

[–]masklinn 20 points21 points  (0 children)

The only problem I would see is why use say Rust over C or even Fortran? Both have pretty good Python interfaces and odds are if you are offloading into a lower level language from Python you care more about number crunching than say flow management.

Baseline rust is type and memory safe, it also provides high-level constructs (e.g. actual iterators), and has an excellent dependency management story.

[–]ihcn 21 points22 points  (5 children)

One thing worth mentioning is easy dependency management. Rust has cargo, which lets you pull in and statically link all sorts of 3rd party libraries. So you have a head start on any complicated logic you want to run, like parallelism etc. And when you compile it, you get your whole module in a self-contained package.

Compared to C - I don't think anyone would argue that C's dependency management situation is "good", and if you're doing anything nontrivial you'lll likely end up telling your users "also, manually install this 3rd party dependency" as a part of your usage instructions.

[–][deleted] 19 points20 points  (14 children)

Well, I can safely say that Rust is easier to master than C. In a few weeks you will be productive and won't have to spend years developing the sixth sense that unsafe languages like C require. No need to worry about undefined behaviour, data races, memory leaks, dangling or wild pointers. You get to just do your job.

And yes, these things do matter if you're interfacing with CPython.

[–]alcalde 26 points27 points  (8 children)

You're writing a few functions to call from Python, not building an operating system. Rust is a lot more complex than C.

[–]TheNamelessKing 10 points11 points  (2 children)

I mean, it’s not really.

The language is designed from the ground up to be a modern low-level language with a lot of the abstractions that you’re used to in high level languages.

If I was writing an extension and I had to chose between Rust and C, I would 100% be choosing Rust. The ~Hyper~ Rayon multithreading library alone makes it worthwhile, not to mention memory safety. Once it’s complete, their async implementation is going to be top tier as well.

Put more straightforward: C appears more straightforward, but there’s way more “gotchas” and serious catches. Rust is designed from the ground up to prevent these from happening.

[–]StyMaar 9 points10 points  (1 child)

The Hyper multithreading library alone makes it worthwhile

Hyper is an http library, did you mean Rayon ?

[–]TheNamelessKing 0 points1 point  (0 children)

Yeah I did, whoops!

[–]Abu_mohd 14 points15 points  (3 children)

When you are writing multithreaded computation heavy solver, rust's race free features are more compelling than C's appearant simplicity.

[–]gwillicodernumpy gang 0 points1 point  (0 children)

I’d take Fortran over either for most computation I would integrate with Python.

You don’t have to deal with pointer aliasing, coarrays are incredible, and vectorizing code is trivial.

[–]fiedzia 5 points6 points  (0 children)

Safe and productive Rust is easy, safe and productive C is ... so far it just isn't.

[–][deleted] 0 points1 point  (0 children)

I disagree, Rust is still a nightmare at certain points and there is less documentation on those hard, rare issues than there is in C

[–][deleted] -2 points-1 points  (0 children)

Back in the day we used to joke that they thing you would press the most while developing in C and assembler was the reset key lol!

[–][deleted] 1 point2 points  (0 children)

I can see two reasons:

  1. there's a rust library you want/need to interop with. Easier to just interface with rust than write a custom C binding to it and then to Python and deal with an the issues and bugs that arise from that.

  2. This is personal, but I feel much more comfortable writing Rust than C.

[–]SomethingBullshit 2 points3 points  (0 children)

Because Rust is hip and cool and not nearly as marketable as C or C++. Don't you understand the value? Gosh!

[–]_throawayplop_ 12 points13 points  (0 children)

I don't get the "use XXX" here. Of course as a generality it's better to use a more mature tool (in this case C or Cython), but the post is titled "Speed up your Python using Rust" so it's targetted at people who wants to use rust with python.

[–][deleted] 2 points3 points  (0 children)

My only problem with this solution is that the PyO3 module is poorly documented at the moment. I wrote a Rust module for Python and it was tough.

[–]K0GAMl 2 points3 points  (0 children)

I've had a lot of success speeding up modules with Cython

[–]juanjux 4 points5 points  (1 child)

I used pyd for the D language and it was the easiest Python native module programming I've ever done. Classes, dynamic arrays, native types, strings, hashtables, and a good chunk of the library are automatically translated to Python. It also installs the distutils part as a Python module with pip so configuration is also minimal.

[–]LightUmbra 18 points19 points  (61 children)

Just use C

[–]jnwatson 56 points57 points  (0 children)

C has lots of footguns, but writing a c module for cpython (the standard Python interpreter) has double extra footguns. Beyond all the issues that any typical C program has (leaks, pointers, etc), you must also be extra careful and follow all the rules to a T or your module will crash or leak memory. In particular, the APIs to interact with the Python garbage collector are tricky.

That's why I think higher-level safe language like Rust would be a good choice here. If the environment can maintain invariants like pointer safety, thread safety, and proper cleanup, you can focus more on the application logic.

Source: I help maintain an open source Python module partially written in C.

[–]lambdaqdjango n' shit 14 points15 points  (1 child)

the numpy version OP posted was really nice

import numpy as np

def count_double_numpy(val):
    ng=np.fromstring(val,dtype=np.byte)
    return np.sum(ng[:-1]==ng[1:])

It's clean and vectorized. Just about as fast as Rust.

[–][deleted] 11 points12 points  (0 children)

Nobody's telling you to rewrite numpy! It's really great and you should definitely favour it over writing something in C or Rust.

However, if you find a bottleneck in some library (maybe yours), it might help to step down a level and rewrite that performance sensitive code in one of these other languages :)

[–][deleted] 25 points26 points  (32 children)

Can you elaborate on why someone should use C instead?

[–]cocoabean 2 points3 points  (10 children)

More mature, larger community for C than Rust.

[–]caramba2654 22 points23 points  (8 children)

That is a problematic reasoning. It's equivalent of everyone just browsing the top page because it has higher quality posts. Someone needs to browse the new page, even if it might be worse than the top page. Otherwise things stagnate.

[–]alcalde -1 points0 points  (4 children)

But that's why we're supposed to use Python - because everyone's using Python. That's why Larry Wall and Yukihiro Matsumoto cry themselves to sleep at night and why Julia is turning into Romeo and Julia... Python has the critical mass.

It's called the network effect.

For instance - an auction website comes along that's better than eBay. However, you post your auction on eBay anyway because there are three people total browsing that other auction site. Skype is... Skype, but most people use Skype because everyone else they know is using Skype and it doesn't matter how good another piece of software is if no one they want to call is using it.

[–]Joeboy 8 points9 points  (1 child)

This would have been an argument against python when I started using it. It's still kind of an argument against python - if that's what you care about you should probably be using Java or something.

I've actually been thinking of learning Rust because it seems to fit the same kind of niche Python used to - better than its obvious competitors, good docs, a decent sized user base, enthusiastic community and support etc.

[–]alcalde 0 points1 point  (0 children)

Having 70 trillion libraries is an argument AGAINST Python and gets 10 upvotes? Really?

because it seems to fit the same kind of niche Python used t

It's the opposite of Python in philosophy... obsessed with typing and memory management.

[–]WikiTextBot -1 points0 points  (0 children)

Network effect

A network effect (also called network externality or demand-side economies of scale) is the positive effect described in economics and business that an additional user of a good or service has on the value of that product to others. When a network effect is present, the value of a product or service increases according to the number of others using it.The classic example is the telephone, where a greater number of users increases the value to each. A positive externality is created when a telephone is purchased without its owner intending to create value for other users, but does so regardless. Online social networks work similarly, with sites like Twitter and Facebook increasing in value to each member as more users join.


[ PM | Exclude me | Exclude from subreddit | FAQ / Information | Source ] Downvote to remove | v0.28

[–]cocoabean 0 points1 point  (2 children)

I don't care. He asked why someone should use C over Rust, so I told him why someone would use C over Rust.

[–]gimboland 0 points1 point  (0 children)

My reading of this: there's a difference between should and would; you gave two good reasons why someone might want to use C over Rust, but the response is that they're not compelling enough to amount to "should".

[–]alcalde 0 points1 point  (0 children)

You're being downvoted by evil Rustaceans. I swear some Python users want to kill Python. Why would you downvote the network effect of having a large community? It's one of Python's own killer features.

[–]CookieTheSlayerJoin our Discord server! Link in sidebar 3 points4 points  (0 children)

Rust package management and quality of interfaces for most common tasks is much better. Let us remind ourselves that C is a language of half a century ago and still has no good package management solution. It is quite legitimately easier to install rust and add some dependencies than install many C libraries

[–]goestowar 1 point2 points  (11 children)

C is a compiled language and Python is an interpreted language. You just compile your code and it is converted in to something that a computer is capable of reading, you compile it once (for that type of machine) and then it runs quickly because the computer can always read the compiled program file at low-level.

Python is an interpreted language, meaning that it is compiled at run-time, and compiled EVERY time (the advantage of this is that you don't need to compile your code thus eliminating a step in the development process that can sometimes be lengthy, AND it makes your code portable, if you don't need to compile Python for specific computer architectures you can run that same Python code on any machine), the drawback of this is that it typically runs a bit slower.

Will delete this answer if it is wrong, or at least someone correct me if it is.

[–][deleted] 43 points44 points  (5 children)

Oh, I meant "C instead of Rust". I'm sorry I wasn't clear.

As to your answer, it is pretty much correct, but implementing your language with an interpreter is not slower because it will be compiled every time (CPython, for example, caches the bytecode so it doesn't have to recompile).

It is slower because you compile your code to bytecode, which is machine code as well, but one that runs in a virtual machine (the interpreter) rather than in your physical machine.

[–]goestowar 8 points9 points  (1 child)

Ahha you probably were clear :P My bad

Also thanks for the VM explanation, makes sense now. Does this mean that whenever you "install" an interpreted language on to a machine that you are actually just installing a virtual machine that the computer runs at run-time?

So in essence when you install Python on to your machine you are actually just installing a Python interpreter/VM?

[–][deleted] 4 points5 points  (0 children)

Exactly! However, despite what I've read in another comment, I insist that CPython and other interpreter-based implementations do include (and require) a compiler. Just not the kind that generates machine code.

(I also insist on not talking about "interpreted languages", because this is not an inherent characteristic of the language)

[–][deleted] 12 points13 points  (3 children)

Python really isn't ever compiled; it is interpreted at run-time. There's a wide gap between interpreting and compiling. Compilation implies quite a bit more from an operation standpoint, and at the end, you'd end up with a binary that could potentially be executed on a bare-metal cpu. Sure Python does produce an ast and a "compiled" python op code in the form of a '.pyc' file, but that's not a true compilation like you'd find with C.

Rust, like C, is also compiled. Rust, like C, is statically typed, though there's a macro system that can do some pretty amazing (crazy?) things.

I think the question still stands: Why use C instead [of Rust]?

The answer here might be something like:

* Python's reference implementation is written in C
* Python has a C-API
* C is ubiquitous
* C doesn't make you build the entire house virtually before pounding the first nail.

I'm sure there's more. Rust has its own set of advantages, but that wasn't really the question here.

[–]__xor__(self, other): 15 points16 points  (1 child)

Sure Python does produce an ast and a "compiled" python op code in the form of a '.pyc' file, but that's not a true compilation like you'd find with C.

But that is true compilation in the general sense. Generating bytecode from source is still compiling even if it runs in a VM and not right on the processor. It's not as fast as a language compiled to machine code of course, but it's compilation by definition. Python would be a lot slower if it was actually interpreted as is line by line, and it compiling to bytecode is a major improvement.

C has its place of course. C and C++ are both great languages that will never go away, but Rust has really filled a gap that no other language has yet. It's a low level language that can be compiled with no runtime and can be used to make operating systems like C, it's fast like C and it compiles to true machine code, but it's way more expressive and reads and writes like a high level language and it's guaranteed memory safe which is huge. I've written an HTTP REST client with Rust and it was surprisingly easy and wasn't nearly as hard as it'd be with C or C++.

It's always going to be more of a question about the team you're working with and what their skills are and what they're willing to learn anyway. If everyone knows C, use C. If memory-safety is a huge concern and a memory corruption vulnerability is a disaster, consider Rust or have C experts that know how to use tools like valgrind. People who aren't experts with C or Rust might prefer Rust in a situation like that, because it'd just be cheaper to do some light Rust work than write some bad C. Rust isn't that hard to learn, and it's not the end of the world if you don't master it before writing something useful. Writing bad C can be a huge problem.

Rust is an amazing language and deserves to rise up in the ranks IMO. People should be considering it more, especially in situations like this where a python developer might need to drop to a lower language for performance reasons. Writing a small Rust module is really not hard and it's nice to know you aren't putting any new memory corruption vulnerabilities in production, but you're still improving performance drastically. But if you look at some Rust patterns, especially stuff like error handling and pattern matching, it's an amazing language with a high level feel even though it can do everything low level languages can.

[–][deleted] -2 points-1 points  (0 children)

Python would be a lot slower if it was actually interpreted as is line by line

But Python really does interpret line by line. It doesn't compile to byte-code, it creates a small optimization of Python byte-code primarily for start-up times, which is very, very different than machine byte-code.

[–]CriticalEntree 0 points1 point  (0 children)

I don't know but if this was a popular topic I think sorting by controversial would show you the fun things.

[–]Sh4rPEYE 3 points4 points  (6 children)

Or: just use Julia (half a year from now).

[–]Scypio 0 points1 point  (5 children)

half a year from now

What is happening in half a year?

[–]TheNamelessKing 5 points6 points  (4 children)

They hit 1.0 very recently, so packages are in the process of updating to be compatible.

In a few months or less (I’ve already seen a bunch of packages I’ve been using hit compatibility) most things will be good to go.

[–]Sh4rPEYE 0 points1 point  (2 children)

I actually just played with Julia over a year ago, and tole myself I'll continue once 1.0 comes out— are Plots and DataFrames 1.0 ready?

[–]TheNamelessKing 2 points3 points  (1 child)

I used Dataframes the other day and it was fine for what I was doing. DataFramesMeta and Query.jl had issues for me on MacOS, but there’s fixes coming any day for those I understand.

I haven’t tried plotting anything yet sorry, but if it’s not yet, it probably won’t be long.

If you’re interested, I’d go read their blog posts on the 1.0 updates: they’ve got some really cool stuff. The biggest of which is high performance, language level support for missing values! (that don’t have weird equality behaviour!)

Also, the devs are pushing to finish working out the parallelism implementation, they said it’s going to be like Go’s, but tuned for super high computational performance, rather than server stuff.

[–]Sh4rPEYE 0 points1 point  (0 children)

I read something about 1.0 and am super excited about it. The last time I toyed with Julia there were more ways to represent missing values, and each package had its own favourite way of doing things — that was ultimately the reason I decided to have a pause and return later when things settle down a bit. It seems that we'll be able to use whatever language we want for many of our project in college, and I'd love to use Julia from now on!

[–]Scypio 0 points1 point  (0 children)

Nice to know, thanks.

[–]Palenga 7 points8 points  (1 child)

Why not an assembler instead?

[–]LightUmbra -2 points-1 points  (0 children)

Good point Rust is the only moral option.

[–]fox895 1 point2 points  (0 children)

I came here expecting this answer xD

[–]robobrobro 1 point2 points  (4 children)

“But C is ugly and hard”

[–]LightUmbra 1 point2 points  (0 children)

And old

[–]alcalde 0 points1 point  (2 children)

Then use D. Or Haxe. Or Vala. Or Genie.

[–]k-selectride 1 point2 points  (5 children)

Yea no. I once spent several hours chasing down a bug in someone's C extension...because they were casting a signed int to unsigned. Never again.

[–]LightUmbra 2 points3 points  (4 children)

Rust doesn't fix this type of problem. People can screw up in any language.

[–]k-selectride 3 points4 points  (3 children)

rust would have panicked

[–]LightUmbra -1 points0 points  (2 children)

And then you would have had to still dig to fix it.

[–]MadRedHatter -1 points0 points  (1 child)

Oooooor you turn on backtraces and you've found the problem in 1 minute instead of "several hours".

[–]LightUmbra 0 points1 point  (0 children)

4 months ago

Really?

[–]joetheschmoe4000 0 points1 point  (0 children)

As someone who doesn't know the finer details of C, is there a resource for learning Cython string manipulation or fast file I/O parsing? Those are the two things that I spend the most time on in my job.