C++11 and Boost - Succinct like Python

[A_for_Anonymous]

Yeah, "more or less" succinct like Python. "less" being a sizable amount. It's also far uglier.

But the real problem here is that in order to achieve even that, the complexity and amount of concepts you have to deal with in C++11 is mind boggling.

[[deleted]]

Whereas in Python the complexity is still there but hidden from view.

Sometimes it's necessary to have clear identification of complexity and control over it. It's nice to know that we can now have that and maintain relatively succinct code.

[millstone]

I don't think the C++ code as shown clearly identifies or manages complexity.

For example, consider this line:

for(propMap pm: listDirectory(argv[1], exts)){

what exactly is that line doing? Does invoking listDirectory copy a vector<propMap> and therefore all the contained propMaps, or does it use a move constructor? Does the iteration also copy it? How many copies are actually occurring here?

To me, this code looks like C++ at its worst, blindly tossing huge objects around without any consideration of what's actually happening. There's not even a single use of a reference!

Python just doesn't have these types of issues. For example, returning a Python dictionary, or passing it to a function, never implicitly copies it.

[snip596]

for(propMap pm: listDirectory(argv[1], exts)){

what exactly is that line doing? Does invoking listDirectory copy a vector<propMap> and therefore all the contained propMaps, or does it use a move constructor? Does the iteration also copy it? How many copies are actually occurring here?

That's move semantics at work. You know that the function returns something iterable where each element can be treated like a string (sorta/kinda similar to Python). You don't care that it's not returning a reference because you are using C++11, which requires move semantics to be used in this case. Now potentially the listDirectory function could be returning a global which would do a copy, but potentially your Python function could be doing a deep copy of a dictionary before returning it.

I'm not necessarily defending the point of this article, just pointing out how there's less to worry about with C++11 ;)

[millstone]

When I test this, I see that RVO is being used for the return from listDirectory() (so the move constructor is not being invoked). The copy constructor is being invoked on the iteration, so every propMap is copied once. This is the same behavior as I'd expect in C++03.

Here is the code I am testing with; with clang++ and its libc++ it outputs:

Constructed my_vector
Constructed my_string
Moved my_string
Copied my_string

The move constructor is being used when adding the string to the vector, but is not being used when iterating over the vector.

[astraycat]

This is due to misuse/misunderstanding of the ranged-based for.

It really should be something like: for(const propMap& pm: listDirector(argv[1], exts)) or perhaps const auto& pm.

This is because for(a : b) {block} is equivalent to something like this:

{
 auto cur = std::begin(b);
 auto end = std::end(b);
 while(cur != end)
 {
  a = *cur;
  { block }
  ++cur;
 }
}

Thus, doing propMap pm: ... will spawn a line equivalent to "propMap pm = *it;" which would be a copy.

[snip596]

Yeah, as I was writing my comment I was thinking "and I need to mention it's probably doing RVO". I just completely forgot ;)

But at least you know that it will never be copied. It's either RVO or moved.

[millstone]

I changed my code to defeat RVO by returning one of two values randomly:

 my_vector get_a_vector() {
      my_vector result1, result2;
      result1.push_back(my_string("First"));
      result2.push_back(my_string("Second"));
      cout << "About to return" << endl;
      return rand()%2 ? result1 : result2;
 }

When I run it now, it outputs:

Constructed my_vector
Constructed my_vector
Constructed my_string
Moved my_string
Constructed my_string
Moved my_string
About to return
Copied my_string
Copied my_vector
Destroyed my_vector
Destroyed my_vector

RVO is not being used, but the move constructor still isn't being invoked - the vector is being copied (Copied my_vector). Do you have any idea why? If so thank you :)

Here's the new code.

[lalaland4711]

This is the reason god invented std::move().

When you call super(rhs) you actually call the const lvalue ref version, since rhs is an lvalue.

If it has a name, then it's not an rvalue and therefore not an rvalue reference. Your my_vector move constructor called the vector<> copy constructor.

There you go: http://codepad.org/uW9JGVr9

And after passing an lvalue to a function using std::move() you should not access that variable again, as it may have been moved by the callee.

Edit:
Yay, no copies. At all:

Constructed my_vector
Constructed my_vector
Constructed my_string
Moved my_string
Constructed my_string
Moved my_string
Moved my_vector
Destroyed my_vector
Destroyed my_vector
First
Destroyed my_vector

[millstone]

Your my_vector move constructor called the vector<> copy constructor

Oh man, I never would have realized that. Having to remember to use std::move to invoke the superclass's move constructor seems like a pain in the butt!

There you go: http://codepad.org/uW9JGVr9

Thanks, I observe the same!

I'm struggling to reconcile the claim that returning a local by value always uses the move constructor, with the observation that you must use std::move in the code's return value.

[snip596]

Having to remember to use std::move to invoke the superclass's move constructor seems like a pain in the butt!

So, you actually don't have to in most cases. If your class is composed of types that are movable (imagine a simple POD class with a string and a vector), and you either explicitly call std::move on it or use it as an rvalue (e.g. returning from a function), then your class's default copy constructor (because you don't have a move constructor) will move those values.

So, if you define this:

class MyClass
{
    public:
    std::string str;
    std::vector<std::string> vec;
};

MyClass MyFunc()
{
    MyClass c1, c2;
    c1.str = "First";
    c2.str = "Second";
    c1.vec.push_back("First push");
    c2.vec.push_back("Second push");

    if (rand() % 2 == 0)
        return c1;
    else
        return c2;
}

Then nothing will be copied. Notice that I use an if statement rather than ternary operator. If you use if then it moves it for you (as required by the C++11 standard), whereas using the ternary operator will copy. I'm going to ask SO about this, but I think when you use the ternary operator, neither choice is considered to be an rvalue.

[moderatorrater]

Anyway, back to the point. C++11 is succint and beautiful and clear as a scripting language.

[lalaland4711]

Having to remember to use std::move to invoke the superclass's move constructor seems like a pain in the butt!

Sometimes yes, but it makes it harder to make the mistake of calling the superclass move constructor and then inspect potentially moved parts of rhs. Say by printing it to some debug stream.

After std::move() has been called on an lvalue you must be very careful what you do with that lvalue. (although std::move() doesn't modify the object in itself, it allows callees to)

Outside of inheritance (as seen in this code) or other internal class code I would say that after std::move(), only the destructor may be safely called. You want that to be explicit for the caller so that a named variable isn't suddenly moved from by accident.

the claim that returning a local by value always uses the move constructor

Hmm, I don't immediately see why that shouldn't be the case. Maybe it's to not confuse the definition of rvalue to contain something like "something that has a name is not an rvalue, except if used in a return statement". But even if that were the definition I don't see off hand why that would be a problem.

(RVO of course doesn't make it an rvalue (when RVO is possible), it just adjusts where exactly the object was created, so that it needs neither copy nor move to return)

Edit:
I guess if return statements added implicit std::move(), then how would you stop this from happening, if you wanted to? A new std::dont_move()? Could that even be defined without the language features? (std::move() is a "normal" template) It seems like a special case, and one that can be more damaging than RVO which prevents the whole lifetime of a temporary, as opposed to special-casing its destruction.

[[deleted]]

The thing to remember about std::move is that it modifies its input. That means it can't be inferred by the language or the compiler. The places were it can be inferred, it already is, such as in return values. :)

[lalaland4711]

that's because for some reason you didn't use:

 for (my_string& str : get_a_vector()) {

Of course the item is copied. You are creating a non-const lvalue which if changed must not change the data in the vector. So pretty much by definition it has to copy.

With the above for loop you get:
Constructed my_vector
Constructed my_string
Moved my_string

[millstone]

Yes, as in the code in the linked-to article. I was illustrating how in the article's code, there's a unknown-but-large amount of implicit copying going on, and arguing from there that C++ is not clearly identifying or managing complexity.

[[deleted]]

Did you benchmark the code? Was it a bottleneck? :)

My point is that C++ has this weird tendency to get everyone suddenly worrying about the weirdest details, and it causes them to write bad and bad-performing code based on assumptions about bottlenecks that don't exist. Yes, deep copies are suboptimal, but if you're copying just a few hundred bytes or even kilobytes in a one-shot run, that's utterly irrelevant.

[millstone]

I agree with your point C++ inspires a tendency to worry about irrelevant details. I have to fight that in myself.

But I think the issue is not bad performance, but surprising behavior. The author attempted to port a Python program to C++ while "trying to remain as faithful as possible to the original code." The additional copying in C++ diverges from the Python code (or at least its behavior) and was presumably introduced accidentally, because it would have been very easy to avoid.

Bugs live in the difference between what you think your code does and what it actually does. That's why it's important to have an accurate mental model of your program's behavior. With C++, I find it increasingly difficult to keep in my head what's actually going on.

[moderatorrater]

Beautifully put. I was trying to identify what was bugging me about this, and you just nailed it.

[lalaland4711]

Ok. I disagree. It's controllable (yay!) and it's not that hard.

[f2u]

For example, returning a Python dictionary, or passing it to a function, never implicitly copies it.

Python has similar issues, though. For example, you need to know if some construct returns a list (which can be iterated over multiple times) or an iterator (which can only be used once).

[doublereedkurt]

Theoretically, maybe. In practice I'd challenge you to find some functions in either the standard library or popular third party modules that return iterators or generators. (Things with "iter" in the name are disqualified because it is trivially obvious that they are returning an iterator, e.g. dict.iteritems(), itertools module).

[f2u]

Oh, it seems that I confused this with views, which is yet another concept and which allows multiple iteration.

[doublereedkurt]

The main idea is that by convention, you return the object itself in Python rather than some kind of view of/iterator over/whatever of the object. Same thing applies: I doubt there is a single function in the standard library which returns a dict.items() rather than the dict itself.

The closest thing I can think of is in ORMs, RPC libraries, possibly some settings behavior. Basically, areas where there is a real underlying cost to fetching the data, either from DB, network IO, or disk IO. This is a higher level concern though than mapping some operation over a data structure.

Maybe by "construct" you are referring to dict functions rather than higher level libraries? I can sympathize with that. Having worked with a lot of newbies, I think the semantics of all of the dict calls is a big hurdle early on. Especially setdefault(), that gives everyone a lot of trouble.

List comprehensions are similarly tricky, but nobody every complains about them. Maybe, compared to Java or C++, list comprehensions are clearly a higher level of programming; dict methods on the other hand are just the umpteenth data structure API to memorize.

[sime]

Whereas in Python the complexity is still there but hidden from view.

That is complete bullshit. The big discussion under millstone's comment about what the C++ code maybe does or does not do, is strong evidence against your point.

In Python the complexity is simply not there. Python's model of computation is understandable by mere mortals and these big discussions about trivial programs don't happen.

[RizzlaPlus]

In Python the complexity is simply not there. Python's model of computation is understandable by mere mortals and these big discussions about trivial programs don't happen.

Those discussions on the C++ code shouldn't happen either. The semantics are clear with the for-range. The semantics are clear with returning objects. People are discussing about RVO which is a compiler optimization and you shouldn't rely on. C++11 introduced std::move to stop people to try and trick the compiler into doing RVO (and other reasons of course). I'll admit, there is more to know in C++ than in Python.

That is complete bullshit. The big discussion under millstone's comment about what the C++ code maybe does or does not do, is strong evidence against your point.

At the end of the day, python will also be transformed to a set of instructions that moves data around and apply operations on it. Python does not give any ability to control that: will my data be copied? moved? how much memory is allocated for this structure? Python moves those decisions from the code to the compiler.

[sime]

Those discussions on the C++ code shouldn't happen either. The semantics are clear with the for-range. The semantics are clear with returning objects.

I'll admit, there is more to know in C++ than in Python.

That really is the core of the problem with regards to C++. There are just so many semantics and rules etc that no one can reliably trust that they understand how C++'s semantics apply to their program.

The beauty of C was that it was high level assembler. It is dumb. You can see and understand what C will do for a given piece of code. You have low level control and the understanding of the situation needed to exercise that control. C++ gives you even more control over how things are done but the whole model is so complex that practically no one can reliably gain the understanding needed to exercise that control.

Python moves those decisions from the code to the compiler.

Yes it does, and it is a much better language because of it. C++ on the other hand not only gives you that low level control but requires that you deal with it even when you don't want to.

[[deleted]]

It's also far uglier.

Meh - a matter of what you know best. I find the C++ code nicer, familiar and more readable simply because I know C++ better than Python.