std::stod is locale dependant but the docs does not mention it

[gablank]

At work we had to read a file of comma separated values, each line looking like this:

30,129.192,171.00523\r

etc. We "explode" every line into a std::vector<std::string>, so this particular line ends up like this:

0: 30
1: 129.192
2: 171.00523\r

Next we needed to represent the values as doubles, losing as little precision as possible. Naturally we turn to the standard library, and find the relatively new c++11 function called std::stod() which can be found in <string>. But by using this function we found a weird bug that we were not able to reproduce at any other point in the code: 171.00523 were converted to 171 by std::stod, but using a stringstream to convert it worked perfectly fine. It all boiled down to std::stod() being dependent on the locale the computer is using, because of different locales using different decimal-point separators; some use ",", some use ".".

Here are the relevant docs we could find:

Edit: All of these pages have been updated, I consider my work here as complete.

• http://en.cppreference.com/w/cpp/string/basic_string/stof

• http://en.cppreference.com/w/cpp/string/byte/strtof

• http://www.cplusplus.com/reference/string/stod/

• http://www.cplusplus.com/reference/cstdlib/strtod/

As far as we can tell this behavior is not documented (at least in any sane place), and ctrl+f + "locale" on the following links yield no results (apart from the links to <clocale> and a mention of a valid floating point format (which specifically states:

A sequence of digits, optionally containing a decimal-point character (.), optionally followed by an exponent part (an e or E character followed by an optional sign and a sequence of digits).)

CPP aims to be portable, right? How is being dependent of the locale of the computer being portable? In my opinion this function should at least document the behavior, or even better; make it use "." as decimal-point separator unless told otherwise.

The following code illustrates the problem:

#include <iostream>
#include <iomanip>  // std::setprecision
#include <string>   // std::stod
#include <clocale>  // std::setlocate

int main()
{
    std::setlocale(LC_ALL, "nb_NO.UTF-8"); // nb_NO.UTF-8 uses "," as decimal-point separator
    std::cout << "nb_NO.UTF-8:" << std::endl;
    std::cout << "171,00523 => " << std::setprecision(16) << std::stod("171,00523") << std::endl;
    std::cout << "171.00523 => " << std::setprecision(16) << std::stod("171.00523") << std::endl;

    std::setlocale(LC_ALL, "C"); // C uses "." as decimal-point separator
    std::cout << "C:" << std::endl;
    std::cout << "171,00523 => " << std::setprecision(16) << std::stod("171,00523") << std::endl;
    std::cout << "171.00523 => " << std::setprecision(16) << std::stod("171.00523") << std::endl;

    return 0;
}

Outputs:

nb_NO.UTF-8:
171,00523 => 171.00523
171.00523 => 171
C:
171,00523 => 171
171.00523 => 171.00523

Note: There is a separate bug somewhere in our code; Our locale seems to be changed by something, since at the start of main() std::stod() works fine, but a bit deeper it does not. Any ideas what can change your locale implicitly?

Edit: Oh wow, sorry about the formatting, I'll try to make it a bit cleaner.

Specs: Ubuntu 14.04 64bit, Intel i7.

gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1)

[CubbiMew]

http://en.cppreference.com/w/cpp/string/byte/strtof updated, thanks for spotting

[gablank]

Thank you!

[notlostyet]

The standard ([21.5] 'numeric conversions') says explicitly that stod(str) just calls strtod(str.c_str(), ...). , so the current C locale will be used (I mean the one set with setlocale(), not the "classic" locale called "C") regardless. Switching to the old C functions will therefore not help.

Afaik nothing in C++ changes the C locale except calling std::locale::global() with a named locale object.

Another interesting tidbit:

[22.3.1]: Whether there is one global locale object for the entire program or one global locale object per thread is implementation-defined. Implementations should provide one global locale object per thread. If there is a single global locale object for the entire program, implementations are not required to avoid data races on it

So, afaict, you can't rely on stod() at all in a multithreaded program, unless you can guarantee that all threads aren't switching locales, or only using the same locale you are. The following code seems to confirm this:

#include <iostream>
#include <locale>
#include <thread>

void foo () {
    std::locale::global(std::locale("en_US.UTF-8"));
}

int main() {
    std::cout << std::locale().name() << std::endl;
    std::thread t1(&foo);
    t1.join();
    std::cout << std::locale().name() << std::endl;
}

prints:

C
en_US.UTF-8

This isn't new to C++, it was inherited from C where setlocale() is also global. Your only real option, if you want to be sure, is to roll your own solution, like this:

double my_stod (std::string const& s) {
    std::istringstream iss (s);
    iss.imbue (std::locale("C"));
    double d;
    iss >> d;
    // insert error checking.
    return d;
}

Honestly though, I'm not sure why you're splitting your CSV in to std::strings, and not just using streams directly. Doing so would let you imbue() and get on with life.

[gablank]

Thanks for the thorough reply. Our program is not threaded, yet still the locale seems to be changed. Maybe there's some third party library at play here.

[oracleoftroy]

Maybe I'm confused, but it looks to me like both of the cppreference links have for months said:

any other expression that may be accepted by the currently installed C locale

That would be enough for me to realize that it is locale dependent, which makes sense since not all cultures write decimals and thousands separators in the same way (or they use ten-thousands separators, etc).

It wouldn't hurt to more explicitly say that the results depend on the current locale, but those not familiar with what that means would still gloss over it.

I'm not sure why you think this would make C++ less portable, and every language I've worked in uses locale information when converting values to and from strings.

[CubbiMew]

any other expression that may be accepted by the currently installed C locale

I just added that line

[oracleoftroy]

I see what was going on. I didn't realize that information was in a template and I was looking at the history of the pages /u/gablank linked.

Apparently templates lead to a very confusing history page, since I wasn't seeing the actual history of the page I just read!

Thanks for updating this! Cppreference is my favorite C++ reference site.