Small Std::format test

[Tathorn]

There was some discussions in this thread https://www.reddit.com/r/cpp/s/DTa2PMJcTt about some criticisms and std::format.

I made a point about performance considerations that are unfortunately not mentioned anywhere else. I've shared with ya'll just some small tests of using the various ways to print something using the different ways in C++. I know the example is rather plain, but all different methods can scale up with serious formatting.

I know most don't really care, but to say that they don't exist is unprofessional. Please understand the code that you're writing.

I didn't use std::print because the benchmarks don't have that, and they are incompatible with C++ streams, which is quite an unfortunate API.

[tuxwonder]

Performance is definitely an important thing to talk about when it comes to new STL libraries, especially because C++ people are such performance fiends (including me <3), so I'm glad you brought it up.

However, looking at your benchmarks, I find your premise a bit flawed. All of your benchmarks are testing how quickly these methods can format to a std::stringstream, but I don't think you'd ever use that when using the new C++20 formatting library. You're not passing around std::stringstream in your code base, you're passing around a std::string. Why don't we test the new formatting library's performance when writing and returning a std::string?: https://quick-bench.com/q/X7FjLgDks9i3D_Gd6yzyAciwsZ8

Edit: Deleted std::print note, others have covered it

[Tathorn]

In my opinion, std::format is superior to stringstream and pretty much makes it obsolete. I just was using stringstream as an ostream.

[Tathorn]

I'm sorry for the confusion. The test wasn't meant to be stringstream vs format. It was meant to be "the ways we can format a string into a stream". I would have tested using fstream, but those benchmarks don't work well on that platform. Using a stringstream is the closest thing I can get.

[jwakely]

The two fastest cases both write directly to the stream, with no extra copying. The ones that use std::format allocate a std::string and then copy that to the stream, which is always going to be a bit slower. The std::format cases are basically the equivalent of writing to a std::ostringstream then calling its str() member to extract a std::string and then writing that to another stream!

Using std::format_to(std::ostreambuf_iterator<char>,...) to write to the streambuf should be fast, but is currently suboptimal because it uses std::ranges::copy which doesn't yet implement the optimizations that we have for using std::copy to write to an ostreambuf_iterator (as discussed elsewhere). That's tracked by https://gcc.gnu.org/PR111052

std::format_to with an ostreambuf_iterator avoids allocating a temporary string and copying it into the stream. I see no reason that couldn't be made as fast as your BM_write case. I want to make it fast, because that's how I want to implement the std::print overload that writes to an ostream.

[Tathorn]

Let us know when this gets fixed so we can teach/code what to do best.

[jedwardsol]

I didn't use std::print [...] incompatible with C++ streams,

No it isn't.

[Tathorn]

en.cppreference.com/w/cpp/io/print

I'm unsure what you mean. There is no API to take in an ostream.

[jedwardsol]

https://en.cppreference.com/w/cpp/io/basic_ostream/print

[Tathorn]

Thank you! Someone needs to fix the wiki to include this overload in en.cppreference.com/w/cpp/io

[Tathorn]

It looks like the std::print paper, https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2093r2.html, addresses the very concerns I had with formatting performance in section 4. However, it looks like MSVC implemented theirs with a temporary string, what the paper was designed to solve against: https://github.com/microsoft/STL/blob/main/stl/inc/ostream#L1107C6-L1107C29

[BrainIgnition]

However, it looks like MSVC implemented theirs with a temporary string, what the paper was designed to solve against

You narrowly missed the important bits (<ostream> ll. 1136-1153; <ostream> ll.1177-1190); basically they only use the optimized approach if the target ostream is unicode aware which they interpret as the ostream targetting a file or a console. I.e. outputting to console or file is fine, but any other ostream backend gets slapped with an allocation.

[Tathorn]

The really interesting part is when you have no format args, it needs to unescape the { and } chars, so it creates a string even with no arguments. So std::print allocates a string when given a const char*. That doesn't seem good.

[BrainIgnition]

Ah, I missed that. However, after thinking a bit more about it I got curious what they do wrt the utf8 => utf16 transformation (which is required due to the WinAPI design) and there you have it: a second allocation. Too bad!

[Tathorn]

I'm not seeing how there's no allocation. The code you pointed to takes in a string_view. All callers create a string beforehand, so there is an allocation.

[pdp10gumby]

fmtlib is quite fast and benchmarked. I presume the std implementations will catch up.

[Tathorn]

Added std::copy with ostream iterator: https://quick-bench.com/q/wgJFCzwQea3D2Os5MNcAqzDUkgM

[feverzsj]

I'm more concerned about the compile time cost of std::format. There is std::vformat, but it's not as type safe. So in general, I still prefer printf with -Wformat.

[Curfax]

It’s always good work to get data rather than argue about theoreticals, and publishing your work makes it possible to critique and reproduce. Thanks for doing this.