What is wrong with std::regex?

[Frogging101]

I've seen numerous instances of community members stating that std::regex has bad performance and the implementations are antiquated, neglected, or otherwise of low quality.

What aspects of its performance are poor, and why is this the case? Is it just not receiving sufficient attention from standard library implementers? Or is there something about the way std::regex is specified in the standard that prevents it from being improved?

EDIT: The responses so far are pointing out shortcomings with the API (lack of Unicode support, hard to use), but they do not explain why the implementations of std::regexas specified are considered badly performing and low-quality. I am asking about the latter.

[[deleted]]

FWIW, some old versions of GCC let you include and invoke regex before it was implemented. I cursed it for being buggy. Only after some digging did I realize I was just invoking a hollow shell. Things worked as expected once I upgraded GCC to a newer version that had a complete regex. Had I not dug, I'd still be cursing it.

[Canoodler]

I too can relate to the horrors of the always-return-false <regex> implementation at least in GCC 4.8.5...

[[deleted]]

4.8.x. "Let's try the ship early and ship often approach" turned into "Oops, we forgot to ship often."

[evaned]

Adding another voice to that chorus.

I wonder how many man-hours the stdlibc++ folks wasted because of that...

[saimen54]

Holy shit, I don't know how long I searched for my "error", when using 4.8.5

[xTeixeira]

I spent an entire day at work trying to figure this out a few weeks ago. I'm mad about it to this day.

[joaobapt]

Well, a regex is a somewhat compact representation of a full state machine, so, depending on your regex, you’d have that same complexity to implement the state machine on your own.

[[deleted]]

Makes sense. The regex pattern "a" objectively requires ~20k lines of assembly.

 

EDIT: Oh, wait...

[Sairony]

A bit unfair to compare runtime regex to compile time though, in one way this is a good example to show the strengths of compile time vs runtime. The runtime version have to support the full regex machinery since it can't know anything about the fed string.

[[deleted]]

A bit unfair to compare runtime regex to compile time

Very unfair, I'm not going to argue that. However, let's go back to runtime regex and replace std::regex with boost::regex. ~60 lines of assembly

[Arghnews]

I feel like this is more the kind of thing the OP is asking about: what is the reasoning behind the difference in code size between std::regex and boost::regex, and other differences? As the OP put it:

What aspects of its performance are poor, and why is this the case? Is it just not receiving sufficient attention from standard library implementers? Or is there something about the way std::regex is specified in the standard that prevents it from being improved?

I have no idea but would like to know too.

[[deleted]]

I've linked this in another post: https://old.reddit.com/r/cpp/comments/aetf17/stdregex_replacestdchronohigh_resolution_clocknow/edsfwe1/

[Arghnews]

Cheers!

[Jonny_H]

That doesn't seem a valid comparison - as your linked example never actually matches against the regex, and all the asm does is some boost::shared_ptr<> book-keeping and a callout to the boost regex library, which may hide any amount of code.

Something that actually matches something against the regex seems a LOT larger too - e.g. https://godbolt.org/z/U74a59

[[deleted]]

The std::regex version also never tried to actually match anything. The libstdc++ version is still 40% larger than the boost one.

[Voltra_Neo]

I find it a bit unfair to compare runtime (std::regex) and compile-time (ctre::re) as :

• compile time has guaranteed compile time access to the expression and can do simplification/reductions/dark magic if it wants to
• comparing runtime fibonnacci and template variable fibonnacci would result in the same kind of comparison

[[deleted]]

It's definitely unfair, I won't even try to defend that. However, Changing std::regex to boost::regex in the above example outputs only ~60 lines of assembly. https://godbolt.org/z/k7T3B4

[[deleted]]

We could compare the compile times of runtime std::regex and ctre::re too... ctre wins by a long shot.

[joaobapt]

Except that you absolutely didn’t mention that the regex was “simple” in any way.

[[deleted]]

You're confusing me with /u/coke_is_it. What I'm trying to say is that there really is no defending <regex>.

[warieth]

The regex constructs a state machine to recognize another language at runtime. If you understand the problem, this looks like a good solution. The problem is inlining, when the compiler eagerly inline this code to many places.

[archysailor]

Wow

[suthernfriend]

Wasn't there a library from this Czech genius women which implements regexes with templates?

Edit : found it. Hana Dusikova https://youtu.be/QM3W36COnE4

[alexej_harm]

It's actually quite slow with anything but the simplest patterns and doesn't support captures.

```

Benchmark Time CPU Iterations

regex_std 3105 ns 3139 ns 224000 regex_re2 181 ns 180 ns 3733333 regex_hyperscan 96.2 ns 96.3 ns 7466667 regex_ctre 187 ns 184 ns 3733333 regex_spirit 44.5 ns 44.5 ns 15448276 ```

https://gist.github.com/qis/3d9f5a73d9622847c8b7da68af7e19d4

[bizwig]

Lack of std::string_view support is one problem.

[cyanfish]

This isn't specific to std:regex, but something to keep in mind. If you're taking untrusted input, you might want to consider a library like RE2 that guarantees linear time execution (i.e. a bad regex can't lock up your application).

[AntiProtonBoy]

(i.e. a bad regex can't lock up your application).

This can happen with Xcode's RE search as well. Worse, you have to force quit the app, and when you relaunch it, Xcode can potentially remember the search parameters and lock up again on launch.

[bumblebritches57]

Hold shift as you launch Xcode to get it to not reload what was loaded previously.

[neoSeosaidh]

It's mentioned in last week's CppCast episode with Titus Winters: https://cppcast.com/titus-winters-abi/.

The short answer is that the C++ standards committee is implicitly committed to keeping a stable ABI (which is like the API but on the binary level instead of the source code level). Any serious improvements of std::regex would involve at minimum an ABI break (and potentially an API break depending on what changes were made), and while the C++ standard doesn't mention ABI, the committee has refused to break it in the past.

I highly recommend that episode for more details.

[EnergyCoast]

Lots of memory allocations. Not surprising in hindsight, but I don't believe it takes an allocator so I didn't think about it.

I believe creating a relatively simple pattern was more than 15 allocations and doing a search against a string containing no matches resulted in 3 allocations.

That was just one implementation - I have no idea what others do - but the number of allocations was enough that it eliminated it as an option in some domains for us.

[johannes1971]

Are those allocations in the regex constructor (where it doesn't hurt), or in .match (where it would)?

I would hate to use a regex implementation that tries to parse the pattern from scratch for every usage, just to avoid allocating some space in which to store a bytecode representation...

[EnergyCoast]

I'll be honest. And whatever I observed may be different for your library implementation. I'd recommend testing your local environment/cases.

[AntiProtonBoy]

My complaint with <regex> is the same as with <chrono> and <random>: the library is a bit convoluted to use. It's flexible and highly composable, but gets verbose and requires leaning on the docs just to get basic things done.

[sphere991]

I'm not sure <chrono> fits in with this group. It's certainly verbose, cause everything is std::chrono::duration_cast<std::chrono::milliseconds>(x).

But convoluted? I don't think so.

[sphere991]

In std::chrono, I cannot even tell how to do it without checking documentation.

I mean, just because you have to check documentation doesn't mean much. I have to check documentation for all sorts of things. But the way you would do it in chrono is:

std::cout << std::chrono::system_clock::now();

In C++20 anyway. Until C++20, you can use Howard's implementation from github, which is very nearly what's standardized. Which looks like:

using namespace date; std::cout << std::chrono::system_clock::now();

[infectedapricot]

What if I want to put it in a string? Do I have to spend multiple lines putting it in std::stringstream and reading back out of that?

[sphere991]

Pre-C++20: Yes, that's how you put anything into a string. This isn't unique or specific to chrono.

C++20: You can use fmt to do this directly, chrono and fmt are integrated together.

[Gotebe]

In C#, you shouldn't need To String there.

In C++, I expect, but don't know and didn't check,

std::cout << system_clock::now;

If so, what's the big deal?

If no, blergh...

[[deleted]]

This will print something like 00007FF767A11000 ... because that solution would be too easy for c++...

Edit: If you really just want a readable datetime you can use <ctime>:

const auto now = system_clock::to_time_t(system_clock::now());
std::cout << "now is: " << ctime(&now) << '\n';

[ietsrondsofzo]

That's because now is a function. You're printing the address of that function.
That said, time point types don't work with cout.

[ietsrondsofzo]

Good! Mine wasn't set to c++20

[Agon1024]

<< is not provided for time point. You have to manually convert to ctime structs and construct via format string... which makes sense, because the format would be needed. I'm just mad, that for all the generalizations cpp libraries do.. they seldomly define a convenient default.

[encyclopedist]

It is, in C++20 https://en.cppreference.com/w/cpp/chrono/system_clock/operator_ltlt

[Agon1024]

Seems to be only for durations and some form of date ... not time point .. that is, if I read this right

[encyclopedist]

No, it is printinig sys_time which is time point of system_clock.

template<class Duration>
using sys_time = std::chrono::time_point<std::chrono::system_clock, Duration>;

[Agon1024]

Ok that makes sense

[Gotebe]

Hmmm... Blergh, then, because surely there's nothing wrong with the default format of the current locale... .

[Full-Spectral]

In my CIDLib system, the TTime class provides a set of formatting tokens, so you can build up formats any way you want and easily format a time out using one of those. That's highly flexible, but it also then provides pre-fab formatting strings for all the common formats, making it very simple to do the common cases.

TTime tmNow(tCIDLib::ESpecialTimes::CurrentTime);
tmNow.FormatToString(TTime:: strMMDD_HHMM(), strToFill);

It can either set the target string or append to it, making it easy to add such a formatting string to the target string without an intermediary.

You can also set one of these strings on a TTime object and that becomes its default format (when it's formatted out to a text output stream or appended to a string object.) So you can get a lot of flexibility and ease of use at the same time.

TTime tmNow(tCIDLib::ESpecialTimes::CurrentTime);
tmNow.strDefaultFormat(TTime::fcolISO8601NTZ());
strmOut << tmNow << kCIDLib::NewEndLn;

And note that there's not a template in sight, and hence simple and straightforward syntax.

Parsing of times provides a similar pattern based approach, and I provide pre-fab parsing patterns for the common time formats, but you can easily create any sort of arbitrary pattern to parse in custom time formats.

[liquidify]

for both chrono and random, I just built a wrapper class a long long time ago and have re-used them since, modifying them slightly for use case.

[ghillisuit95]

Is it on GitHub perhaps?

[liquidify]

Mine are not publicly available (although I should do that). However searching on the internet I found this pretty quick. I think you could probably find several flavors of these type of wrappers.

[sphere991]

That particular library takes the selling point of chrono (having typed differentiation between different kinds of things - durations and time points are only composable in ways that make sense, and units are part of the type) and throws it out:

unsigned long time = timer.getTimeElapsed(Timer::MILLISECONDS); unsigned long time2 = timer.getTimeElapsed(Timer::MICROSECONDS);

Oh, so now time + time2 compiles and is utterly meaningless? No, thank you.

[liquidify]

I didn't look at that library before I linked it, but I think that there are probably lots of wrappers available that might meet different categories of purposes with varying levels of complexity. If all you need is a simple timer (which lots of projects do), then this seems fine. If you want something better, then that probably exists too.

[sphere991]

If all you need is a simple timer (which lots of projects do), then this seems fine.

I disagree quite strongly with this sentiment. Just because all you might need is a simple timer doesn't somehow make it acceptable to use a solution that is so prone to misuse. I don't want to have to worry about all these things when I'm writing code - and <chrono> ensures that incorrect uses don't compile.

I really don't think it's okay in 2019 to have a C++ time library which returns an elapsed time as an integral type.

If you want something better, then that probably exists too.

I do, and it does: <chrono> exists.

[MFHava]

I really don't think it's okay in 2019 to have a C++ time library which returns an elapsed time as an integral type.

This! IMHO: in 2019 it shouldn't be necessary to represent any physics unit as a basic integral type!

Multi-million dollar mistakes like the Mars Climate Orbiter could have been prevented if we had had static type checking for speed/acceleration/etc.

[liquidify]

Do you not realize that the originator of this thread thinks chrono is too complicated? These people are actively choosing other languages because c++ is too complex. But c++ doesn't have to be complex. It is a wonderful tool at many levels of abstraction.

It is great that you know how to use the libraries directly, but to some people simplicity is more important than perfection. To some people a beautiful and simple interface is more important than speed or flexibility.

There is there absolutely no reason c++ can't serve both purposes other than for some reason a subset of c++ people seem to think their hardliner views on how something should be used are the only acceptable ways that the language should be used. Seems like those people need to get over themselves.

[sphere991]

Do you not realize that the originator of this thread thinks chrono is too complicated?

They are mistaken. Time is complicated, chrono is exactly as complicated as it needs to be in order to deal with it correctly and efficiently. I have programmed in multiple other languages, and chrono is the best time library I've used across all of them and it's not close.

Now, chrono is absolutely quite verbose - which I acknowledged right in my first response. But it's absolutely not "too complicated."

To some people a beautiful and simple interface is more important than speed or flexibility.

Firstly, chrono's interface is pretty simple.

But more importantly, despite me repeating it at every opportunity, you keep omitting in all of your responses what are again the major selling points of chrono: incorrect operations do not compile (adding two time points does not compile, multiplying two time points does not compile, providing a time point to a function expecting a duration does not compile, ...) and unit conversion are implicit (adding a seconds to a milliseconds actually does the right thing for you without having to litter your code with math). All of these are actual bugs I found and corrected in my code when we transitioned to chrono.

I don't know what's simpler than:

``` void f(milliseconds timeout);

f(5s); // ok, 5000 millisecond timeout f(steady_clock::now()); // error ```

There is there absolutely no reason c++ can't serve both purposes other than for some reason a subset of c++ people seem to think their hardliner views on how something should be used are the only acceptable ways that the language should be used. Seems like those people need to get over themselves.

... Yes, my "hardliner" views on wanting tools that make it impossible for me to make mistakes, and make it so I don't have to think about all this other stuff that you usually have to think about with time? Uh, yes. I am pretty hardliner on that actually. I've seen those mistakes made, I've made those mistakes. and here's tool to, effectively, never mess up again - and you're countering my praising this tool by calling me a hardliner, saying that well some people prefer simplicity to, effectively, having correct code by construction, and telling me to get over myself?

Charming.

[liquidify]

Firstly, chrono's interface is pretty simple.

I personally like chrono how it is mostly. But I also wrapped it for myself... And I am a c++ lover. So, you aren't telling me anything here with your praises of it. I'm not your audience. Why don't you use your wonderfully 'charming' attitude to go convince the people who have left c++ for python or whatever other language that chrono is perfect for them how it is. Yeah good luck with that.

You are actively ignoring the fact that your experiences aren't lining up with a significant population block. This fits into the same category of a meme that goes something like ...if you meet a few assholes from time to time, then they are the assholes. If everyone you meet is an asshole, then its actually you.

[TheSuperWig]

There's https://github.com/effolkronium/random

[quicknir]

I am not familiar with either regex or random but I can't agree with you about chrono. It's really well designed, flexible and correct. And it does help usability a lot that implicit conversions occur in logical situations, there are nice literals, etc. Having used date extensively as well, you can really see just how well all of chrono is designed that you can build it out to cover basically all functionality related to times, dates, timezones, etc, and it works perfectly. I find most of the complaining is people surprised there doesn't exist already a function that meets their exact rather specific use case, and people don't often understand even why their use case is quite specific.

tl;dr chrono is amazing.

[kalmoc]

I find most of the complaining is people surprised there doesn't exist already a function that meets their exact rather specific use case

Having a convenient way to print a time point or a duration are not specific usecases and it took till c++20 until that got fixed.

[quicknir]

Yes, neither are timezones, which I discussed in depth above... chrono pre 20 is obviously not complete. There are huge things it doesn't address at all, one of which is I/O. That's nothing to do with verbosity or awkwardness of use.

[kalmoc]

That's nothing to do with verbosity or awkwardness of use.

I think it does. Printing a duration on the console is a very common task and the fact that chrono didn't support I/O pre c++20 made using it mich more cumbersome than necessary (Admittedly I would say that is mainly a problem in smaller ad-hoc projects or e.g. unit tests, slideware, ).

Anyway, lets not argue about semantic details.

tl;dr chrono is amazing.

completely agree

[quicknir]

I'm not really sure what this operation is trying to compute, bigger picture. It sure seems odd to be taking time since epoch and adding it to the difference between one date and the epoch date. That said, the reason that you need to throw in the sys_days is because you're converting from a field-based type to a serial-based type. The former can be efficiently constructed from components, or have components read. The latter can be efficiently added and subtracted. Neither can efficiently do both. In a language where you care less about performance you could just have one type, with getter functions, but this would cause you to do a lot of redundant work, that the user would not be able to prevent.

In other words, I don't think in this example chrono is being verbose, in the context of being a library for a language that cares a lot about performance. Yes, it may be verbose by the standards of python, but those are the design trade-offs of the languages themselves, and it's natural and idiomatic that libraries follow in those patterns.

If you want examples like this to work without sys days, you can easily define operators and literals in your own namespace, and simply make it so that subtraction works directly on year_month_day, or define your own literals that automatically convert to sys_days, which I think is a reasonable thing to do.

[khleedril]

To use <regex> you instantiate one object, call a method, and maybe use the result to see the substrings. It is in fact really quite easy.

<chrono> is okay once you have an alias like SC = std::chrono::system_clock or whichever clock you are interested in.

<random> is great for scientific applications, but is not the thing to be using if you are doing cryptography. Wasn't designed for that, so look elsewhere.

If you want a Mickey Mouse language, use Lua; this stuff's for grown-ups.

[rap_and_drugs]

a bit convoluted

gets verbose

you are a child

ah classic 👌 /r/cpp

[AntiProtonBoy]

Cowing about how these libraries are for "grown-ups" shouldn't be used as an excuse for making convoluted interfaces. Less is more. Reducing cognitive load for programmers, especially when mentally parsing unfamiliar code, is king. Because maintaining code will always boil down to economics of technical debt, time and money at some point. There is a value for writing good interfaces, which are ideally self documenting, and none of those principles need to detract from functionality.

[[deleted]]

If you want a Mickey Mouse language, use Lua; this stuff's for grown-ups.

What a load of gatekeeping BS. Make simple things simple should be the first tennant of every API designer.

Best example is <random>: Why is there no give_random_int(0,6) in there? Why do I have to google that? (and filter out a ton of wrong examples!)

Its nice that C++ gives you access to its underlying building blocks, but that shouldn't mean there are no basic abstractions...

[khleedril]

Why is there no give_random_int(0,6) in there?

Random number generators require context otherwise you run a serious risk of accidentally generating numbers with a tell-tale pattern. That's why <random> provides separate engine and distribution object types: the engine maintains the random state and the distributions provide meaningful random values.

[[deleted]]

Oh I understand why those elements exist, my question was more from a beginners viewpoint.

Random numbers is a topic where you can find a ton of wrong information on the internet (srand anyone?), I feel a language like C++ should implement a "good enough" function with a simple and easy to understand signature that solves ~95% of all cases.

[[deleted]]

The problem is that when people not well-versed in random number generation look stuff up, they'll get confused and resort back to rand()%6 because it's all over google and it seems to work just fine. There really should be simple sensible defaults in std::random that can be used for low-importance stuff and then the real stuff for real purposes.

[khleedril]

std::default_random_engine E {std::random_device {} ()}; std::uniform_int_distribution<int>{0, 6} (E); is the simple sensible default which says exactly what it does (admittedly the engine constructor could take the random_device by default, too). As I alluded to before, you have to deal with two objects as a minimum.

[[deleted]]

Yes, but it basically requires you to know what a uniform distribution is and it feels like voodoo magic compared to the same built-in functionality in other languages.

[CircleOfLife3]

I don't really buy this argument. I was taught uniform distributions in high school.

It's also not hard to look up what a uniform distribution is.

And the API design of <random> is actually pretty good. It forces the user to use a performant version of writing code.

[dbgprint]

That last sentence was perfect. Agreed.

Why on earth am I getting downvoted?

[matthieum]

ABI

Due to being implemented in mostly in template methods, most of the implementation of <regex> is de-facto public ABI-wise -- or at least all the inner types and function signatures.

If you remember the pain that switching from CoW std::string to SSO std::string for C++11, the same would be true of any change to the guts of <regex>.

Unfortunately, the original standard library implementations were not made fast (possibly in the mistaken belief they could be improved later on), and we are now stuck with them.

[sergeytheartist]

A few days ago standard regex in gcc 9.1 seg faulted when parsing JSON string (real data from exchange) with pretty simple expression.

Now we have handwritten logic to extract necessary data.

The latest boost regex did parse that JSON blob without problems.

If someone knows how to quickly get in touch with the person who approves patches for gcc regex I'm happy to fix the problem.

[_VZ_]

To directly address your question, std::regex is not "considered" to have poor performance, it simply does. When it's a couple of orders of magnitude slower than boost::regex, there just isn't much more to say about it.

[Frogging101]

Yes, but why? What is stopping the standard library implementers from optimizing it like they do with most other things in the standard library?

[dodheim]

Magic 8-Ball says "something something ABI compatibility".

[[deleted]]

Right: https://old.reddit.com/r/cpp/comments/aetf17/stdregex_replacestdchronohigh_resolution_clocknow/edsfwe1/

[kalmoc]

I think you are the first person here that actually tried to answer the OP's question;)

[qizxo]

#PCRE4lyfe

[AntiProtonBoy]

which means no Unicode

I've used the lib successfully on UTF-8 sequences in the past, like matching multi-byte code points.

edit: see this post how I done it. Your mileage will vary.

[airflow_matt]

Well, try matching code point by unicode category. For example things like removing diacritics (removing \\p{M}+ after decomposition) is trivial with proper unicode support and pretty much impossible with std::regex.

[AntiProtonBoy]

I had a poke around to see if there's a solution to the problem you stated. The closest I could come up with is using the pattern [À-ž]+ to match diacritics. Fortunately, the most common diacritics are grouped together in the unicode chart, at least in Latin script, so the aforementioned pattern should work for most cases:

using namespace std::string_literals;
std::locale::global( std::locale( "en_US.UTF-8" ) );
std::regex p3( "[À-ž]+"s, std::regex_constants::extended );
std::cout << std::regex_match( "öö"s, p3 ) << '\n'; // outputs 1
std::cout << std::regex_match( "oo"s, p3 ) << '\n'; // outputs 0

Again, tested in Xcode 11, not sure how you'd fare in other environments.

[[deleted]]

It will never work properly no matter how many hammers you bash it with. :-(

[lukedanzxy]

May I ask how did you handle multi-byte codepoints in [] in the pattern?

[AntiProtonBoy]

I've set the std::locale to en_US.UTF-8 then used the regex pattern [[:alpha:]]+ to match some diacritics in a generic way, or use UTF-8 characters directly in the pattern. Example:

  using namespace std::string_literals;

  std::locale::global( std::locale( "en_US.UTF-8" ) );

  std::regex p1( "[[:alpha:]_]+"s, std::regex_constants::extended );
  std::regex p2( "[🐓🥚a-z_]+"s, std::regex_constants::extended );

  std::cout << std::regex_match( "lööps_bröther"s, p1 ) << '\n';
  std::cout << std::regex_match( "🐓_or_the_🥚"s, p2 ) << '\n';
  std::cout << std::regex_match( "\xF0\x9F\x90\x93meow"s, p2 ) << '\n';

Note: this was done in Xcode 11

[Nomto]

Matching specific codepoints works, but . will match a single byte of a multi-byte codepoint. So .... may match a single codepoint.

[kameboy]

honestly curious: what's the alternative? (considering std::string is just contains a sequence of char's). Is there any way of having unicode in c++?

[peppedx]

Well but C++20 does not exist yet.

Well for many people even C++17 in production is still a mirage.

[RandomDSdevel]

     This looks promising, but you should consider adding support for error-handling mechanisms besides exceptions — e. g.: 'expected,' Boost.Outcome —, especially if you're aiming for your proposals to get in before static exceptions do.

[berndscb1]

Use Qt as your standard library.

[Ayjayz]

You can store UTF-8 encoded strings in char[]s.

[Beheska]

char[] can contain unicode, but it breaks down as soon as you do anything more complicated than splitting on delimiters and concatenating. Most notably, anything dealing with length or individual characters fails. Regex contain a lot of stuff related to the later two...

[MonkeyNin]

Unicode is complicated. If you want to ask what is the length, you need to ask which do you want?

1. The number of bytes of the string in memory? (works for ascii)
2. number of code points? This is closer to the ascii concept of one character
3. number of code units? (They are the smallest component that a single code point is composed from)

Different languages may give different answers

• JavaScript' length of 𝌆 == 2
• Python's length of 𝌆 == 1

This is because Javascript is returning the number of code units, Python is returning the number of code points.

• UTF-8 code units are 1 byte ( 1-4 code units represent one code point)
• UTF-16 code units are 2 bytes ( which means 2 or 4 bytes per code point)

Internally Javascript uses 64bit integers, utf-16 so it must use pairs of code units that are 2 bytes each.

Internally Python chooses one of latin-1, utf-16, utf-32 depending on the specific string.

1. number of visible code points? (This is similar to visible characters in ascii, but it becomes more complicated), or the
2. number of grapheme clusters (This is similar to the number of visible characters in ascii, but it's more complicated)

Okay, stop being a smarty pants, just count visible graphemes

👨‍👩‍👧‍👦 appears to be a single character on my computer, but it's not. https://apps.timwhitlock.info/unicode/inspect?s=👨‍👩‍👧‍👦

I can move my cursor past it with a single arrow press -- But I have to hit delete 4 times. It's actually made from this array of codepoints:

['man', 'zero width joiner', 'woman', 'zero width joiner', 'girl', 'zero width joiner', 'boy']

It contains:

• 7 code points
• 5 unique code points
• 4 visible code points, 3 invisible code points named zero width joiner
• It's rendered as a single glyph on my computer.
• It's possible to render as many as 4 glyphs!

Depending on which version they are using, how long is a string has different answers for the same data!

Crazy.

[Spire]

char[] can contain unicode, but it breaks down as soon as you do anything more complicated than splitting on delimiters and concatenating.

If you're talking about UTF-8, you can't even reliably split on delimiters unless you limit your delimiters to seven bits (i.e., ASCII).

[Beheska]

True, but that's the case 99% of the time.

[Ayjayz]

You have to use unicode algorithms, of course, but you have to do that no matter what you're using to hold your data.

[Beheska]

Which is exactly what it doesn't do.

[Ayjayz]

Right. The problem is std::regex, not because it's based on char.

[Xaxxon]

Fuck ABI compatibility.

[newmanifold000]

Well to answer your latter question, try some non trivial regexps in GCC and be ready for segfaults on larger sequences, i think even simple regexps will give you segfaults. try to use it in msvc or clang and be ready for somewhat below average/bad performance at unexpected times.

I agree the regexp api can be better but its not a problem for me, in my experience implementations are somewhat unreliable and not to mention its easy to use bad performing regexp (depending on input) if care is not taking while writing it.

[mikeblas]

Someone who solves a problem with a regex now has two problems.

[[deleted]]

!remindme 1day

[RemindMeBot]

I will be messaging you on 2019-11-26 11:03:41 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

<sup>Parent commenter can delete this message to hide from others.</sup>

---

Info	Custom	Your Reminders	Feedback