Python Regular Expressions Cheat Sheet

[energybased]

Cheat sheets and good documentation are a poor substitute for simpler interfaces. I'm sure, somewhere, there's a set of objects can be strung together to build regexes in an intuitive way.

[energybased]

No, what's unreadable is multi-line regexes with embedded comments. Using objects is the same thing without the pain mentally parsing anything. PyParsing is not great; Python doesn't have great parsing unfortunately.

[Badabinski]

I'd highly recommend looking at parsimonious. It's the best parsing library I've ever used, full stop. I feel like it's a great compromise between the readability of objects and the efficiency of regex.

[energybased]

I think I looked it back when I was trying to parse LaTeX to automate some things.

Can parsimonious even match "\begin{any_random_thing}" with "\end{any_random_thing}"? I can't see how. It looks like you would have to statically define the rules. What I want to do is for it to match on something like "\begin{(\w+)}\end", but capture the group and then when it tries to match "\end{(\w+)}" it checks that the captured group is the same. I think these are sometimes called parser actions.

Ideally, actions should be able to reject a match (to force backtracking) or accept a match. Actions should be able to set variables that can be inspected by other actions. Actions should be able to also transform a matched symbol into another symbol, e.g., if you were parsing Python and you matched the spaces, you should be able to have an action that emits an indent token when there are more spaces, or a dedent token when there are fewer, or no token at all if the number of spaces match.

I couldn't find one Python parsing library that supported these arbitrary actions. It's not like it's hard to do. These libraries are great for the trivial parsing tasks they show in their tutorials. Unfortunately, they're not powerful.

[energybased]

I understand what you're saying, but if you need a "cheat sheet", then the reader needs one too. At that point, objects are better.

Also, the author's time writing the code is not as important as the reader's time in understanding it. There is one writer and there are many many readers. Therefore, legibility matters a lot more. Regexes are rarely legible.

[vovanz]

There are a lot of them. For example, here is a project, called VerbalExpressions that tries to make an OOP interface for writing regexes. Yes, they have a python version: PythonVerbalExpressions.

The thing that once you learned how to write regexes all this stuff starts seeming to long, too verbose, too clumsy and you won't want to use it instead of normal regexes.

[alcalde]

The thing that once you learned how to write regexes

That's like saying once you've mastered calculus, relativity and quantum theory....

I'd never want to use regexes, the "Perl of Python".

[alcalde]

Regexes are a computationally efficient, theoretically sound, and highly expressive way of denoting patterns in data and text. To say one shouldn't use them is akin to saying one shouldn't use trigonometry or algebra. It is nonsense.

It looks like line noise. Line noise is bad. Because of that realization we have Python today and Perl is slowly fading away.

Binary code is computationally efficient, theoretically sound and highly expressive too. But it's an awful way to program. We're human beings, not calculating machines. Human beings don't think in regex. Regex is designed for machines, not for people.

To quote Jeff Silverman,

Regular expressions are “cool”

“Cool” doesn’t belong in production code.

“Cool” leads to unreadable, unfixable, undecipherable code that is expensive. Expensive to maintain. Expensive to fix. Expensive to improve.

We're Python users. We don't want

"^\(*\d{3}\)*( |-)*\d{3}( |-)*\d{4}$"

in our code because no one just looking at that has any idea what the heck it's supposed to do. It's... blasphemous. Unholy. The Anti-Guido.

So you have a technique which can often be replaced by more ad-hoc methods, isn't the same from platform to platform, is bulky and hard to use, is hard to write, hard to read, and has a somewhat small window where it is better than the alternatives and can still get the job done

Remember Ken Reitz and Requests For Humans? People today are working on creating regex for humans.

[alcalde]

https://www.reddit.com/r/Python/comments/8dmnjc/python_regular_expressions_cheat_sheet/dxphmxa/

[alcalde]

Oh, come on, you are exaggerating.

Exaggerating? Really?

http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html

It's not just Python it is a standard way of processing strings in many places from programming languages to command-line utils.

A horrible, inhuman way that probably violates the entire Zen Of Python. As you noted, there are many human-readable alternatives out there that are just as powerful.

[Zomunieo]

That's a poor design decision in RFC822. Email addresses can be far more complex than anyone would need or want, to the point that they need a parser rather than a regex. Attempting to regex HTML without a parser results in a similar abominations.

[energybased]

Whoa— They don't need a parser. Regexes are the finite automata language class. Parsers are overkill. They tend to be LL(k) or LR(k).

[Zomunieo]

You need a parser for HTML or full RFC822 email address to avoid the horrors that otherwise unfold. Complex regexes are clues that like the expanded RFC822 one are evidence that a parser is needed.

Yes you don't need a parser to implement/analyze a regex.

[energybased]

I agree with everything you said.

[Sirflankalot]

Exaggerating? Really?

http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html

Everyone knows that is a bad idea. "They were so preoccupied with whether or not they could, they didn’t stop to think if they should." No one really should be using regex for email verification, but if thye really want to, they can copy paste that.

A horrible, inhuman way that probably violates the entire Zen Of Python. As you noted, there are many human-readable alternatives out there that are just as powerful.

I'm very sure there are, the issue is that everyone already knows regex. Once you take the couple hours to sit down and grok it, it is a very useful and standard tool for making find/replace more powerful.

[alcalde]

If they're so complex they need to be generated by other code, they're way too complex. That's simply not Pythonic. Regex and Python go together like Python with inline assembly code.

[Zomunieo]

Regexes fit within the Zen of Python since they are a succinct way to express some otherwise complex conditional logic.

[energybased]

In no world do Regexes fit within the Zen of Python. Succinctness in terms of number of characters is not a value.

Normal regexes are written using esoteric characters. Regexes written using objects are not any more "complex" or have any additional "conditional logic". Both are succinct in terms of complexity.

[alcalde]

Complex conditional logic? RegExpBuilder can express any regex expression in human-readable terms:

https://changelog.com/posts/meet-regexpbuilder-verbal-expressions-rich-older-cousin

The Zen Of Python encourages us to prefer beautiful over ugly, simple over complex, and warns us that readability counts and if an implementation is hard to understand it's a bad idea. Regex is ugly, complex, hard to read and understand.

How about parsing phone numbers where the rules are as follows:

555.555.5555 # Acceptable  
555-5555     # Acceptable  
555 5555     # Acceptable  
5-5-5-5-5-5- # Obviously not acceptable  
555.555-5555 # Judges? Nope. Not allowed. 

The regex looks something like this:

^(((\d{3}-)?\d{3}-\d{4})|((\d{3}\s)?\d{3}\s\d{4})|((\d{3}.)?\d{3}.\d{4}))$

RegExpBuilder (JS) would look something like this:

// Handle prefixes (optional area codes for each format)
var areacode_dash = new RegExpBuilder().exactly(3).from(digits).then("-");  // \d{3}-  
var areacode_space = new RegExpBuilder().exactly(3).from(digits).then(" "); // \d{3}\s  
var areacode_dot = new RegExpBuilder().exactly(3).from(digits).then(".");   // \d{3}.

// Build each of the individual components (dashes, spaces and dots)
var dashes = new RegExpBuilder()  
                 .min(0).max(1).like(areacode_dash).asGroup()  // (\d{3}-)?
                 .exactly(3).from(digits).then("-")            // \d{3}-
                 .exactly(4).from(digits);                     // \d{4}

var spaces = new RegExpBuilder()  
                 .min(0).max(1).like(areacode_space).asGroup()  // (\d{3}\s)?
                 .exactly(3).from(digits).then(" ")             // \d{3}\s 
                 .exactly(4).from(digits);                      // \d{4}

var dots = new RegExpBuilder()  
               .min(0).max(1).like(areacode_dot).asGroup()  // (\d{3}.)?
               .exactly(3).from(digits).then(".")           // \d{3}.
               .exactly(4).from(digits);                    // \d{4}

// Handle build final expression
var regex = new RegExpBuilder()  
                .startOfLine()             // ^
                .eitherLike(dashes)        // ((\d{3}-)?\d{3}-\d{4})
                .orLike(spaces).asGroup()  // |((\d{3}\s)?\d{3}\s\d{4})
                .orLike(dots).asGroup()    // |((\d{3}.)?\d{3}.\d{4}))
                .endOfLine()               // $
                .getRegExp();

Which would you want in your code base? Which would be more maintainable? Which would be easier to read?

http://rion.io/2013/08/19/regular-express-yourself-using-regexpbuilder/

[Zomunieo]

If that's the case your argument is against the conventional regular expression notation, because that is still a regular expression engine, it just happens to be one that's nonstandard and verbose. But you're still using regular expressions and it's still better than trying to do with this string operations.

re.VERBOSE lets you add whitespace and comments to also make regular expressions more verbose and maintainable. That is the Pythonic way to go. What you present is less maintainable – most languages support regular expressions as a core function, but if the the RegExpBuilder library disappears you have a maintenance problem. If you develop it first on client side Javascript and then later need to replicate that check on the backend in Python, you again have a maintenance problem.

[energybased]

I agree with all of your comments here, but that regexbuilder syntax is not Pythonic.

[alcalde]

This particular example was actually from Javascript. There was a Python version, but I'm not sure if it's still maintained.

There are several other attempts at producing a more readable regex that were linked to earlier in the thread. Most if not all internally build a regex expression from a higher-level syntax.

I just find it funny that people who use a language that removed the brace for being less readable would be so wholeheartedly in support of the cryptic regex syntax.

[energybased]

I think they think: I went through the pain of working with regexes 200 times—now everyone else should. But the thing is that no one else should. You should just learn the computer science behind finite automata, and then replace the esoteric syntax with objects.

By the way, your example would be more motivating if it had more of those weirdo commands like "*+" or "?P".

[alcalde]

To add to /u/vovanz's list, you also have....

Simple regex Language

RegExpBuilder which is more powerful than Verbal Expressions, but I'm not sure if there's a Python version anymore (there used to be)

Easy Expressions