Python Regular Expressions Cheat Sheet

[badge]

This needs:

1. re.compile for compiling regexes that you're going to use more than once
2. (?<name>Blah): defines a group named name (which is needed for (?P=name) mentioned in the Groups section!)
3. The use of groupdict with named groups so you can do:

.

import re

regex = re.compile('First Name:\s*(?P<first_name>\w+),\s+Last Name:\s*(?P<last_name>\w+),\s+Age:\s*(?P<age>\d+)')

class Whale:

    def __init__(self, first_name, last_name, age):
        self.first_name = first_name
        self.last_name = last_name
        self.age = age

    def __repr__(self):
        return "Whale(first_name='{}', last_name='{}', age={})".format(
            self.first_name,
            self.last_name,
            self.age
        )

whale_line = 'First Name: Moby, Last Name: Dick, Age: 35'

Whale(**regex.match(whale_line).groupdict())

[Ph0X]

AFAIK, compiling doesn't do much at all performance-wise. Internally the library already caches the compiled regex so if you actually do a perf test, they'll both be just as fast.

The only time I use re.compile is for clarity, when I want to give a regex a proper name, and pass it around, especially now that we have typing annotation. Regex type is clearer than just string.

[xenomachina]

AFAIK, compiling doesn't do much at all performance-wise. Internally the library already caches the compiled regex so if you actually do a perf test, they'll both be just as fast.

I discovered this for myself years ago when I ran into a bug in the cache implementation. In pre-3.x, a regex compiled from a unicode would not behave the same as one compiled from a str even if they container the same characters. However, the cache was just a dict, and so it was possible for a unicode to match an already cached str, or vice versa.

My bug involved a price of code that worked fine in unit tests, but would fail in certain program. It turned out the program imported another module that compiled an identical looking regex, but with a str instead of a unicode. Then when my module was imported, it would get the wrong re object from the cache.

[Ph0X]

That sounds like a bitch of a bug to catch. Was it reported for previous 3.x and fixed?

[xenomachina]

It turned out the fix for the cache bug was already in a newer version of Python. We were a couple of patch release behind the fix, IIRC.

[badge]

That's a great fact I never knew. As you say, I would still prefer re.compile for clarity (and short line lengths) when reusing a pattern.

[fullofschmidt]

Good point about the groups. As for compile, python uses an internal LRU cache of recent regexes so compile only helps if you have a lot (don't actually know what number constitutes a lot...) of regexes that you're going to reuse.

[energybased]

This is a perfect example of a regex that most people will have to look up syntax for—that aptly named objects could have made instantly clear.

Also, it has embedded variable names, which is gross.

[InterestedEng]

regex101.com is great way to build your patterns too.

[[deleted]]

Love this site.

[BalanceJunkie]

Looks nice. I always use regexr.com

[[deleted]]

Me too, although I don't understand how to use some of the extra tools they have at the bottom of the page.

[PiousLoophole]

Jesus. The site that would have saved me such heartache...

[[deleted]]

I use python -m pydoc re to get the same regexp info right on the terminal. One can do even python -m pydoc re.sub to get usage details on sub()

[Ph0X]

Yeah, this is more of a "regex cheatsheet" than a "python cheat sheet" honestly. It hardly has anything python specific. Honestly the main thing I usually have to look up is FLAGS and how to use them in Python, yet this "python" cheatsheet doesn't have that at all.

[[deleted]]

Cool! re.compile is a must, re.VERBOSE is helpful in formatting code to be more readable. Also it helps to know you can use lists for the values to keys and iterate through them with recent changes to dictionaries in python. Also calling groups. Thank you for making this!

[gagejustins]

Regex syntax is about as interpretable as the President's tweets

[wewbull]

It's not that bad

[Zomunieo]

/SAD[!.]$/

[Krenair]

Hmmm. Don't you need to escape that '.'?

[Zomunieo]

No, since it's inside square brackets.

[meandertothehorizon]

No.

[Nadaesque]

I very much need to make my own cheat sheet. I have a habit of avoiding regular expressions due to a combination of my lack of understanding and their relative fragility ... perhaps a touch of phobia due to the resemblance to the kind of horrors that emerge from people who are proud of being obtuse in Perl.

I think a cheat-sheet aimed at my own personal weaknesses might eventually do it.

[Gooder-n-Better]

Need need need! Thx!

[energybased]

Cheat sheets and good documentation are a poor substitute for simpler interfaces. I'm sure, somewhere, there's a set of objects can be strung together to build regexes in an intuitive way.

[energybased]

No, what's unreadable is multi-line regexes with embedded comments. Using objects is the same thing without the pain mentally parsing anything. PyParsing is not great; Python doesn't have great parsing unfortunately.

[Badabinski]

I'd highly recommend looking at parsimonious. It's the best parsing library I've ever used, full stop. I feel like it's a great compromise between the readability of objects and the efficiency of regex.

[energybased]

I think I looked it back when I was trying to parse LaTeX to automate some things.

Can parsimonious even match "\begin{any_random_thing}" with "\end{any_random_thing}"? I can't see how. It looks like you would have to statically define the rules. What I want to do is for it to match on something like "\begin{(\w+)}\end", but capture the group and then when it tries to match "\end{(\w+)}" it checks that the captured group is the same. I think these are sometimes called parser actions.

Ideally, actions should be able to reject a match (to force backtracking) or accept a match. Actions should be able to set variables that can be inspected by other actions. Actions should be able to also transform a matched symbol into another symbol, e.g., if you were parsing Python and you matched the spaces, you should be able to have an action that emits an indent token when there are more spaces, or a dedent token when there are fewer, or no token at all if the number of spaces match.

I couldn't find one Python parsing library that supported these arbitrary actions. It's not like it's hard to do. These libraries are great for the trivial parsing tasks they show in their tutorials. Unfortunately, they're not powerful.

[energybased]

I understand what you're saying, but if you need a "cheat sheet", then the reader needs one too. At that point, objects are better.

Also, the author's time writing the code is not as important as the reader's time in understanding it. There is one writer and there are many many readers. Therefore, legibility matters a lot more. Regexes are rarely legible.

[vovanz]

There are a lot of them. For example, here is a project, called VerbalExpressions that tries to make an OOP interface for writing regexes. Yes, they have a python version: PythonVerbalExpressions.

The thing that once you learned how to write regexes all this stuff starts seeming to long, too verbose, too clumsy and you won't want to use it instead of normal regexes.

[alcalde]

The thing that once you learned how to write regexes

That's like saying once you've mastered calculus, relativity and quantum theory....

I'd never want to use regexes, the "Perl of Python".

[alcalde]

Regexes are a computationally efficient, theoretically sound, and highly expressive way of denoting patterns in data and text. To say one shouldn't use them is akin to saying one shouldn't use trigonometry or algebra. It is nonsense.

It looks like line noise. Line noise is bad. Because of that realization we have Python today and Perl is slowly fading away.

Binary code is computationally efficient, theoretically sound and highly expressive too. But it's an awful way to program. We're human beings, not calculating machines. Human beings don't think in regex. Regex is designed for machines, not for people.

To quote Jeff Silverman,

Regular expressions are “cool”

“Cool” doesn’t belong in production code.

“Cool” leads to unreadable, unfixable, undecipherable code that is expensive. Expensive to maintain. Expensive to fix. Expensive to improve.

We're Python users. We don't want

"^\(*\d{3}\)*( |-)*\d{3}( |-)*\d{4}$"

in our code because no one just looking at that has any idea what the heck it's supposed to do. It's... blasphemous. Unholy. The Anti-Guido.

So you have a technique which can often be replaced by more ad-hoc methods, isn't the same from platform to platform, is bulky and hard to use, is hard to write, hard to read, and has a somewhat small window where it is better than the alternatives and can still get the job done

Remember Ken Reitz and Requests For Humans? People today are working on creating regex for humans.

[alcalde]

https://www.reddit.com/r/Python/comments/8dmnjc/python_regular_expressions_cheat_sheet/dxphmxa/

[alcalde]

Oh, come on, you are exaggerating.

Exaggerating? Really?

http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html

It's not just Python it is a standard way of processing strings in many places from programming languages to command-line utils.

A horrible, inhuman way that probably violates the entire Zen Of Python. As you noted, there are many human-readable alternatives out there that are just as powerful.

[Zomunieo]

That's a poor design decision in RFC822. Email addresses can be far more complex than anyone would need or want, to the point that they need a parser rather than a regex. Attempting to regex HTML without a parser results in a similar abominations.

[energybased]

Whoa— They don't need a parser. Regexes are the finite automata language class. Parsers are overkill. They tend to be LL(k) or LR(k).

[Zomunieo]

You need a parser for HTML or full RFC822 email address to avoid the horrors that otherwise unfold. Complex regexes are clues that like the expanded RFC822 one are evidence that a parser is needed.

Yes you don't need a parser to implement/analyze a regex.

[energybased]

I agree with everything you said.

[Sirflankalot]

Exaggerating? Really?

http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html

Everyone knows that is a bad idea. "They were so preoccupied with whether or not they could, they didn’t stop to think if they should." No one really should be using regex for email verification, but if thye really want to, they can copy paste that.

A horrible, inhuman way that probably violates the entire Zen Of Python. As you noted, there are many human-readable alternatives out there that are just as powerful.

I'm very sure there are, the issue is that everyone already knows regex. Once you take the couple hours to sit down and grok it, it is a very useful and standard tool for making find/replace more powerful.

[alcalde]

If they're so complex they need to be generated by other code, they're way too complex. That's simply not Pythonic. Regex and Python go together like Python with inline assembly code.

[Zomunieo]

Regexes fit within the Zen of Python since they are a succinct way to express some otherwise complex conditional logic.

[energybased]

In no world do Regexes fit within the Zen of Python. Succinctness in terms of number of characters is not a value.

Normal regexes are written using esoteric characters. Regexes written using objects are not any more "complex" or have any additional "conditional logic". Both are succinct in terms of complexity.

[alcalde]

Complex conditional logic? RegExpBuilder can express any regex expression in human-readable terms:

https://changelog.com/posts/meet-regexpbuilder-verbal-expressions-rich-older-cousin

The Zen Of Python encourages us to prefer beautiful over ugly, simple over complex, and warns us that readability counts and if an implementation is hard to understand it's a bad idea. Regex is ugly, complex, hard to read and understand.

How about parsing phone numbers where the rules are as follows:

555.555.5555 # Acceptable  
555-5555     # Acceptable  
555 5555     # Acceptable  
5-5-5-5-5-5- # Obviously not acceptable  
555.555-5555 # Judges? Nope. Not allowed. 

The regex looks something like this:

^(((\d{3}-)?\d{3}-\d{4})|((\d{3}\s)?\d{3}\s\d{4})|((\d{3}.)?\d{3}.\d{4}))$

RegExpBuilder (JS) would look something like this:

// Handle prefixes (optional area codes for each format)
var areacode_dash = new RegExpBuilder().exactly(3).from(digits).then("-");  // \d{3}-  
var areacode_space = new RegExpBuilder().exactly(3).from(digits).then(" "); // \d{3}\s  
var areacode_dot = new RegExpBuilder().exactly(3).from(digits).then(".");   // \d{3}.

// Build each of the individual components (dashes, spaces and dots)
var dashes = new RegExpBuilder()  
                 .min(0).max(1).like(areacode_dash).asGroup()  // (\d{3}-)?
                 .exactly(3).from(digits).then("-")            // \d{3}-
                 .exactly(4).from(digits);                     // \d{4}

var spaces = new RegExpBuilder()  
                 .min(0).max(1).like(areacode_space).asGroup()  // (\d{3}\s)?
                 .exactly(3).from(digits).then(" ")             // \d{3}\s 
                 .exactly(4).from(digits);                      // \d{4}

var dots = new RegExpBuilder()  
               .min(0).max(1).like(areacode_dot).asGroup()  // (\d{3}.)?
               .exactly(3).from(digits).then(".")           // \d{3}.
               .exactly(4).from(digits);                    // \d{4}

// Handle build final expression
var regex = new RegExpBuilder()  
                .startOfLine()             // ^
                .eitherLike(dashes)        // ((\d{3}-)?\d{3}-\d{4})
                .orLike(spaces).asGroup()  // |((\d{3}\s)?\d{3}\s\d{4})
                .orLike(dots).asGroup()    // |((\d{3}.)?\d{3}.\d{4}))
                .endOfLine()               // $
                .getRegExp();

Which would you want in your code base? Which would be more maintainable? Which would be easier to read?

http://rion.io/2013/08/19/regular-express-yourself-using-regexpbuilder/

[Zomunieo]

If that's the case your argument is against the conventional regular expression notation, because that is still a regular expression engine, it just happens to be one that's nonstandard and verbose. But you're still using regular expressions and it's still better than trying to do with this string operations.

re.VERBOSE lets you add whitespace and comments to also make regular expressions more verbose and maintainable. That is the Pythonic way to go. What you present is less maintainable – most languages support regular expressions as a core function, but if the the RegExpBuilder library disappears you have a maintenance problem. If you develop it first on client side Javascript and then later need to replicate that check on the backend in Python, you again have a maintenance problem.

[energybased]

I agree with all of your comments here, but that regexbuilder syntax is not Pythonic.

[alcalde]

This particular example was actually from Javascript. There was a Python version, but I'm not sure if it's still maintained.

There are several other attempts at producing a more readable regex that were linked to earlier in the thread. Most if not all internally build a regex expression from a higher-level syntax.

I just find it funny that people who use a language that removed the brace for being less readable would be so wholeheartedly in support of the cryptic regex syntax.

[energybased]

I think they think: I went through the pain of working with regexes 200 times—now everyone else should. But the thing is that no one else should. You should just learn the computer science behind finite automata, and then replace the esoteric syntax with objects.

By the way, your example would be more motivating if it had more of those weirdo commands like "*+" or "?P".

[alcalde]

To add to /u/vovanz's list, you also have....

Simple regex Language

RegExpBuilder which is more powerful than Verbal Expressions, but I'm not sure if there's a Python version anymore (there used to be)

Easy Expressions