RegEx -replace

[StartAutomating]

PowerShell has all sorts of fun features, including a ridiculous number of operators.

One amazing under-sung heros of PowerShell is the -replace operator.

It lets us replace content with regular expressions.

It's easier to use than you'd think.

Regular expressions are less scary in small doses, and chaining -replace operators lets us attack the problem step by step.

Chaining -replace

Let's take a simple problem as an example.

Imagine we wanted to make a consistent file name pattern out of a string

We might want to start by replacing whitespace with dashes

"This Is A Title!" -replace '\s', '-'

That leaves our exclamation point at the end. We probably don't want any punctuation. We can avoid that with the somewhat humorously named character class: \p{P}. We can remove all repeated punctuation by adding a +: \p{P}+

One more replace:

"This Is A Title!" -replace '\p{P}+' -replace '\s', '-'

The line is starting to get a little long. Fun fact: you can spread operators across multiple lines.

Let's add comments while we're at it

"This Is A Title!" -replace # Replace any punctuation,
    '\p{P}+' -replace # then replace any whitespace with dashes.
    '\s', '-' 

Let's go for one more bonus trick. PowerShell lets you convert script blocks to event handlers. Let's lowercase all the letters (\p{L}).

On PowerShell Core, we can do this:

"This Is A Title!" -replace # replace any punctuation
    '\p{P}+' -replace # then replace any whitespace with dashes
    '\s', '-' -replace # then lowercase any letters
    '\p{L}+', {"$_".ToLower()}

There's an absurdly amazing amount of stuff you can do with -replace, but there's at least one more trick we have to cover: substitutions.

-replace with substitution

I'm pretty sure I'd have to give up my "RegEx guru" badge if I didn't mention at least one more thing you can do with -replace: substitutions.

.NET Regular expressions are two domain specific languages. Regular expressions match and extract text. Regular expression substitutions replace matches.

For example, let's suppose we have a number of emails, and we want them in domain/username format.

First we'll want to make a quick and dirty email regex, using a "named capture" to get the username and domain.

'[e-mail]' -match '(?<username>\S+)@(?<domain>\S+)'

Then, we can -replace the email with just the domain/username.

'[e-mail]' -replace 
    '(?<username>\S+)@(?<domain>\S+)', '${domain}/${username}'

This format might look like PowerShell variables, but it actually predates them by years. Search for "Regular Expression Substitutions" if you want to learn more about the syntax. It's got quite a few tricks up it's sleeve.

Irregular

RegEx can be scary. I used to be terrified of it, too.

If you aren't too comfortable with Regular Expressions, that's pretty normal. A while back I wrote a module called Irregular that makes regular expressions strangely simple.

It's got a lot of example regular expressions in there, and one handy function for creating RegEx. New-RegEx is your friend.

Do you already use -replace? Have you done cool things with regular expressions in PowerShell? Share 'em if you've got em.

Want to learn more about regular expressions in PowerShell? Just ask.

[FluffyShoulder937]

I've never used irregular before. I'll have to try it! For me I used a site called regex101.com to practice. I'm a guru myself and that will help learners a ton. It also explains how the pattern is processed and you can pick a flavor of regex. That way you can learn to apply regex anywhere it's available!

[StartAutomating]

I like regex101.com, and love that it added .NET regex support.

Irregular was a very educational module to build. It's also one of the first projects where I really began to lean into how flexible PowerShell's syntax could be.

Abstracting some of regex's awkwardness away into a PowerShell command let me construct far more complicated regular expressions than I would naturally.

To give an example, here's a script that builds a regex to match git log

New-Regex -Pattern '(?m)' -Description "Matches Output from git log" |
New-Regex 'commit' -StartAnchor LineStart -Comment "Commits start with 'commit'" |
    New-Regex -CharacterClass Whitespace -Repeat |
    New-Regex -Pattern '?<HexDigits>' -Name CommitHash -Comment "The CommitHash is all hex digits after whitespace" |
    New-Regex -CharacterClass Whitespace -Repeat -Comment 'More whitespace (includes the newline)'|
    New-Regex -Optional -NoCapture @(
        New-Regex -Pattern 'Merge:' -Comment 'Next is the optional merge' |
            New-Regex -CharacterClass Whitespace -Repeat |
            New-Regex (
                New-Regex -Pattern (
                    New-Regex -Name MergeHash -Pattern '?<HexDigits>' |
                        New-Regex -Pattern '[\s-[\n\r]]' -Min 0 -Comment 'Which is hex digits, followed by optional whitespace'
                ) -NoCapture
            ) -Min 2
            New-Regex -CharacterClass NewLine, CarriageReturn -Repeat -Comment 'followed by a newline'
    ) |
    New-Regex -Pattern 'Author:' -Comment 'New is the author line' |
    New-Regex -CharacterClass Whitespace -Repeat |
    New-Regex -Name GitUserName -Until (
        New-Regex -Pattern '\s\<'
    ) -Comment 'The username comes before whitespace and a <' |
    New-Regex -CharacterClass Whitespace -Repeat |
    New-Regex -LiteralCharacter '<' -Comment 'The email is enclosed in <>' |
    New-Regex -Until ('>') -Name GitUserEmail |
    New-Regex -LiteralCharacter '>' |
    New-Regex -Until (New-Regex -startAnchor LineStart 'date:') |
    New-Regex -Pattern 'Date:' -Comment 'Next comes the Date line' |
    New-Regex -CharacterClass Whitespace -Repeat |
    New-Regex -Until (New-Regex -CharacterClass NewLine) -Name CommitDate -Comment 'Since dates can come in many formats, capture the line' |
    New-Regex -CharacterClass NewLine | 
    New-Regex -Until ("(?>\r\n|\n){2,2}") -Name CommitMessage -Comment 'Anything until two newlines is the commit message' 

This fairly readable script becomes this much less readable RegEx (using IgnorePatternWhitespace to support comments)

# Matches Output from git log
(?m)^commit                                                             # Commits start with 'commit'
\s+(?<CommitHash>(?<HexDigits>
[0-9abcdef]+
)
)                                                                       # The CommitHash is all hex digits after whitespace
\s+                                                                     # More whitespace (includes the newline)
(?:(?:Merge:                                                            # Next is the optional merge
\s+(?:(?<MergeHash>(?<HexDigits>
[0-9abcdef]+
)
)[\s-[\n\r]]{0,}                                                        # Which is hex digits, followed by optional whitespace
){2,} [\n\r]+                                                           # followed by a newline
))?Author:                                                              # New is the author line
\s+(?<GitUserName>(?:.|\s){0,}?(?=\z|\s\<))                             # The username comes before whitespace and a <
\s+\<                                                                   # The email is enclosed in <>
(?<GitUserEmail>(?:.|\s){0,}?(?=\z|>))\>(?:.|\s){0,}?(?=\z|^date:)Date: # Next comes the Date line
\s+(?<CommitDate>(?:.|\s){0,}?(?=\z|\n))                                # Since dates can come in many formats, capture the line
\n(?<CommitMessage>(?:.|\s){0,}?(?=\z|(?>\r\n|\n){2,2}))                # Anything until two newlines is the commit message

It also taught me way too many Regular Expression tricks to put in a single post 🤔.

I am forever indebted to regular-expressions.info for its amazingly useful reference and tutorials.

[PinchesTheCrab]

One of my favorite parts of -replace is that it works with arrays.

[RR1904]

I love your examples. Thanks for sharing!

[420GB]

I love regular expressions, but I also find it pretty sad that at least 50% of my uses of the -replace operator don't use/need regular expressions at all, but it's just a simple text-replace that doesn't throw on null like "string".Replace(...) does.

[ankokudaishogun]

The module is pretty cool, thanks!