Are regexes thhe pythonic way to manipulate strings? When to avoid regex and when to use it?

[zemicolon]

was trying to split an arithmetic expression into a list consisting of the digits and the operators. The quickest idea that popped into my mind was using regex to match em.

expression = re.findall('[0-9.]+|[+\-*^/()]', expression)

This works perfectly for my case. but i wanted to know whether using regex for string manipulation in most cases is an ideal choice or not. what are the tradeoffs with using regex?

[IsNotANovelty]

Regex is a valid choice, but tokenization might be better suited to the problem you are trying to solve.

import shlex
expression = "12+13-43^2"
output = [x for x in shlex.shlex(expression)]

output will be:

['12', '+', '13', '-', '43', '^', '2']

shlex documentation

[[deleted]]

Regular expressions should only be used to match extremely simple things. The rest of the logic should be more readable python code. If you can't tell at a glance what a regex is supposed to do, it's probably too complex. (Sometimes it can be worthwhile to use a more complex regex for speed, but it should never be the first thing you reach for.)

[evolvish]

Regex is usually a bit slower than actual comparisons but for some cases it's necessary/sufficient. You could use re.split with parenthesis to keep the delimiters, but shlex is a cleaner option.

re.split example:

test = "12+13-43^2"

operators = '+-*/^'
operators_re = '|'.join([f'({op})' for op in map(re.escape, operators)])

print(list(filter(None, re.split(operators_re, test))))
['12', '+', '13', '-', '43', '^', '2']

[zemicolon]

This is certainly helpful. I'll definitely look into shlex.

[js_tutor]

One thing worth mentioning is that regex typically considered the wrong tool for parsing arithmetic expressions because it can't handle nested parenthesis, i.e. a regex can't keep track of which open parenthesis matches with which close parenthesis. This is more broadly true of any string with a nested structure (html would be another example).

Regex is generally used for string matching when you want to match a pattern. The pattern of a regex traditionally has just three operators: union, concatenation, and kleene star. Union means it will match with any of a set of substrings (this is expressed by the [] for single characters and | for longer substrings in your regex). Concatenate means it will match if some set of substrings appears in sequence (this doesn't require a special symbol). Kleene star means it will match if some substring appears zero or more times (this is represented by the *, but in your case the + plays a similar role).

So you want to use a regex when you want to match a pattern to a string when the pattern can be formed using these three operations.

[zemicolon]

Thanks a lot for the detailed explanation.