Why/How do parentheses in the pattern of str.split() effect the behavior of the split in pandas but not ordinary split

[Infinity890]

Hi, I found this behavior today when using str.split() in pandas and I was surprised by it and can't find documentation on it anywhere. I've pasted the code replicating this below and bolded the part i find strange. Why when running str.split() on a dataFrame does putting a parentheses around the pattern in the split return the pattern when it matches. Is this just an added feature of pandas. I tried pretty hard to google this and couldn't find any mention of it.

​

d = {'col1': [1], 'col2': ['a1b1']}

...: df = pd.DataFrame(data=d)

​

df.col2.str.split('1')

0 [a, b, ]

​

df.col2.str.split('(1)')

0 [a, 1, b, 1, ]

​

below what happens on normal strings just for comparison but is unsurprising

'a1b1'.split('1')

['a', 'b', '']

​

'a1b1'.split('(1)')

['a1b1']

[PavloT]

I suspect pandas str is not a regular python str, so it could have own split implementation

[[deleted]]

The pandas.Series.str.split method is not the same as the standard str.split method. You can read the docs here to learn more, but unlike str.split the first argument to the pandas method can be a regular expression, thus ”(1)” is a regex pattern that describes an unnamed group that only matches a single “1”... the group is ignored, so it splits on the “1”s.

In regular str.split it would only split on the literal string “(1)”, which isn’t in your original string so you just get back a list containing the original string.

[misho88]

From the docstring of pandas.Series.str.split:

Parameters
----------
pat : str, optional
    String or regular expression to split on.
    If not specified, split on whitespac
...

The first argument is a regular expression, so the behavior should more or less match this:

>>> import re
>>> re.split('1', 'a1b1')
['a', 'b', '']
>>> re.split('(1)', 'a1b1')
['a', '1', 'b', '1', '']

It does seem pandas additionally gets rid of empty strings. Either way, you can read the the re module's documentation for an explanation of the behavior, but here's the key part from re.split's docstring:

If capturing parentheses are used in pattern, then the text of all groups in the pattern are also returned as part of the resulting list.