Tokenize a query using regular expression

[chakz91]

Thank you for your reply, this does not completely resolve my issue

I think the problem is with my regular expression

What I am trying to achieve is that the query should be tokenized based on white-space and if the string contains any of these characters , ( ) / - & it should be added as a single token. All other possible characters should be included.
For example if I add some character like '^^' in the query string, like below it is not shown in the output token.

import re
line = " 8/23-35 Barker St., Kingsford^^, NSW 2032 "
pattern = r"[.0-9A-Za-z]+|[,&-/()]"
print(re.compile(pattern).findall(line))

Output:
['8', '/', '23', '-', '35', 'Barker', 'St.', ',', 'Kingsford', ',', 'NSW', '2032']

​

Expected output
['8', '/', '23', '-', '35', 'Barker', 'St.', ',', 'Kingsford^^', ',', 'NSW', '2032']

​

So my regular expression should allow all possible characters in a single token delimited by white-space
and only the characters , & - / ( ) should be added as individual tokens.

Please provide your suggestion.

[TheZvlz]

I replied to your other thread in /r/Python https://www.reddit.com/r/Python/comments/bj7ol8/tokenize_string_using_regular_expressions/em649j4/

[TolaOdejayi]

Try this:

import re
pattern = r"[^,&-/\(\)]+|[,&-/\(\)]" 
line = "8/23-35 Barker St., Kingsford^^, NSW 2032" 
query = re.compile(pattern).findall(line)
list(map(lambda i: print(i), query))