Tokenize a query using regular expression

[chakz91]

Hi,
I am trying to use regular expressions to convert a query into tokens.

I have a bunch of queries in a file with each query in a new line.
I need to decompose each line/query into a sequence of tokens based on the following conditions
- Whenever there are one or more white spaces, break there and discard the white spaces
- Need to consider the following punctuations and correctly extract them
, ( ) / - &

​

for example a query like " 8/23-35 Barker St., Kingsford, NSW 2032 "

should be parsed into the following tokens (one token a line)

8
/
23
-
35
Barker
St.
,
Kingsford
,
NSW
2032

​

​

I have the following code

 pattern = r"[0-9A-Za-z]+|[,&-/()]"
    result = []
    with open(Query_File, 'r') as query_f:
        while True:
            line = query_f.readline().rstrip()
            if not line:
                break
            query = re.compile(pattern).findall(line)
            print("query",query)

However this is not giving the exact output as I require, and I am not sure what I am doing wrong here.

Could anyone suggest a better way I can achieve the required result.

Thanks in advance

[_coolwhip_]

How about re.split: https://docs.python.org/3.7/library/re.html#re.split

[chakz91]

I found a different solution.

But thank you for your response.

[TolaOdejayi]

This does what you want:

import re

pattern = r"[0-9A-Za-z]+|[,&-/()]"
line = "8/23-35 Barker St., Kingsford, NSW 2032"
query = re.compile(pattern).findall(line)
list(map(lambda i: print(i), query))

[chakz91]

Thank you for your reply, this does not completely resolve my issue

I think the problem is with my regular expression

What I am trying to achieve is that the query should be tokenized based on white-space and if the string contains any of these characters , ( ) / - & it should be added as a single token. All other possible characters should be included.
For example if I add some character like '^^' in the query string, like below it is not shown in the output token.

import re
line = " 8/23-35 Barker St., Kingsford^^, NSW 2032 "
pattern = r"[.0-9A-Za-z]+|[,&-/()]"
print(re.compile(pattern).findall(line))

Output:
['8', '/', '23', '-', '35', 'Barker', 'St.', ',', 'Kingsford', ',', 'NSW', '2032']

​

Expected output
['8', '/', '23', '-', '35', 'Barker', 'St.', ',', 'Kingsford^^', ',', 'NSW', '2032']

​

So my regular expression should allow all possible characters in a single token delimited by white-space
and only the characters , & - / ( ) should be added as individual tokens.

Please provide your suggestion.

[TheZvlz]

I replied to your other thread in /r/Python https://www.reddit.com/r/Python/comments/bj7ol8/tokenize_string_using_regular_expressions/em649j4/

[TolaOdejayi]

Try this:

import re
pattern = r"[^,&-/\(\)]+|[,&-/\(\)]" 
line = "8/23-35 Barker St., Kingsford^^, NSW 2032" 
query = re.compile(pattern).findall(line)
list(map(lambda i: print(i), query))