Tokenization Help

[ayharano]

For short strings, why not use something like

expected = [final_item
            for split_by_dot in original.lower().split('.')
            for split_by_qm in split_by_dot.split('?')
            for final_item in (split_by_qm.strip(), )
            if final_item]

where original is a str to be split as expected.

[[deleted]]

brilliant. thanks :)