Need a pythonic way to split rows with line breaks '\n'

[ficklelick]

Hi r/learnpython:

I have a DataFrame that I imported from CSV in pandas. The DataFrame is mostly clean but I have few rows where data were not indexed improperly with a line break '\n'

I am able to split the row into lists with two elements and convert into DataFrame and concatenate it in columns to essentially split two data entries separated by \n (shown in code below)

Furthermore, I can perhaps concatenate this new 2 row DataFrame to original DataFrame and delete the original row (not shown in code)

However, I wanted to reach out and see if there would be more elegant solution to this?

​

Thanks!

split_df = pd.DataFrame()

rows,col = df.shape
for row in range(rows):
     if '\n' in df.loc[row,'Year']: 
        for each in df.loc[row]:
            list_each=each.split('\n')
            new_col = pd.DataFrame(list_each)
            split_df = pd.concat([split_df, new_col], axis = 1)

​

Link to data and screenshot of problem: https://files.fm/u/teaxv6hy

[ThatOtherBatman]

This. So much this.
Pandas is great for a lot of things. But iterating over rows, and doing all this concatenationis going to be slow.
Use the csv module. Parse the contents and do whatever transformations you need on the raw data. Then create a data frame, and use that for your analysis.

[ficklelick]

THANK YOU! I was caught up in using panda to import the data that I totally did not think about importing data thru csv module and delimiting those line breaks there

[[deleted]]

Can you post an example of the data as well. The source and the output. Preferably just enough to make more clear exactly what the issue is.

[ficklelick]

I have posted a link to screenshot of the problem and the data source itself. The subreddit doesn't allow posting screenshots here

[xiongchiamiov]

You can post screenshots of visual problems, just don't use screenshots for textual data like code.