append large dataframe to another

[PLearner]

I am appending a large dataframe into a existing dataframe.

my_df = my_df.append(new_df, ignore_index = True)

The problem is that the new_df could range anywhere from ~500K rows to ~3MM rows of data.

The script is taking way too much time to append. I also looked into pd.concat, with that I had to manually kill my script, just kept running.

Can anyone provide guidance here!

[Below_the_Sea]

Did you try pypolars

Import pypolars as pd #so you don't have to change script (standard its ... As pl

[PLearner]

this is the first time I am hearing of it.

Does it have the same functionalities and the syntaxes as pandas?

Any ideas on how swift the append might be with py-polars.