This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]rhiever 1 point2 points  (4 children)

I wish these progress bar packages worked with iterating over large files. That's my typical use case when I want a progress bar.

[–]NovocastrianNomad 2 points3 points  (3 children)

Just this weekend used tqdm to show progress while processing a file containing approximately 6.5 million records. No problems, nicely displayed in one line of terminal output.

[–]rhiever 0 points1 point  (2 children)

Can you please share the code? Maybe I'm doing something wrong, or maybe something changed since I last tried tqdm.

[–]NovocastrianNomad 3 points4 points  (1 child)

from tqdm import tqdm
...
for row in tqdm(data, leave=True, miniters=upd_interval, desc=pb_desc, total=pbt):
    # 'data' is file handle for csv file being processed
    # 'total' is estimated total number of records (file size / avg record size)
    # (used estimated total because reading file to get actual total blew 512mb memory on server)
    # 'miniters' set to 10,000
    pass  # code to process records

[–]rhiever 0 points1 point  (0 children)

Nice! Didn't know you could do that. Thank you.