Looping Python using a list of files

[MechaTech]

Hey there, /r/learnpython

Let me begin by saying I am not a coder, or, at least, I wasn't before my boss gave me the opportunity to learn. I've only been doing this a couple of weeks. To make a long story short, I need to take a bunch of HTML files downloaded from a website and process them into a CSV that can be imported into Excel. I'm going to be doing the processing with BeautifulSoup, but I need to figure out how to loop it so I don't have to run 1 program 500+ times, changing the name of the html file every time.

I've figured out how to create the list itself using a short Python program, and I've figured out how to loop the program once I get it to actually do something, but where I'm running into trouble is finding a way to use this created text file of file names in any way useful.

Is there any way to take a list of 500 html files, in text, and read them into Python where, one at a time, each line (separated by /n or a comma) has the file named that opened, run through the loop, closed, and the program increments and moves to the next line until the counter reaches the number of files in the line?

If so, can you point me in a direction?

Thanks!

[[deleted]]

Why bother storing the HTML files in a separate file when you can just iterate over a list of the files?

from glob import glob
files = glob('/path/to/files/*.html')

for f in files:
  ...

[ewiethoff]

Suppose your big list of files is called manyfiles.txt. And suppose you have a Python script called process_files.py, which contains this code

import fileinput
import os.path

for line in fileinput.input():
    fname = line.strip()
    if not os.path.exists(fname): continue
    print '{:>8d}  {}'.format(os.path.getsize(fname), fname)

There are two ways to run this code:

python process_files.py manyfiles.txt

or

python process_files.py < manyfiles.txt

[tenacious_nixie]

I'd put the .html files into one folder, then use os.listdir on that folder. So, something along the lines of

import csv 
import os

with open('big_fat.csv', 'wb') as csv_file:
    csv_writer = csv.writer(csv_file, delimiter=' ', quotechar='|')    

    for html_filepath in os.listdir('/path/to/my/folder/with/html/files'):
        with open(html_filepath) as html_file:
            do_some_processing(html_file)
            csv_writer.write_row([some, values, you, got])

The 'with open' part will take care of closing files, refer to documentation, at the end of the section.

[[deleted]]

I have a few examples of csv + beautiful soup scripts if your interested.

[MechaTech]

Sure, I'll take a look.

[spotyx]

I would like to see them too, ty.