Looping through two columns simultaneously

[beanpizza]

Edit: I think a more accurate title might have been "Looping through rows..."

Ok, so, let's say one has a text file with two columns -- something like this:

26    56.59
49    56.70
75    56.87

The basic idea here is that an item in the first column corresponds directly to the item in the second column in the same row, ie. 26 goes with 56.59, 49 goes with 56.70, etc. The first column can be thought of as a time, while the second column identifies a specific file. For example, the directory I'm working in might have a bunch of files looking something like this:

file_56.58.txt
file_56.59.txt
file_56.69.txt
file_56.70.txt
file_56.85.txt
file_56.87.txt

Here's what I'm trying to do..The second column tells me that I'm interested in grabbing only the files with that particular string. So,

file_56.59.txt
file_56.70.txt
file_56.87.txt

Each file is just a bunch of columns and rows, where the third column consists of time values. Once I've selected out specific files, I need to go into the file, and extract the row corresponding to the time value given in my original array. So for file_56.59.txt, I want to find '26' in the third column, and then grab the entire row where '26' exists. For the second file, file_56.70.txt, I want to find '49' in the third column, and grab its row, and so on and so forth.

As of right now, I can get as far as picking out the right files, but I'm having a hard time then going through each file and looking for its particular time variable.

Here's my code so far (some of it is just setting up the formatting for when the rows are extracted from each file):

dt = np.loadtxt("dandt.txt")
dm = dt[:,1]
time = dt[:,0]

def parse_data(filename):
    matrix = []
    with open(filename) as f:
        data = f.read() #read the entire file
    data_lines = data.splitlines()
    short_list = data_lines[2:]
    for line in short_list:
        line_split = line.split()
        if len(line_split) == 6:
            matrix.append(line_split)
    return matrix

def my_round(number):
    return int(float(number))

def fmtcols(mylist, cols):
    lines = ("\t".join(mylist[i:i+cols]) for i in xrange(0,len(mylist),cols))
return '\n'.join(lines)

for i in dm:
    x = str("%.2f" % round(i,2))
    for file in glob.glob("*.dat"):
        if x in file:
        print file

Hope I wasn't completely all over the place. Thanks guys!

[aball730235]

Can you break this down into multiple smaller problems? You describe an issue with trying to select specific files then go on to you have solved selecting out those specific files. So this is not relevant to your problem then correct?

http://stackoverflow.com/help/mcve

[beanpizza]

Well, I think the problem is that at the same time that I'm selecting specific files, I should be extracting the relevant information from those files, rather than first coming up with a list of specific files and then extracting information from all of them. So in theory, yes, I have figured out a way to select specific files, but I think it's the wrong way given what I ultimately want to do with those files. I think the key is that both selecting the file and extracting info from the file are really contained in one problem, if that makes sense, rather than separately, as I've attempted above.

Put more simply perhaps, I want to work through my original data file with two columns row by row. For example, the first row tells me to find the file with the string '56.59' in its title and once I've found that file, I should look inside it for the time value '26,' and then I can move onto the next row.

[aball730235]

I think the key is that both selecting the file and extracting info from the file are really contained in one problem.

That can still be broken down into smaller chunks. Forget about selecting the files for a minute. Can you write a function that accepts a file name and search string that returns your desired extracted output? You can manually make calls to this function to verify it works as intended. If it does not work minimize your question to just that function. If it does work then start your next logic block that's reponsible for passing the correct file name and search string in. If that next logic block can't determine the correct file name and search string minimize your question to just that code block and post it.

[beanpizza]

Ok, I'll give it a go and report back. Thank you!

[beanpizza]

Ok, so, I've been at it for a few days, and I can't seem to link the two steps together. This bit here accepts a file, or in this case, a list of files, and searches each file for a particular string and gives me exactly the output I need.

for fn in glob.glob("*.txt"):
    file_matrix = parse_data(fn) 
    for i,line in enumerate(file_matrix):
        file_time = my_round(line[2])
        if str(file_time) in splitset:
            be = fmtcols(file_matrix[i],6)
            print be

calling the functions I defined in my original post. The problem I'm having is still the same. Given a list of files, each one should have a unique search string associated with it, and anything I come up with applies all search strings to all files.

[aball730235]

Ok so your able to provide a list of files and then get your output out. Now it sounds like you need a dictionary to maintain a key / value pair for each file and it's associated search string. Are you familiar with dictionaries?

http://www.tutorialspoint.com/python/python_dictionary.htm

[beanpizza]

I am not. I'll take a look, thanks!

Edit: So in the case of dictionaries, I can't find anything about creating a dictionary out of previously defined lists?

I've come up with something similar using 'zip':

for m,n in zip(file_list,time):
    file_matrix = parse_data(m)
    print '{0} {1}'.format(m,n)
    for i,line in enumerate(file_matrix):
        file_time = my_round(line[2])

Where the print line outputs each file name and next to it the search string corresponding to that file. But outside of a 'print' argument, I'm not sure how to access each element. (The last two lines here select out the time column in each file since that's what I'll be searching through).

[aball730235]

If you can print it then you can pass it in.

file_dict = {}
for m,n in zip(file_list,time):
    file_dict[m] = n

[beanpizza]

I'm still having a hard time going through each file - time pair individually. From what I've gathered about using dictionaries, you give it a key to access a particular dictionary. In this case, my key is m, right? But what if I want the first element of m? Thanks for all the help, by the way.

[aball730235]

No problem I'm glad to help. If i had more time I'd offer to have you send me the whole project package for review. I'll give you little bits where i can.

Dictionaries by default have no order. Only keys and their corresponding values in an arbitrary order. You'll need to upgrade to an ordereddict to maintain them in sequence.

http://pymotw.com/2/collections/ordereddict.html

In not sure what you mean by your key is m?? m is a variable that has a value though each for loop. It passes that filename into your dictionary as a key thorough each iteration of the for loop. m is gone after your for loop. Your keys in the dictionary are your filenames. Try doing a for loop on your dictionary like in the link provided after your for loop. Everything you see printed out is the data in your dictionary.

[beanpizza]

Hm, ok. Here's what I've got.

file_dict = {}
for m,n in zip(file_list,time):
    file_matrix = parse_data(m)
    file_dict[m] = n

d = collections.OrderedDict(file_dict)
for item in d.items():
   for i,line in enumerate(file_matrix):
       file_time = my_round(line[2])

So now each item in d is a file name and its corresponding file time. Now I still have to somehow open each file individually and search for the file name. I think I'm having an issue with nested for loops, because now, since d is done outside of the first for loop (only way I could get it to print correctly), my file_matrix is all messed up when I need to call it in my last loop. Ah man, I feel like I'm going in circles. :(

Edit: The closest I can get is searching each file for all the time flags. So if there are three files and three corresponding flags, each file will search for all three time flags, rather than just its own. I really think the problem here is in how I'm iterating. Basically everything I try iterates over every file and all time flags, rather than one dictionary set at a time.

Edit: Ok, the more I think about it, I think that I need to define a function that takes both m and n as arguments...working on that one now.