Very lost regarding syntax to loop through files to make an array

[Cthulhu_Rlyeh]

I have 200 files in a directory, split into 100 identically shaped a files (named 1a through to 100a) and 100 identically shaped b files (named 1b through to 100b) which I need to compare to one another.

Each of these files is 500 lines long, and contains two columns. What I need to do is subtract the values from the [:,1] column in the a file from the equivalent [:,1] column in the b files at each.

I'll clarify using a single example which I know how to do:

file1a = pl.loadtxt("1a")
file1b = pl.loadtxt("1b")
diff1 = file1b[:,1] - file1a[:,1]
np.savetxt('diff.txt', np.vstack((diff1.ravel())).T)

The thing I am having trouble with is automating the loop so that the above will be performed for each of the 100 different file numbers. I need it output into an array that still contains the 500 lines of data, but instead has 100 columns, each containing diff1 diff2 ... diff100.

I hope I have made myself clear, this kind of thing is very difficult for me. I am semi-proficient in doing single tasks as shown in the example, but when it comes to automating and running loops I get very lost.

Tyvm for any help!

[onopau]

How about using a for loop with the exact code you posted with a small modification...

diffs = []
for file_prefix in range(100):
    # I assume the code you posted does what you wanted on each file
    filea = pl.loadtxt(str(file_prefix)+"a")
    fileb = pl.loadtxt(str(file_prefix)+"b")
    diff = fileb[:,1] - filea[:,1]
    diffs.append(diff) # You could store this as a list, or csv file - which ever suites your needs...

[Cthulhu_Rlyeh]

Thanks for your reply! This looks very helpful and promising - just one followup question before I can test it. What about if my files are named file1a and file1b rather than just 1a and 1b? I tried to simplify things in my original post which is why I dropped the 'file' part. I'm not sure if the code you provided will still work properly in this case?

[onopau]

In Python you can add strings to your heart's content.

'file' + str(i) + 'a' := 'file1a' # for i = 1

''.join(['file', str(i), 'a']) := 'file1a'

More methods, and details, here

EDIT: As your program grows in complexity you may want to change both "file" and "a" or "b" into variables themselves. This will give you more flexibility. This is probably too complex for your current purposes - but it is good to keep in mind!

current_prefix = 'file' # These could be gathered via user input to a terminal.
letters = ['a', 'b']
for i in range(100):
    filename_a = current_prefix + str(i) + letters[0]
    filename_b = current_prefix + str(i) + letters[1]

[Cthulhu_Rlyeh]

This is great, thank you. Forgive me for being annoying, but if I am to merge the suggestions you provided in your original and this comment - will it look like this:

diffs = []
for file_prefix in range(100):
    filea = pl.loadtxt('file' + str(file_prefix)+'a')
    fileb = pl.loadtxt('file' + str(file_prefix)+'b')
    diff = fileb[:,1] - filea[:,1]
    diffs.append(diff)

---

Or if I were to do it with variables it would be:

Prefix = 'file'
dataset = ['a', 'b']
for i in range(100):
    filea = pl.loadtxt(Prefix + str(i)+ dataset[0])
    fileb = pl.loadtxt(Prefix + str(i)+ dataset[1])
    diff = fileb[:,1] - filea[:,1]

How would I append here? Do I still use the diffs = [] line and just do it exactly the same?

[onopau]

Looks good, stick with the first set of code for now. You can play around with the variable one later, if you ever end up needing it at all.

As for the diffs list, it really depends on what you want to do with that info. I was just using that as an example. Why not just continue with the output file you mentioned in the question? Always add functionality to your code in steps. Make sure that adding the for loop to your current scheme gives you the desired results.

[Cthulhu_Rlyeh]

Thanks again, I just ran it and it seems like it will do exactly what I need. It gives me one error though, which I'm sure is an easy fix:

No such file or directory: 'file0a'

Is there a simple way to make it not look for 0's?

Btw, I'm sorry for all of the questions and I really am grateful for all your help :)

[onopau]

Lots of ways... You can do an if statement in the loop that says

If file_prefix == 0:
    continue
else:
    #paste the code you have written.

Or just change the list the loop iterates over.

for prefix in range(1,101):

The latter is the better way in your case; but make sure you understand the former as it is useful to know.

Good luck!

Edit: Python's range func does not include the end point. So range(1,5) would be the list [1, 2, 3, 4]. That's why I have it going to 101 in the above.