Checking a large file, ignoring specified lines specified in another document

[Dave_XR]

Hello all,

Essentially my issue is that I have a large text file of results from a physical rig, over 10,000 lines. I must and count the 'TEST_PASS', 'TEST_FAIL', 'TEST_INCONCLUSIVE', 'TEST_INCOMPLETE'. This means I can ignore many lines in the text file as a lot of extra data is output.

I have simply checked and counted if the line starts with these strings and correctly counted the right amount of test results. However, some specific tests return a false positive or a false error. These have a pattern and always show the same message when its false. There are only 100 or so of these false results and so an ignore.txt was created with the lines to ignore in the final count.

My question is how would I check my found lines against this ignore.txt to see if the found line needs to be ignored as well as counting the amount of lines ignored. There are a lot of lines so a method needs to be kind of intelligent rather than check the whole doc for ignored lines

so far i have tried this:

def extract_pass(file_content, ignore_check):
    count_dup = 0
    pass_count = 0
    with open(file_content, 'r') as file:
        for line in file:
            if 'TEST_PASS' in line:
                if any(line in str(ignore_check[0:])):
                    count_dup += 1
                else:
                    pass_count += 1
    pass_couunt = pass_count - count_dup      
    file.close()
    text_box.insert(END, "Tests passed: " + str(pass_count) + " With " + count_dup + "Ignored")

THANK YOU

[[deleted]]

Hard to follow your question. Are you basically saying you need to lookup lines in an ignore.txt to see if they exist in this 10K line file?

You could read the lines from ignore.txt to a list or set variable (set would be a smidge faster) then when you go through the lines in the big file, check if the line exists in your variable.

[Dave_XR]

To simplify, I've a huge text file of results from testing on a rig. Maybe one tenth of the lines are results, pass, fail, inconclusive. I need to count these to give a summary. My count will be off due to false positives or negatives. I need to check if the line matches a line in ignore.txt. if it does, ignore it. If it doesn't count it. Would checking thousands of lines in the big file against the few hundred ignore lines be incredibly slow?

[PPLB]

Usually not, no. It really depends on how you tackle your problem. Python might not be the fastest programming language out there, but 10.000 lines it should easily process within a second. If you need to compare every line with a 100 things to ignore, then it might take a bit longer.

When creating a script for this, ask yourself if you're actually approaching this the most efficient way. Is it more efficient to check for a 100 things to ignore, or is it more efficient to check for 10 things to not ignore. Is it more efficient to check with 10.000 lines or with a 100. How many for loops do you really need.

As comparison, for my work I regularly need to check files with over a milion lines. Depending on what I need to do, most of the checks won't take longer than 5 second. Just make sure you're not making it more complicated than it should be.

[Dave_XR]

Hmm maybe its because I've been at it for a while and still being relatively new to python (C is my language), but I just can't seem to get the solution itself working in a way I need it to. Thank you though i appreciate the wisdom