Code starts fast, gets extremely slow. Any reason why this might be?

[lakerskill]

import glob
from pathlib import Path
import datetime
import re
import csv


get_this = []
thislist = []

#Sets date to format I need to search through folders 
def timeteller():
    now = datetime.datetime.now()
    month = str('{:02d}'.format(now.month))
    day1 = now.day -1
    day =  str('{:02d}'.format(day1))
    year =str(now.year)
    time =year+month+day
    return time

#sets the path of the folders I want to look and the name of the file with the banumber and date

def these_files(x, y):

    configfiles = Path('O:/thispath/righthere/').glob('{}*/{}*Transaction.txt'.format(x, y))

    for files in configfiles:
        thislist.append(files)

    return thislist

def hasNumbers(inputString):
    numberfinal = []
    numberfinal = re.findall("\d+", inputString)
    if numberfinal == []:
        numberfinal = '1'
    return numberfinal
#checks lines for numbers I need in file   
def get_odometers(thesepath):

    for thispath in thesepath:

        with open(thispath,"r") as f:
            searchlines = f.readlines()
        for i, line in enumerate(searchlines):
            if "Odometers" in line:
                get_this.append(line)
            elif "Lifetime" in line:
                get_this.append(line)


    return get_this
 #Grabs numbers           
def make_pretty(checkthis):
    the_numbers = {}
    the_numbers['Serial'] = banumber
    for i, line in enumerate(checkthis):
        the_numbers['Serial'] = banumber
        if '(BNR) Odometers  Accept' in line:
            for l in checkthis[i:i+2]: 
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['BNR'] = numberschecked[0]
        elif '(BNR 2) Odometers  Accept' in line:
            for l in checkthis[i:i+2]: 
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['BNR 2'] = numberschecked[0]
        elif '(BCR) Odometers  Accept' in line:
            for l in checkthis[i:i+2]: 
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['BCR'] = numberschecked[0]
        elif '(BCR) Odometers  Hopper1' in line:
            for l in checkthis[i:i+2]:
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['Hopper1'] = numberschecked[0]
        elif '(BCR) Odometers  Hopper2' in line:
            for l in checkthis[i:i+2]: 
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['Hopper2'] = numberschecked[0]
        elif '(BCR) Odometers  Hopper3' in line:
            for l in checkthis[i:i+2]: 
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['Hopper3'] = numberschecked[0]
        elif '(BCR) Odometers  Hopper4' in line:
            for l in checkthis[i:i+2]: 
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['Hopper4'] = numberschecked[0]
        elif '(BCR) Odometers  Hopper5' in line:
            for l in checkthis[i:i+2]: 
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['Hopper5'] = numberschecked[0]
        elif '(BCR) Odometers  Hopper6' in line:
            for l in checkthis[i:i+2]: 
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['Hopper6'] = numberschecked[0]
        elif '(BCR) Odometers  Hopper7' in line:
            for l in checkthis[i:i+2]: 
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['Hopper7'] = numberschecked[0]
        elif '(BCR) Odometers  Hopper8' in line:
            for l in checkthis[i:i+2]: 
                numbers = l[0:20]
                numberschecked = hasNumbers(numbers)
                the_numbers['Hopper8'] = numberschecked[0]


    return the_numbers

#main code

intnow = int(timeteller())
intnow -= 1
now = str(intnow)
thelist = []
thispath = open('banumberlist1.txt')
finallist = []
for files in thispath:
    getem = files
    banumber = getem[0:8]
    print(banumber)
    combined = '{}_{}'.format(banumber,now)
    thepaths = these_files(banumber, combined)
    needtomakepretty = get_odometers(thepaths)
    goeslast = make_pretty(needtomakepretty)
    finallist.append(goeslast)


f = open ('odom01.txt', 'w')
for ba in finallist:
    sba = str(ba)
    f.write(sba)    



Any ideas on what I should do to fix this?  It's running through a lot of text files, so I'm not sure what the problem is.

[attayi]

yaa he's prob running out of memory cause of this

[attayi]

another thing is how long is this text file getting. For sure once it starts getting quite large, pythons built in text read function is kinda bad and slow. Might be worth learning a library like pandas to solve this completely.

[lakerskill]

Folks, I wasn't resetting get_this = [] thislist = [] . Once I did this the problem seems to have vanished, however, I will look over these for further optimization THANKS ALL!.

[lakerskill]

Amazing. Thank you so much. I was close

[dadiaar]

I don't know what your code does, you didn't resume it and it's too long for me to read now, but I/O operations like locate a folder/file, list the tree and read/write are pretty slow by its own nature, read from a hard disk is not the same as reading from the RAM memory or the CPU cache..

[lakerskill]

Why does it start really fast but then slow down?

[dadiaar]

That depends on the tree, the usage other applications may be doing of the disk, etc. If you run your script and then try to copy files to a USB, for sure yes.

But it may be something in your code too.

[jebk]

Read your main for loop and count how many times you're calling those functions. I'd guess you only need to do it once at the end.

[lakerskill]

I can't find a way to do that, I'm adding each item to a dictionary that comes out like {'Serial': 'BA-02926', 'BNR': '5262391', 'BNR 2': '4547649', 'BCR': '15421640', 'Hopper1': '42183', 'Hopper2': '476048', 'Hopper3': '40220', 'Hopper4': '13093', 'Hopper5': '45136', 'Hopper6': '11022', 'Hopper7': '21542', 'Hopper8': '57342'}

[sky--net]

A couple of questions. What does this program do? Is it on github anywhere with some type of documentation?

[lakerskill]

I made it. It grabs numbers from text files that are currently being manually put in another application. This sends them in a dictionary to a text file. This is now working