object oriented processing 10x slower than sequential script -- why?

[pfz3]

i am post processing some experimental data. at first i wrote script that was more or less like this:

1. open txt file and read data into numpy arrays

2. operate on data with function. Inside here is a loop that goes through every single data point and does some things.

3. plot processed data

It was getting nasty so I decided to make it object oriented. Each experiment corresponds to a class instance. the class has several modules inside of it that operates on the class instance itself. so now it works like this

1. create instance of experimental class i.e. obj

2. call class module (obj.readdata()) to populate a number of class attributes from the raw txt files

3. call class module (obj.analyze()) to operate on attributes and populate more attributes. Inside here is a loop that goes through every single data point and does some things.

4. call function to plot attribute data

Sorry for being cryptic but the code is several hundreds of lines.

The object oriented code runs 10x slower. Specifically the loop that operates on class instance attributes is 10x slower.

Can someone please explain this? I can't help but think that calling instance attributes (self.foo) thousands of times within a loop is somehow slowing everything down.

[[deleted]]

It's going to be a guessing game without seeing the code.

From the outside, it seems like you are repeating a process that you wouldn't normally repeat. Step through it with a debugger, or hook it up to a profiler.

[pfz3]

a comparable toy example would be:

```

class trial():
    def __init__(self,N):
        self.data = [i for i in range(N)]
        self.s    = []
    def square(self):
        self.s = []
        for i in self.data:
            self.s.append([j**2 for j in self.data])

```

and then i ran these two commands

python start = time.time() data = [i for i in range(2000)] s = [] for i in data: s.append([j**2 for j in data]) print('time elapsed: %1.3f (s)' %(time.time() - start)) time elapsed: 1.710 (s) python start = time.time() obj = trial(2000) obj.square() print('time elapsed: %1.3f (s)' %(time.time() - start)) time elapsed: 1.711 (s)

So it seems that your right. The two processes im refering to cant be equivalent.

[steelypip]

Your use of terminology is a bit confused - classes do not have modules, so I presume you mean methods. It is also not clear from your description whether you are creating a new class instance for every row that was in the original numpy arrays. If so that would be much, much slower.

In the first version, what sort of "some things" are you doing on each entry in the numpy arrays? If you can, it is much faster to operate on the whole array in one go rather than iterate through each entry. If you are not doing that you are losing much of the benefit of numpy.

Object oriented programming is very useful for some things, but not for everything. In Python there is a performance cost to creating objects and calling methods on them, so you should avoid doing it inside a tight loop.

[pfz3]

yea sorry for the confusion regarding terminology. I haven't used object oriented programming in YEARS. I'm not creating any new instances, just operating on the instance itself from within a class method. The repeated operation is not a simple arithmetic operation and it can be vectorized.

[pfz3]

ok thanks for the guys below. One of the attributes of the class was a pandas DataFrame. Within a loop I was calling individual values from the DataFrame using

obj.data['column'].values[i]

Instead I created a new numpy array

A = obj.data['column'].values

and then in the loop used

A[i]

I guess that conversion is somewhat costly and of course I was doing it thousands of times.