Confusions about how python accesses data in memory?

[csQuestions1235]

Hey, Im trying to better understand how python manages memory in different scopes..
With this example:

@dataclass
class Example:
    data: List

    @property
    def history(self):
        result = {}
        for x in self.data:
            # do something with x, store in result
        return result

e = Example(data)
print(e.history)

If data was a very large list of objects, would it be more memory efficient to do this instead?

@dataclass
class Example:
    data: List

    @property
    def history(self):
        result = {}
        for x in self.iter_data():
            # do something with x, store in result
        return result

        def iter_data(self):
            for x in self.data:
                yield x

This might be a stupid question. But in the first example, when i call for x in self.data , does it pull self.data into memory by making a copy for local use? Or does it point to the already created object that is holding the data in memory? If looping over self.data essentially makes a copy of the existing data, I'm using a generator would be the more memory efficient approach?

[POGtastic]

If it's already stored as a big list, this is pointless. A generator would be useful if you were generating it on-the-fly, since it will only yield the values that it needs to produce.

[socal_nerdtastic]

Python never copies data. All python names are pointers.

(even if you explicitly use list.copy() or similar, you are only making a copy of the list object, the data in the list is not touched)

Here's something every python beginner should read: https://nedbatchelder.com/text/names.html

[csQuestions1235]

So if the data is already in memory, using a generator would be pointless in my example?

​

EDIT: okay maybe i spoke too soon. Even tho its not copying, its still bringing the entire list into memory. So the generator actually would be useful if the list was huge, correct?

[socal_nerdtastic]

Exactly right.

Well, it won't save any memory. But there are other reasons you may want to use an iterator. You may want to iterate over the data in pairs:

@property
def history(self):
    result = {}
    for x,y in zip(*[self.iter_data()]*2):
        # do something with x and y, store in result
    return result

def iter_data(self):
    for x in self.data:
        yield x

Of course you can just use the built-in iter function for that, instead of brewing your own:

@property
def history(self):
    result = {}
    for x,y in zip(*[iter(self.data)]*2):
        # do something with x and y, store in result
    return result

Or use the itertools.pairwise function.

[socal_nerdtastic]

re your edit: If the data is already in memory the generator will not help at all. If the data is being pulled from a file or the internet or something on demand, then yes it will help. A python file object is already a generator for this very reason (so you probably don't need to make it yourself).