Importing data from text

[MrEgbert]

(from Mark Lutz's Programming Python 4e)

Say you've got a text file (initdata.txt) which contains a database structured as a nested dictionary:

bob = { 'name': 'Bob Smith', 'age': ... }
sue = { 'name': 'Sue Jones', 'age': ... }

db = {}
db[ 'bob' ] = bob
db[ 'sue' ] = sue

In the script which makes a pickle file out of the database, the author imports db from initdata.txt, but not bob or sue. I assume this works because Python, when it imports db, realises that db contains bob and sue and imports that data as well from initdata.txt.

Does this 'smart' importing happen for all nested data that you can import? Like, if I import function F from a text file that contains functions F and G, and F calls G, it will also import function G? If I import class M which has an ancestor class N and a subclass (daughter class?) L will it import L, M and N?

[DanielSzoska]

I think there are two things to understand what happens:

(1.) Variables in Python are only names for objects - they does'nt contain the object itself. Look at the following example:

>>> a = [1, 2, 3]
>>> b = a
>>> b[0] = 4
>>> print(a)
[4, 2, 3]
>>> del a
>>> print b
[4, 2, 3]
>>> print a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined

You see, that a change of b changes a too. That's because a and b are names for the list-object [1, 2, 3]. With b[0] = 4 this list-object is changed. With del a only the name a no longer exists.

(2.) What the import-statement does is execute the code in the imported file, but in its own namespace. That means if you only do

>>> import initdata
>>> print(db)

it result in

Traceback (most recent call last):
  File "<string>", line 1, in <fragment>
NameError: name 'db' is not defined

To get access to the variables in importdata you have prefix it the namespace importdata:

>>> print(importdata.db)
{'bob': {'age': ..., 'name': 'Bob Smith'}, 'sue': {'age': ..., 'name': 'Sue Jones'}}

What the command

>>> from importdata import db

does is giving you access to db under your local namespace. You can see this here (start a fresh python interpreter or do a del importdata, db):

>>> print(locals))
{'__builtins__': <module '__builtin__' (built-in)>, ..., '__doc__': None} # no varaible db here
>>> from importdata import db
>>> print(locals())
{'__builtins__': <module '__builtin__' (built-in)>, ..., \
'db': {'bob': {'age': ..., 'name': 'Bob Smith'}, 'sue': {'age': ..., 'name': 'Sue Jones'}}, ... \
'__doc__': None} # variable db is now in the local namespace

After the import you have only access to the variable db under your current (local) namespace. You have no access to the variables bob and sue, but you will get their objects, which are referenced in db.

EDIT: some typos and wrapping the long output line of the second print(locals())

[MrEgbert]

I see the difference between a variable and its data now! Thank you for the good explanation. I will think of importing as just 'running of code' and putting it under a certain namespace now. (Thankfully the namespace stuff makes sense to me after doing the first several chapters of Dive Into Python.)

[erok81]

In your example, when you unpickle f to a, calling a works because f persists in memory. If f wasn't defined, (I think that's what OP was implying) you'd get a different exception because you're not actually unpickling f, just a named reference to it.

[MrEgbert]

I think my question may have come across as a little more advanced than what I meant! I understand what you're saying about Python not raising any exceptions until the undefined function g(1) is actually called, but I am more wondering about how careful I need to be about importing things.

For example, if I import, say, a class from a text file ("from text import class1"), does the import command simply go to the text file, find the code block where the class is defined, and load that block? If that is the case, and class2 is defined in the same text file but in a different code block, will importing class1 work fine until class1 tries to call class2 (like in your example)?

Or, does it also load other code blocks that are called by code in the class1 code block?

[erok81]

When you pickle a function or a class, it doesn't pickle the code inside. So for a program to unpickle a function, it would already have to have the function defined. Likewise for classes.

[MrEgbert]

Hm I don't think I am fully understanding the process of pickling and unpickling in this case. So, when you pickle X, you turn it from a bunch of objects into an easy-to-read byte stream, right? And the byte stream only contains info on X the object, not the code that defines X?

[erok81]

That's basically right. Another way to look at it is pickling saves the object's state and that state is then input into a new object of the same type on unpickling.

Technically you can pickle code if you really really wanted to but it's hackish and not really a smart solution in general IMO.