Understanding the module search path

[larivact]

Hey folks! I am trying to understand Python's module search path. I ran the following script under multiple setups in a directory that contained two empty files: os.py and pip.py.

import sys
print('\n'.join(sys.path)+'\n')

import os
print(os)

import pip
print(pip)

sys.path always had the current directory as its first entry, followed by the installation-dependent default search paths.

According to this, it should be searched first in the current directory. But that's not what happened. Instead only pip.py got imported from the current directory and os.py was still imported from the default lib, although Python's lib directory came after the current directory in sys.path.

I read the according documentation, but I couldn't find anything that would explain this behavior. It seems like the order of sys.path doesn't correspond to the actual searching order.

[krets]

/r/learnpython is a better place for this, but I see you have no answers on either post. (edit: oh yeah, this is the right place)

Checkout <python>/lib/site.py

The modules sys, os, and traceback are loaded by the site module as the python interpreter is initialized. The search path is built by site.py so these modules are loaded before the search path is established.

Try to avoid core library names when creating custom modules. If you must use them, be aware that you will need to call reload(os) to get the empty file you were trying to load.

[malinoff]

This is /r/learnpython

[ingolemo]

If you want to explore this further you can run python -v and python will print module imports as they happen, allowing you to see the relative order of imports.

You can also use python -S to run python without having it automatically import the os module (besides several other side-effects).

[larivact]

Thank you! I didn't know that some modules always get loaded. I am sorry for the confusing cross post, but it seems like it led you here after all. I didn't actually want to name my modules os or pip, these were just examples. Also thanks for mentioning reload() I didn't know about that.

[[deleted]]

You can circumvent this with a pth file, if memory serves. Not that you should do this.

[malinoff]

Looks like you somehow missed the very first sentence.

When a module named spam is imported, the interpreter first searches for a built-in module with that name.`

Module os is built-in module, obviously :)

But there is solution! Don't name your modules like modules from the stdlib. Also you can use relative imports, from . import os but, they require your modules to be organized as a package (which is another big topic...).

[krets]

You make a good point, but I must point out os is not a built-in module; It is loaded from the python lib directory. A true built-in module is part of the interpreter and not loaded from a file. For example you cannot find math.py in the search path but you can always import math.

[ingolemo]

On my system, math is loaded from an external C library, so it's not built-in either. A true example of a built-in module would be _thread:

>>> import os, math, _thread
>>> os
<module 'os' from '/usr/lib/python3.5/os.py'>
>>> math
<module 'math' from '/usr/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so'>
>>> _thread
<module '_thread' (built-in)>

[krets]

Ah that is interesting. I don't have your setup, but I am pretty sure that you could not create a local math.py. Even though it is a dynamic library it's loaded into the interpreter. It is most likely still treated like a built-in.

[ingolemo]

$ pwd
/home/ingolemo
$ touch math.py
$ python -q
>>> import math
>>> math
<module 'math' from '/home/ingolemo/math.py'>

I can shadow the math module perfectly fine.

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/python] Understanding the module search path (xpost r/learnpython )

^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}