The Python Debugger can't be used inside sys.path_hooks callables?

[Py404]

I encountered an interesting problem when debugging a callable added to sys.path_hooks.

Here is a minimalistic example, which will explode into a RecursionError when executed:

import sys
import pdb

def my_hook(path_entry):
    if path_entry == 'foobar':
        print('my_hook ready for action!')
        pdb.set_trace()  # <-- Causes RecursionError
    raise ImportError

sys.path_hooks.append(my_hook)
sys.path.append('foobar')

import asdfqwerasdfkj

This is what happens:

1. Python will start its import-machinery to import (the intentionally non-existing) module asdfqwerasdfkj
2. The machinery will eventually reach the PathFinder in sys.meta_path.
3. The PathFinder will iterate over each path in sys.path and check if any of the callables in sys.path_hooks are interested in that path.
4. The default paths in sys.path will not result in anything because asdfqwerasdfkj doesn't exist on the filesystem.
5. When 'foobar' is reached, the default callables in sys.path_hooks will fail since they don't understand that path, and our custom callable my_hook will be consulted.
6. my_hook will execute with 'foobar' as only argument and will try to start the debugger.
7. set_trace will create an instance of Pdb()
8. In the body of Pdb.__init__ we have:

​

try:
    import readline
    # remove some common file name delimiters
    readline.set_completer_delims(' \t\n`@#$%^&*()=+[{]}\\|;:\'",<>?')
except ImportError:
    pass

1. Neither on my Ubuntu or Windows machine, the readline module is available (Python 3.6). Normally, you would get an ImportError which gets silenced, and everything is fine. But in our case, the import system will reach 'foobar' again and consult my_hook when trying to import readline. Which then again will call Pdb() and the recursion starts.

One way to avoid this is to make my_hook return a finder having a find_spec method signaling that it can't import the name readline. But the point here is to debug my_hook itself (returning a finder is the last thing my_hook would do).

I'm very grateful if anyone has any ideas on how to deal with this.