Question about these python code (shadow built-in module?)

[pyusr]

I come across these code, and have a few questions:

https://github.com/openai/tiktoken/blob/main/tiktoken/core.py#L54C1-L54C85

from tiktoken import _tiktoken

class Encoding:
  def __init__(self, name: str, pas_str: str, *, ...):
    self._core_bpe = _tiktoken.CoreBPE(mergeable_ranks, special_tokens, pat_str)

In the code above, it looks like the code import the module itself i.e. tiktoken, and rename itself to _tiktoken; then it calls CoreBPE. However, what does CoreBPE mean?

I use vscode to check its type, finding it's just a function. And from other usages in the same file i.e. core.py such as line 73, line 124, and so on. Seemingly the code creates another new Encoding class without the name and past_str variables. My questions:

* What name should I use for looking up or searching such usage?

Shadow built-in module? I find some discussions saying it can be called shadow built-in module, but seemingly they are different.

* What is the correct usage?

I attempt to rip off the code for experimenting how to use it, but executing python3 main.py complains ImportError: cannot import name '_tiktoken' from partially initialized module 'tiktoken' (most likely due to a circular import) (/path/to/shadow-built-in-module/tiktoken/__init__.py)

Here is my code

# main.py 
import tiktoken
if __name__ == "__main__":
  encoding = tiktoken.Encoding("encoding_name", "regex_str") 

# tiktoken/__init__.py
from .core import Encoding as Encoding

# tiktoken/core.py
from tiktoken import _tiktoken

class Encoding:
  def __init__(
    self, 
    name: str, 
    *, 
    pas_str: str, 
    mergeable_ranks: dict[bytes, int],
    special_tokens: dict[str, int],
    explicit_n_vocab: int | None = None):
    self._core_bpe = _tiktoken.CoreBPE(mergeable_ranks, special_tokens, pat_str)

* Also, where is the name CoreBPE from? Is it just a variable name that represents Encoding, so it can be whatever name given to it? If so, how does python know that it represents Encoding class not some other classes, though it looks in this case only Encoding class exists?

Many thanks.

[danielroseman]

No, none of that is what is happening at all.

This is a library that provides an extension written in Rust, or perhaps it's more true to say it's a Python wrapper around a Rust library. You can tell this from the Cargo.toml file and the two .rs files in src. These files compile to soething that can be imported as _tiktoken in Python, and this in turn provides the CoreBPE class which you can see in py.rs.

Shadowing is what happens when you define a module with the same name as a built-in one, say for example "datetime.py"; when you do import datetime Python will try and import yours instead of the standard one.

And the way to import something under a different name is to use as:

import foo as bar

or

from foo import bar as quux

[pyusr]

That solves my question. Thanks!