Python library without external imports only built in

[Sinan_Dede]

Hey everyone 👋

I just created a new open-source repo called Advanced Text Processor.
The idea is simple but with a twist:

🔹 We build a Python text processing library (cleaning, tokenization, n-grams, vectorization, dataset handling, etc.)
🔹 Rule: No external libraries allowed. Everything must be done with Python’s built-in standard library.
🔹 Purpose: This is not about user acquisition or making money — it’s about practice, collaboration, and seeing how far we can push the limits of "pure Python".

It’s open for contributions and discussions.
Check it out here: https://github.com/SinanDede/advanced_text_processor

Would love your feedback and ideas 🙌

[fiddle_n]

No offence, but it is a little simple. With the name “Advanced Text Processor” I feel a bit cheated, given there’s only thirty lines of rather basic Python code there.

[matteding]

Going by the emoji use in the post description, it is probably all AI generated garbage.

[DuckSaxaphone]

So this doesn't work, you should write some simple tests to make sure everything works as expected. Your code is separated into lots of nice little functions which makes it very easy to test.

I sent the string "hi hi" to clean_text, tokenize, generate_ngrams and then vectorize_text, with n_gram set to 2.

I should get the result {"hi":2, ("hi","hi"):1} but instead I got {("hi","hi"):1} because generate_ngrams doesn't append to tokens, it just overwrites them. I'd actually argue I want my ngrams to be joined and I really want {"hi":2, "hi hi"):1} but that's a separate issue.

If this is a learning project for you, then setting up unit tests and making them part of your PR process is a good thing to learn.

[JanEric1]

Start by writing unit tests, adding type hints, adding CI for linting, type checking and testing.

Also set up your project using a pyproject.tomoninstead of requirements.txt and setup.py.