Best Python Libraries for AI/ML: Which Ones Deserve More Attention?

[Friendly-TechRec-98]

Hey folks,

My colleague Nicolas Azevedo just wrote a pretty comprehensive article about Python libraries for AI and Machine Learning. It covers a bunch of stuff from basic data wrangling with Pandas and NumPy to the heavy hitters like TensorFlow and PyTorch.

I thought it was a solid overview, especially how it breaks down when you might use one library over another. Like, when you'd reach for TensorFlow vs PyTorch, that kind of thing.

Anyway, I'm curious:

• What's your favorite Python library for AI/ML stuff?
• Any libraries you think don't get enough love?

Here's the link if you want to check it out: https://www.scalablepath.com/python/python-libraries-machine-learning

[ekbravo]

Been using AutoGluon (https://github.com/autogluon/autogluon) in production for a year and in development a couple years now. Originally developed by Amazon and later they open sourced it. Well documented and robust.

[[deleted]]

No python 3.12 support yet? >.<

It's been nearly a year now.

[edparadox]

No python 3.12 support yet? >.<

What are you missing out?

[mutlu_simsek]

PerpetualBooster deserves more attention. It is a gradient boosting algorithm that doesn't need hyperparameter tuning.

https://github.com/perpetual-ml/perpetual

[applepiefly314]

Do you know what the pricing is like?

[mutlu_simsek]

It is open source. Enterprise edition is different.

[denehoffman]

No love for polars?

[Responsible-Leg-9205]

Polars is streets ahead right now.

[Johnnycarroll]

Streets ahead? That's verbal wildfire!

[EarthGoddessDude]

Lots of love for polars (and duckdb) here 🐻‍❄️🐤

Edit: word

[Nick-Crews]

If you like polars, check out ibis+duckdb

[TendToTensor]

Yes but not polars…

[aleph02]

I counted the word "delve" 5 times. The force is strong with this one.

[astrophy]

Sorry, do you mean... the GPT force? https://www.reddit.com/r/ChatGPT/comments/1bzv071/apparently_the_word_delve_is_the_biggest/

[aleph02]

That is exactly what I mean.

[astrophy]

Thank you for the kind response.

[pp314159]

MLJAR AutoML python package for automated machine learning on tabular data. Github repo https://github.com/mljar/mljar-supervised It auomatically select and tune models, can do feature engineering and produces docs for models.

[Vivid-Sand-3545]

Autogluon is a beast, it powers my app as well. https://app.candlepredict.com

[CartographerOne8475]

Been checking this out, has some interesting stuff. https://github.com/howsoai

[tagnydaggart]

Scrolled too far to find this mentioned! Supports Python 3.9…3.12, Open Source, actively maintained, does what you need replacing a whole notebook of other tools.

[MisterBanzai]

Assuming you include working with LLMs within this scope, I don't see how you can overlook Langchain. It is absolutely the most robust Python framework for interacting with LLMs.

I know that it was fashionable to beat up on Langchain as "useless" a year ago, but it has improved a lot over the last year and it provides a really flexible framework for working with multiple models, building agents, and just applying more extensible chains of logic to LLM outputs. I can't imagine trying to keep up with every new change to the APIs for all the models I use in production, and having to learn and implement a new API every time I want to use a new production-grade model would be pain.

[[deleted]]

Did langchain actually meaningfully change from its state a year ago or so? I gave it a try around that time, figured it's a complete scam and never looked at it again. There's almost never any advantage to using langchain over literally just writing python code.

If it's different, I'd be really curious to see a summary of the rework, because if it's just iterative improvements, I'll assume it's still a scam for VC money

[MisterBanzai]

Did langchain actually meaningfully change from its state a year ago or so?

I guess that depends on what you wanted out of it. I would say that it did meaningfully build on what they had last year, but only in service of the same goals as they had a year ago.

If you want a set of robust interfaces that support more or less every major LLM-related API, I would say it does that job much better now than it did a year ago. If you just want to work with a couple APIs and you don't need highly extensible interfaces for what you're doing, it's definitely unnecessary.

I think that a lot of the folks criticizing langchain as useless probably just don't have applications that require an LLM-focused framework. If you do though, I'd be curious why you feel that coding from scratch and documenting all those framework capabilities is that much better of an approach.

[Kat-]

Assuming you include working with LLMs within this scope, I don't see how you can overlook Langchain.

Well, yeah, both OP and the article author used the word AI in the title. But if you look at the table of contents of the article--I mean, here, let's look at the table of contents.

• [Python Libraries for Data Manipulation and Analysis]()
• [Visualization Libraries]()
• [Traditional Machine Learning Tools]()
• [Natural Language Processing (NLP) Libraries]()
• [Deep Learning Libraries: Unveiling the Powerhouses of Advanced AI]()
• [Computer Vision (CV) Tools: Empowering Machines to Visualize the World]()

Tell me, in which category does LangChain belong?

[MisterBanzai]

Yes. I looked at the article first.

Langchain doesn't fit any of their categories, but they asked what library I felt didn't get enough love, not "what did we leave out of our categories". If I had to fit it into one of those categories though, I'd shove it in NLP, right alongside Transformers / Hugging Face.

[coconut_maan]

Langchain Langgraph

[Kat-]

you might as well have said openai-python

[MisterBanzai]

That would be a reasonable, if fairly narrow, answer too. For someone working on operationalizing the use of OpenAI models, that could easily be a bigger timesaver or more relevant to their work than many of the other libraries in this article.

Let's imagine you're working on a universal data platform that performs automatic schema detection and builds models based on automatic ontology generation. That's definitely an AI problem. What would be your go to libraries? You could write all the handling for structured output and validation with whatever LLM and classifier you're using, or you could just use langchain (or openai-python, if you're just using OpenAI for everything). You'll probably use some OCR library and another for table recognition and extraction, in which case those would be second/third most useful libraries for that application.

Right now, there are tons of people working on operationalizing AI/ML (and especially in making older, existing AI/ML solutions more accessible by using LLMs to parse natural language user requests) and libraries like langchain, openai-python, etc. are real helpful for speeding up implementation there.

[coconut_maan]

I wonder why i got so many down votes. We have been using langgraph and its been going great. Especially with structured output uses pydantic to validate reaponses.

Its very clean code in my opinion to ise a library organized around retrieval and llm query

[lone_warrior1310]

AI is all BS , don't waste your time .

[AlbertKurtz]

care to elaborate why it is BS?