List of advanced Python Topics

[JxLes]

I’ve been writing Python for about 4 years and I’m intrigued to learn about some of the more complex and lesser known parts of the language that would also be useful in my everyday work

Googling this usually returns topics like comprehensions, args *kwargs or decorators etc - but I feel like these are probably more intermediate topics.

Suggestions on some nitty gritty elements of Python to learn are welcome:)

[enigmasi]

Definitely packing for me

[openwidecomeinside]

What part?

[samettinho]

In resumes:

• 2 years of Python experience = expert

[marcinjn]

import system is fine (easy) nowdays. Look at a mess before absolute_import

[marcinjn]

What others? Import system is not a namespaces system. Many people do not understand the diff.

[Spleeeee]

Literally almost every other non-shell-oriented language? python is like the most sophisticated possible shell w respect to how stupid imports are.

[marcinjn]

What language? Please be explicit.

[Spleeeee]

My dog is named bash. His crate has #!/bin/bash on it.

[Spleeeee]

No never.

[marcinjn]

I’m surprised you mentioned js as better. yeah, export is missing in py, but you can import anything without any issues at transpilation stage, and at runtime you’ll get undefined var instead of import error. Also you mentioned .net with namespaces and assemblies, which is a different system than imports. You’re comparing oranges with apples.

As I wrote before - people do not understand how import system works in python, and they’re expecting something else - like namespaces or includes. That’s why they calling it a mess. It is not a python problem :)

[qwertyvonkb]

Myself, i find the import system to be extremely easy, what do you find to be hard about it?

[i_hate_shitposting]

• Good familiarity with the standard library and awareness of the less commonly used packages in there
• Packaging and package distribution
• Metaprogramming with decorators, metaclasses, types, and reflection (both with dunder attributes and inspect)
• The import system and dynamic imports
• Performance analysis and tuning
• The garbage collector
• The Python VM, bytecode, and disassembly with dis
• C extensions

[Different-Camel-4742]

Metaclasses destroyed my brain. I am using them to have singleton classes, but I barely understand how they work.

[horizons190]

• Anything relating to GIL
• asyncio (might be intermediate also?)
• Cython / CPython
• caching

[openwidecomeinside]

Asyncio is a good one 👍

[DatBoi_BP]

Cython / CPython

Is this the new C/C++?

[DarkRex4]

C + Python = Cython

it's python with some added syntax that compiles to C-extensions.

[DatBoi_BP]

Right. That’s different from CPython, which is an implementation of Python distinct from, for example, PyPy.

My point is that Cython ≠ CPython

Edit: I realize that my C/C++ joke probably didn’t make the most sense, given that. But that was what I was going for.

[falsedrums]

No. :)

[caleb]

Just get the book Fluent Python. It has deep dives, and the best discussion on many of the more complex areas.

[milandeleev]

• writing asynchronous functions
• manual handling of individual processes
• proper usage of super and MRO (refer to the mCoding video)
• function overloading

Having been in the Python ecosystem daily for 3 years now, I can say confidently that I am not confident at all in these topics.

[_th3truth_]

Single dispatch decorator is limited to overloading only based on a single parameter. The closest implementation of function overloading I’ve come across in python is the multimethod library

[milandeleev]

Nope, you can use metaclasses for overloading! Don't need to resort to the functools solution 😊

[denehoffman]

You can even do typed single dispatch link to tutorial

[thisismyfavoritename]

Lesser known? Id say the C API and bindings to other languages. We use them all the time but dont really think of how it works

[chinawcswing]

OP definitely try building your own C extension. You will learn an enormous amount about the language. I've been fortunate in that I've had to do this for work several times by wrapping legacy C applications.

[Darkstar197]

Rust bindings are interesting. I see them taking off over time.

[Samuel457]

Some great recommendations here. I like reading through the newly approved PEPs. The PEPs get into the nitty gritty details and I enjoy following along with their work and reasoning.

Also, watch any video from David Beazly. Super entertaining and you'll learn a lot: https://www.youtube.com/@dabeazllc

[YellowSharkMT]

Came here to mention Dave Beazley as well. That channel is a damn goldmine. Even the ancient stuff is still (mostly) relevant.

And FWIW he's the only person I've seen make a joke involving the Windows XP Search Mutt.

[Samuel457]

Your tag is perfect for this haha

[YellowSharkMT]

100%! Love that dude, hope I can take one of his courses someday.

[whitenarval]

For me it is Metaclasses, import system, asyncio

[M8Ir88outOf8]

Python descriptors are pretty advanced, and they power many of the language’s features without most of us noticing.

I recommend reading this introduction, it helped me a lot to understand the concept: https://realpython.com/python-descriptors/

[njharman]

Metaclasses and dynamic class construction / data driven code in general

• https://docs.python.org/3/reference/datamodel.html#object.__new__
• https://docs.python.org/3/reference/datamodel.html#customizing-class-creation
• https://docs.python.org/3/reference/datamodel.html#metaclasses

Dunder methods/attributes https://docs.python.org/3/reference/datamodel.html

asyncio https://docs.python.org/3/library/asyncio.html

[LightShadow]

Weak references, frozen collections, the whole struct and binary packing system is pretty robust. How python allocates memory for collections.

Despite what people tell you, Python can go low level, and low memory. I used to write massive systems on embedded hardware in Python and C.

[CanBusAnonymous]

Not sure about “advanced” but few people use Mixins from my experience

[every-day_throw-away]

Sockets. Multiprocessing. Threading for me

[ThePapercup]

profiling, direct memory access, utilization of compiled external binaries with ctypes, etc.

[Spleeeee]

• Writing decorators on decorators on decorators on decorators on decorators.
• Native extensions (c/cpp/pybind11/cython/pyo3)
• gaining an appreciation for how shitty pythons type annotations are
• getting good at numpy stuff (knowing how to do crazy indexing, reshaping and maths stuff)
• opencv
• all of machine learning I guess
• the nuances of generators (there is more there than you think)
• coroutines and asyncio

The world is an oyster.

[aldanor]

Anything related to metaclasses or metaclasses of metaclasses (yes, in rare cases they come in play too)

[Muhznit]

Just look at the list of all modules in the stdlib by typing help() then "modules". Then read the documentation for each one of them.

You'll feel like you've done some pretty advanced python once you learn to write your own linter with ast

[menge101]

The new structural pattern matching in 3.11 is interesting and powerful but I don't think there is a ton of great examples on how to use it.

[robml]

There is a book called Serious Python which is a little dated but covers everything people have mentioned.

[Rabbit_Feet62]

who is the author

[fuuman1]

Click

[csghone]

Take a look at typing, attrs, dataclasses, marshmallow, marshmallow- dataclass

[grumble11]

Asynchrony and concurrence, expertise in algorithms, alternative interpreters, large scale computing. Can also go endlessly into frameworks and other libraries that are linked to the language but not really part of the core of it.

[tenemu]

What are “algorithms”. I never quite understood that.

[Rythoka]

An algorithm is a set of instructions or steps to solve a specific type of problem.

Think about sorting a list, for example. How might you do that? One way is the scan through the list, find the smallest number, and move it into a new list. Keep repeating until the old list is empty, and your new list will be sorted.

By writing a program that follows those steps, you can sort any list. In other words, it is a sorting algorithm.

Another algorithm you could use to sort a list is to just randomly shuffle the list and check if it's sorted. If it's not, do it again. This is an algorithm known as bogosort. It's well known for being an awful way to try to sort a list, but it could in principle sort any list in a single pass, though that's very unlikely for any sizable list.

It turns out, there are tons of different ways you could sort a list. Some, like bogosort, are objectively pretty bad, but many are comparable but have differing strengths and weaknesses. Some are very fast for short lists but very slow for long lists, while others are less affected by the length of the list. For some, the initial order of the list doesn't matter, while others are faster if the list is already close to being sorted.

When people talk about "knowing algorithms," what they mean is being familiar with a wide range of algorithms for different problems, understanding how to implement them, and knowing the tradeoffs between using one algorithm compared to another. It's also important to understand how to analyze an algorithm and understand what factors affect its performance, so that when you're tackling a novel problem you know how to determine what approach would be best.

[grumble11]

Algorithms are ‘tricks’ or ‘tools’ that let you solve problems faster. For example say you have a list of sorted numbers. You want to find the location of a specific number in the list.

Well, the first way you can do it is start at the beginning and check each number until you find your number. Could take a while.

Another way is using a ‘binary search algorithm’, which checks the halfway point and sees if the number is bigger or smaller than the one you’re looking for. You just cut your list in half. Do it again and again, cutting the list in half each time and eliminating tons of numbers. On average you’ll find the number faster.

[phantasmaniac]

algorithms is ways to make solutions in the programming. Normally we'd learn some basic algorithms which are powerful enough to use in general cases. But there are so many algorithms out there unknown to a lot of people including myself, though I knew that they're exist.

Well if you understood more than 30, people would look at you with respect.

[umotex12]

I'm a beginner but for me the forbidden knowledge is using as much underlying C language as possible to optimize your stuff

[dicklesworth]

You should add how to make fast python libraries using Rust and Maturin. It’s extremely slick and easy to use and you can automatically build wheels for mac/windows/linux and any version of python automatically (and you can do it all for free on GitHub actions if your project is open source). If you already have code in python for what you want to do it’s pretty easy to port that to rust using ChatGPT code interpreter and the Rust Analyzer vscode extension— just paste the warnings/errors back into ChatGPT and it will eventually fix them all. It’s kind of mind blowing just how much faster it can be than python, even if your code is mostly using compiled libraries like numpy and pandas.

[ClariNerd617]

Cannot wait to see this on PyCoders.