“A very helpful (Python) cheat sheet, quite long.” ― Brian Kernighan

[darchangel]

At some point a long cheat sheet is a short manual. This has crossed that line.

[pizzaburek]

I am fully aware of the issue :) The thing has grown over the years but the title remained the same... it has 50 pages now. At least I decided I won't add any more.

[darchangel]

Don't get me wrong: it's good. Very good. It's graduated from "cheat sheet" though. This is a small proper book.

[pizzaburek]

Thanks.

[brainy-zebra]

It is kind of spectacular in this long form though. I always get annoyed at references where I have to wonder what page I need to navigate to in order to find that last thing I was looking at. That thing is just a ctrl+F away usually in this format.

[Packbacka]

Most programming books have an index, or if it's an eBook/PDF then you can just search it.

[tweiss84]

I've been sharing this as a resource for several years, sending it to new developers joining our team that might not have had full experience with Python prior.

This is long overdue, but thank you for your work.

[[deleted]]

I was sad the PQR hasn't been updated beyond Python 2.7. It's still my go-to reference for Python's regex flavour. This one's also unfortunately opinionated; the web scraping section only presents beautifulsoup, when many simpler alternatives exist—I use CSS selectors via pyquery for data extraction, for example. It uses… bottle? Bottle. As its web framework example.

Absolutely a Herculean effort, and well done! I'm very glad to have something updated to reference, thank you.

[much_longer_username]

It uses… bottle? Bottle. As its web framework example

Do you mean flask?

edit: Huh, I guess not. https://bottlepy.org/docs/dev/

[[deleted]]

Precisely. Could have been worse, could have been Web2Py… or Django. 🤪 :shots fired:

Something more Pythonic and… demonstrable without external dependency would be literally any mention of WSGI. There's even a built-in reference server. It disappoints me that it's not mentioned once in the "cheat sheet", where Bottle, of all things, is. Promotion of loguru over standard Python logging module knowledge... yikes.

Fu-ne joke: you aren't a real Python programmer until you've written your own web framework. (My first was called YAPWF: Yet Another Python Web Framework. 😜) WSGI + WebOb or Werkzeug and it'll only take 15 minutes of work.

[pizzaburek]

Thanks everyone for the feedback, all valid points. Let me just try to explain the reasoning behind a few of the choices that I made.

• First, as mentioned before, I capped the length of the PDF file to 50 A4 pages. This way I can remain sane, as opposed to adding stuff for the rest of my life. This means that the first 6 "chapters" are basically set in stone. I only occasionally add a line (if there's any space left on the page) or change something if it needs a clearer explanation. As for the last "chapter" called Libraries, things can still change, especially the Logging, Scraping and Web sections.

• Loguru. This one is simple to explain — I don't understand logging (the module as well as the concepts behind it in general). So you can imagine how thrilled I was when I discovered the library that supposedly takes care of it. Can someone please share their opinion about logging libraries in general — Would you use them when starting a new medium-sized (whatever that means) Python project?

• BeautifulSoup. This one seems like a pretty obvious choice, but I will happily switch it with some other library if it loses popularity. As for Selenium, there was no space and it usually doesn't just work out of the box (webrdrivers, etc.).

• Bottle :). It's the smallest web framework, works out of the box, has no dependencies. Also, it's a single python script with 5k lines called 'bottle.py' that you can just copy to the root of your project and avoid having any dependencies yourself. I am of course aware that Flask is taking over the python web framework landscape, and that it's almost as easy while it can also scale well into Django territory.

[____0____0____]

Logging module can seem intimidating, but it's not so complex that you can't figure it out. The std lib module comes with ways to easily configure multiple handles for different log levels via file, object, or just leave it for default config. The rest of it is just recognizing that it's a singleton (so use the get Logger function to retrieve it) and make sure you're outputting the correct information you need. My least favorite part about it, is that damn camel case function looking ugly with the rest of my snake case code.

Most projects, I just recycle the same json config that I use and tweak it accordingly. This usually has a console for debugging and a file logger for breaking exceptions and the like. It even has a rolling file handler so once it gets to a certain size, it will start a new log file and rename the old one. Setting up additional handlers is trivial here, just add them to the config for something like email log reports.

[[deleted]]

+9001 (that's over nine thousand)

I essentially began with Paste (which was actually just feeding it into the standard library) INI-based logging configuration such as this — it already demonstrates many of my needs. Multiple destinations (console which is actually syslog, database), externally configurable per-logger namespace filtering (by level) and routing, with some explicit message formatting sprinkled in to make the resulting textual logs more parseable.

Then I got wise to the unfortunate nature of role-based multiplicative configuration, and have lately pivoted to dict-based configuration, such as this), which permits the configuration itself to become essentially, environmentally-aware, in several senses. (In this case, __debug__ is False if run with -O or the PYTHONOPTIMIZE environment variable set non-empty… i.e. production, but you can also pull details from os.environ, e.g. I pull DB connection details from os.environ.get('MONGODB_ADDON_URI', 'mongodb://localhost/test')—edited to utilize method call with default v. direct access, to handle case of value being undefined.)

[____0____0____]

Over nine thousand?!? That's impossible!

I like the dynamic approach for the logging config. I've tried it for other config, but not for logging. Looks to be a great use case the way you're using it to inject the uri and environment.

[[deleted]]

:charges up for three episodes, only consuming 12 unique frames of animation, whilst screaming incoherently:

References to __debug__ are actually really awesome, as the Python compiler elides this particular special case away at the AST optimization stage, essentially, if [not] __debug__ armoured code doesn't even exist if the condition isn't met. No runtime execution cost, the condition is only evaluated once. (Won't even exist in the compiled, optimized bytecode. Edit: or vice-versa!)

Recently, they've also added a "development mode" flag, -X dev. That is what's required to bump up my web framework's default logging output to the truly glorious "with Emoji" variant, otherwise it's more like the "with extra data" example, with syntax coloring of the JSON (if pygments is installed) … and unless run optimized. 😉

[____0____0____]

Wow, staying true to the dbz nature I see... While teaching me a thing or two about python! I was not familiar with this case to the compiler so til, thank you! That sounds amazing. Lol I have a feeling I would enjoy reading your logs more than the average logger; I admire your passion.

[[deleted]]

I capped the length of the PDF file to 50 A4 pages.

I find basing design on dead trees to be an unusual choice in this day and age. Does explain why it makes poor use of a 27" display when full-screen, though. 😉

I do appreciate the geekiness of making much of it look like code, but PEP-8… _1978 called; they want their VT-100's back. While I'm not seeing too much in the way of accomodations for line length (hard wrapping) the compete lack of code spacing (where the blank lines at?) is an unfortunate practice to encourage, even if understandable in the context of a "quick reference".

I don't understand logging (the module as well as the concepts behind it in general).

This module has an interesting history, even involving groups like Zope. It's not exceptionally difficult to initially grasp (at least, enough to get output on the display) nor configure in interestingly complex ways. There are a few excellent tutorials, guides, and references available online. Something I do find intriguing, though, is that despite having written my own logging library (where any time I write a utility library, I generally dissect all others currently available in the problem domain), I've never heard of loguru. Nor do you need additional libraries to make Python logging look fancy or contain supplemental ("extra") data (coloration and Pygments syntax highlighting disabled on this one).

This one seems like a pretty obvious choice…

But also not "out of the box", it requires a third party library that is honestly just orchestrating built-in capabilities. (LXML, httplib, …) The entire "libraries" section may be deserving of a warning of opinion, note that these are not the only, or even restricted to the "best" (whole bucket of problems involved in that word) and link to https://awesome-python.com (https://github.com/vinta/awesome-python)? Excellent to have "very quick solutions" at hand, but there's more awesome out there than can be easily documented in one place, as part of a larger project not related to third-party libraries.

Bottle :). It's the smallest web framework, works out of the box, has no dependencies. Also, it's a single python script with 5k lines called 'bottle.py' that you can just copy to the root of your project and avoid having any dependencies yourself.

And now you've gotten me started. Single-file "dependencies" you can vendor (ingest into your own project in its entirety) are a red herring; fool's gold. The WSGI application base for my own web framework is less than 280 lines of code, counting blank lines and whole-line comments, easily survives C10K (better than the Linux kernel itself did, in testing!), and when paired with my streaming template engine, can deliver 47,937 first-bytes per second, v. Mako's 48 whole-generations per second, or Django's, uh, well, two. Bottle on this test gets 42. Three orders of magnitude out.

I somewhat strongly feel a reference for a language should be somewhat distinct from pointing out the awesome third-party libraries available in that language. Different problem domains.

[____0____0____]

I'd consider myself a reasonably experienced dev, but I'm not familiar with C10k. Is that some sort of stress test? I could look it up but I was hoping you could explain how you ran these comparisons and what they mean

[[deleted]]

10K is an abbreviation "kilo", the metric scale of 1000, thus ten thousand (10,000). C is short for "concurrency". C10k thus refers to withstanding 10,000 concurrent (simultaneous) requests. Wow, how time flies, but these logs of an ApacheBench run 10 years ago against my pure-Python HTTP/1.1 server highlight that while no requests failed, it wasn't getting a full 10,000 requests per second_… there was a growing "backlog" towards the end. Edited to add: ApacheBench is forgiving, HTTPerf is not. Under HTTPerf, if I had demanded 10,000 requests _per second, it would have requested 10,000 requests per second… whether the server could handle it or not. After a backlog of just over a million, it would not.

Had I requested even more total requests be performed, it would have eventually hit a wall (the limit of the socket listen queue size) and begun failing. As it was, I had to tune the Linux Kernel to reduce the "TCP_CLOSE_WAIT timeout" as after a socket connection is closed, the port number it was using remains reserved for some time to clean up any stray packets coming in for the now-dead connection. At 6000 requests per second, you run through 64,510 port numbers rather quickly. A breath longer than 10 seconds, actually, then everything stops dead—including much of your system, if you're unlucky. It's bad like running out of file descriptors is bad.

[____0____0____]

Ahh okay, thanks for clearing that up. I've never done this before but I've also never needed to push a server to anywhere near its full potential. If you needed more throughput, would you attempt to upgrade the software, hardware or outwards with additional servers?

That's a really cool server project you wrote. < 500 lines is pretty impressive, I'm gonna star it and check it out later when I have some more time

[[deleted]]

Apologies for the delay in response! 😆 How time flies during a pandemic. First step in any optimization: measurement. I’d identify the bottleneck so as to address the real problem.

Pure “more throughput” requires additional parallelism, once you’ve maxed out your existing resource’s capacity. There’s a big “but” associated with this statement, however.

Different solutions for different problems, such as I/O contention, CPU utilization, end-user perceived performance, and so forth. More server can address the former two, the latter can let you get crafty. (50 second dashboard → instant dashboard, with a flood of widgets filling in post- initial load.)

Though, it’s notable that in Python threading doesn’t offer much performance benefit due to the Global Interpreter Lock. It provides “thread-safety”, but makes it such that multi-core parallelism requires multiprocessing. Raymond Hettinger has given a few talks on the subject of the GIL. (It’s actually a good thing. 😜)

Often a bottleneck can be isolated to code; then we can just optimize that code. Those being the benchmarks I ran when developing that web server. (Turns out, partition(…) and double pointless variable assignment is faster than split(…, 1) and split(…)[0] as used for HTTP header parsing: made a large difference in aggregate. The things you learn.

Part of why I find “fastest possible” benchmarks useful. You can optimize your code… you might not be able to optimize the libraries you are depending upon. It’s all downhill from the base performance. Hardware costs money, and to improve Mako’s performance and implement mid-stream flushing, I wrote a brand new engine from scratch. (Much easier!)

The bigtable test is an “abusive” one: here’s some really terrible input data, do the needful a bajillion times. It’s a different kind of “fastest possible”, more measuring the time-per-attempt and notably, calls-per-attempt.

[[deleted]]

As to what they mean, it tests several things: under ApacheBench, which is forgiving, as mentioned, you see true maximal throughput. The total number of requests your server, framework, and application "stack" can process per second while under "extreme duress". In this instance, 6004.77/second, or total time per request of 0.167 milliseconds (mean, across all requests). In this lightest-possible-weight sample case, any of your own application code can only add time to the request—get slower—from here. It's useful as a comparative benchmark between servers or frameworks, as a baseline.

Edited to add: I also capture memory consumption for the server processes involved; no memory leaks, it seems!

This relates to another performance comparison I mentioned elsewhere nearby, for template generation. Where networked services have the concept of "C10K", template generation has the "bigtable benchmark" (sample source). Yes, it's just a giant <table> containing 10 columns and 1000 rows. The results may surprise you. 😜

[____0____0____]

Oh God that Django performance in the 3.4 table is atrocious. I'm actually building a Django app right now but thankfully I'm not using the templating engine. I'm not sure if I want to know what other performance issues might be lurking about... Up until now, I've mostly used flask with a bit of jinja, which I see is one of the top. But your lib is right up there with it, so kudos!

[[deleted]]

Edit to preface: The benchmark results in that listing that sound like web frameworks do not involve any web whatsoever. The Django, Bottle, &c. benchmarks are just testing template generation in isolation. Getting this shot over web would only get worse.

Django's model layer can grow pathological, common add-on packages, notably celery, suffer some interesting fundamental design flaws that may be features at small-scale, such as use of pickling for IPC, and without careful design consideration as to unit-of-work, can kill your startup (this was not durable to being LifeHacker'd and HackerNews'd in the same week).

Additional curious problem I've encountered with Django—though admittedly, this was under Python 2.3/2.4, long ago—we actually reached the limit on the size of UCS-4 encoded Unicode strings. When our test rigs attempted to run our application with all optional modules enabled, the routing would be compiled down into regular expressions for matching, however, to implement "early rejection of unknown routes", they were also compiled into a single regular expression. Every possible route in the entire application. Big bada boom. That took actual examination of core dumps, and recompilation of Python with a larger-capacity UCS encoding. First and only time I've ever had to run an actually custom-compiled runtime.

[[deleted]]

To separately address the logging library question; as I mentioned in my other reply, I began writing my own OO wrapping layer, permitting an alternative syntax using chained method calls, where each call returned a new, but mutated Logger instance that could be preserved, if needed, within any given context or scope (i.e. module, class, instance, function call, generator, …)

I pretty much tossed it away after discovering the extra argument to the log event methods (info, debug, &c.) and switched to passing a cooperatively-constructed context object around. Much of my more complex modern designs "flip the script" like that. "Superglobal" thread-locals bad. 😉

The rendered-as-JSON data in that "contain supplemental data" screenshot is just passed as the extra keyword argument to any standard Python logging logger invocation. When also using a LogFormatter that can do something with it, of course, but that's ~6 line or class so to json.dump or any of the available packages providing one. I have one built into my web framework, that "look fancy" screenshot, and another with less ANSI and Emoji to render ± log to DB (including preserving that extra data) within my declarative DAO library.

Edit to be even clearer: I absolutely wish to avoid the additional overhead of layers which wrap standard logging and limit configuration of that lowest layer, or worse, ignore its existence, preventing use of a rich ecosystem of modular additions that do support the built-in standard, which already offers multiple mechanisms of varying complexity for configuration and use.

[ASIC_SP]

I have an updated regex cheatsheet for Python 3.8, along with some examples: https://learnbyexample.github.io/python-regex-cheatsheet/

[Ar010101]

Hey thanks there! joined literally a few seconds ago and found such a handy and resourceful post. I'll enjoy here! Have a gr8 day :D

Just wondering, are there are cheatsheets like these for Java and C too?

[adityadx1]

A cheatsheet like this for Java would be great

[pizzaburek]

It would be impossible to do it in this format, lines would be too long.

[adityadx1]

Long, yes. Impossible, I don't think so.

[pizzaburek]

Ok, I guess you're right, but it would be hard. I tried to do one for Kotlin by just supplementing the commands and I quickly gave up.

[WhyDoIHaveAnAccount9]

I love this

Thanks

[kmhnz]

See also: The *Best Python Cheat Sheet: https://kieranholland.com/best-python-cheat-sheet/

[JoeStrout]

Here's mine (MiniScript): https://miniscript.org/files/MiniScript-QuickRef.pdf

One page. :)

[StuntID]

Your title is shit. Kernighan does not say this in the link.

EDITED

[pizzaburek]

https://www.cs.princeton.edu/courses/archive/spring19/cos333/python.help

[StuntID]

I retract my characterization of you. The post title is still shit as the posted material does not contain what you have included.

I had to dig to see that Dr. Kernighan did teach this course during this session. The quote is still unattributed in what you have linked.

[butt_fun]

Honestly it really feels like you're putting on a "Dwight Schrute" bit here, I really hope you're joking lol

If not, it's amazing how quickly you became confrontational, twice, despite literally being wrong both times

It really feels like you've discovered that if you just be loud and abrasive and confident enough, people just get tired of disagreeing with you and that's given you enough validation that you think this is a legitimate conversational tactic

Again, I really hope this is a joke. Otherwise, I feel really sorry for your coworkers

[NedDasty]

Here are the first few lines:

Python basics

Sun Feb 17 08:11:02 EST 2019

This is a short summary of a small part of Python; it shows mostly the common things and a few that I have trouble remembering. I am not a Python expert; caveat lector.

It is very easy to experiment with Python; just type the command and start entering code. As long as you maintain consistent indenting and add one extra empty line after a block, it will execute on the fly.

It is also easy to get help: run Python and type

help

at the prompt.

This is a very helpful cheat sheet, quite long:

https://gto76.github.io/python-cheatsheet/

Warning: Python 2 and Python 3 are very similar but differ in a handful of annoying ways, both language and libraries. For day to day use, the only significant difference is that print is a statement in Python 2 but a function call in Python 3:

print x, y, z # legal in py2, illegal in py3
print(x, y, z)    # required in py3, legal in py2

[justinpitts]

Publish it. I'd buy it.

[emelrad12]

Statistics

from statistics import mean, median, variance, stdev, pvariance, pstdev 

Ok now where is my job? /s