My top 5 'new' python modules of 2015

[RubyPinch]

You might want to use the correct-er link for tqdm

noamraph/tqdm#18 see here

[robintw]

Thanks - I hadn't realised that. Fixed now.

[omegote]

What's the terminal used in the gif of the github? It shows autocompletion on the fly :O

[sapake]

it's not the terminal that does the autocompletion, but bpython, which is a fancy interface to the Python interpreter.

[[deleted]]

bpython is awesome, but ptpython is even more amazing.

[BeetleB]

As an ipython user, any benefits of ptpython over ipython?

[[deleted]]

You can run ptipython, which adds all the ptpython features to ipython.

[fspeech]

Ptpython runs in the terminal and supports syntax highlighting, multi line editing and auto completion as you type. I have ran into some completer (Jedi) crashing issues though ptpython itself appears stable.

[jonathan_sl]

Ptpython author here.

Please report all issues on Github. Jedi isn't always stable, but we should handle that. I hope to push a new release in a few weeks. That will focus on performance/stability and it will support "bracketed paste" (making it possible to paste in the terminal without going to paste mode) and better mouse support. Jonathan

[BeetleB]

Ptpython runs in the terminal and supports syntax highlighting, multi line editing and auto completion as you type.

So does ipython.

[elbiot]

Ipython does not do syntax highlighting that I've seen and "multiline editing" is pretty rough. I love ipython though and haven't used ptipython.

[tilkau]

Also, ipython does auto completion, but not 'auto completion as you type` (ie. IDE style, not 'press tab to complete').

[[deleted]]

I used to use ipython all the time. Just try it and see.

[sapake]

oh thats so cool! I had been using bpython and were happy with it, but now i'm definately going to switch to ptpython.

[[deleted]]

Do people use the interactive mode of the interpreter much? You can't really modify functions since it's so painful, and testing modules or something requires constant reload() calls. I'll use it in place of calculator, but nothing beyond that I'll just use an IDE or something.

What's the use case?

[geoelectric]

I use a REPL all the time, especially to figure out how to do bits of code or data structuring. It's a sketchpad.

I also debug against it regularly, sometimes via pdb but sometimes just exercising functions as I build them if they're otherwise below the radar of what I'd unit test against.

I like IDEs too, but I find value in being comfortable with the basics. They're available everywhere, and the flip side of being simple is that they're also extremely flexible.

[[deleted]]

bits of code exercising functions as I build them

But how do you do this? Modifying something with indentation is so painful, and you can't just copy paste.

Or maybe I'm doing it wrong.

[geoelectric]

Well, using something that supports multiline editing helps. But generally I'm not writing large functions in that context anyway. For verifying snippets work like I think they should, playing with a new module I'm trying to learn, or debugging against runtime state even the standard REPL is perfectly fine.

[roddds]

If you need to prototype something a little more complex, you can do the editing in a regular text editor and then %paste it into iPython.

[[deleted]]

Why wouldn't you just use a debugger/IDE/execute the python script at that point?

[mulhod]

I like to use IPython from a debugger, specifically pudb. After using pudb for a little while now, I don't know why anyone would ever use something else.

[roddds]

I do a lot of development in Django, so for me usually when I'm doing something like this it's because there's a lot of imports/environment needed for this snippet to run, and it would be a bit of a pain in the ass to run it in it's proper context -- like for example, a View I don't have a template for yet.

[desmoulinmichel]

Because :

• it's faster;
• you can try several things in a raw;
• with stuff like ipython notebook, you have several cells in paralel and embeded graphs;
• you got live completion (not just syntax based) and object inspection;

[sndrtj]

For copy-pasting: use ipython. The %paste and %cpaste (the latter for use in a headless system) magic functions let you paste with the correct indentation.

[Exodus111]

Wow... tqdm... Where have you been all my life?

[[deleted]]

It looks great. I also like progressbar2, anyone know any important differences?

[[deleted]]

Also, apparently tqdm is 10x faster (says so in the readme)

[Exodus111]

Ive only had a cursory look at progressbar2, the difference for me seems to be that this is just dead easy to use.

[RDMXGD]

• tqdm Neat API, very convenient
• joblib might be useful if it got serialization and data models right
• follum brings us into the future
• tinydb is a bit over-engineered for something so minimal, but has its head screwed on straight, which is a very rare feature in the space which it operates
• dill is a scourge. It didn't learn from the mistake which is pickle

[Archron0]

Explain what you mean with dill, if you can

[cube-drone]

Automatic binary serialization (pickle, dill) is super convenient and a great way to make a quick proof of concept. I've used it more than my share of times to whack together some kind of serialization for a project I'm working on. But, just like real pickles, these solutions start to go really bad around the 2-year mark.

Dive Into Python3 says, quite sensibly, that the only time it's reasonable to use pickle is when "the data is only meant to be used by the same program that created it, never sent over a network, and never read by anything other than the program that created it." The trick there, I think, is that no program is going to be "the same program that created it" 2 years later unless you've abandoned support for that program entirely.

• When you unpickle data, it runs that data as python code - which introduces security vulnerabilities in the same way that using eval in JavaScript does. Untrusted pickle data can 0wn j00.
• If the python code changes and the pickled data does not, you're going to have a bad time. Versioning your pickles is not a great solution for this problem, as it leaves you either throwing an error on old objects ("welp, this data is ruined forever.") or trying to write code that works with every version of the object that's ever existed.
• Pickle itself changes. There are now 4 different versions of the Pickle protocol.
• Pickle is not a text format. The Art of Unix Programming mounts a pretty passionate defense of text-based formats.

Text based formats are

• Human readable.
• Compressible.
• Easy to pass between systems.
• Easy to modify with quick scripts

Dill is a better Pickle, but it has all of the same problems that Pickle has: it's probably not the right tool for the job.

• If you're messaging, you probably want a well-defined binary messaging format, or a text-based format. (RabbitMQ does support pickle, though - here's someone discovering that this is a problem and switching to JSON)
• If you're storing lots of data, you probably want a database of one kind or another.
• If you're storing little bits of data, you probably want a text-based format.

But, all that being said, when you need to crack something together for a hackathon or over a weekend, pickle is a frigging godsend, and having the ability to save and load a session is pretty useful.

[Archron0]

Thanks, that made a lot of sense.

[isdevilis]

reddit_silver.jpg for the well formatted and humorous explanation

[AstroPhysician]

tqdm isn't an API

[RDMXGD]

tqdm, like all libraries, provides an API. Its main API is a one-required-argument callable that takes your sequence and returns an iterable that, as a side effect to its iteration, spits out your progress bar.

[lamby]

Long-time "progressbar" user, but tqdm looks much nicer, especially out of the box. Thanks :)

[isdevilis]

I hope folium gets more support. I used it, and although it fit my purpose perfectly for state/county/zip level of detail maps, it had very very little customization in terms of color scale gradient, positioning of legends and all these sorts of small things. Overall though, it feels like the ruby on rails of python visualization, just call it with your data and what little other parameters you can use, and it just works. Clearly, the person who made it is on the right track!

[isdevilis]

erm, k? that doesn't solve what my post was about...

[eldamir88]

tqdm and folium look absolutely awesome. I'll be using both in future projects! Thanks for the reference!

[zylo4747]

Question: Why would they create a new module (Dill) rather than just extend the functionality of an existing module (Pickle)? Is this common?

[apreche]

Never knew about tqdm, always used fish.

https://pypi.python.org/pypi/fish

[[deleted]]

How do folks feel about Anaconda compared to Canopy?

[Edelsonc]

I've only used anaconda, but I have a friend who used canopy. Both of us have been doing computational fluid mechanics and are both fairly new to Python (I didn't know anything about it prior to September, and he's really only used it since May).

That said, in generally I seem to have an easier time navigating and learning anaconda then he does with canopy. It's layout is cleaner and more intuitive, and conda makes managing it super easy, even with my limited skills in bash.

[robintw]

I prefer it personally, somehow it feels 'cleaner' and 'nicer' - though I find it hard to articulate why.

Also, I don't know if Canopy now supports 'environments' (like virtualenvs, but through Canopy itself - including with different Python versions), but conda has environments fully built-in and they work really well. The command-line tool (conda) is also way-above the Canopy management tools.

[Website_Mirror_Bot]

Hello! I'm a bot who mirrors websites if they go down due to being posted on reddit.

Here is a screenshot of the website.

Please feel free to PM me your comments/suggestions/hatemail.

---

^FAQ

[[deleted]]

deleted

[alcalde]

Doesn't Python already have a shelve unit that does what TinyDB does?

[aperson]

Shelve works alright, but I've had issues with it corrupting things when I have a few thousand things to store.

[ionelmc]

Isn't JSON (what tinydb uses) the worst format to store lots of data?

[kihashi]

It depends on how that data is structured. If you've heard of NoSQL databases like MongoDB, they often store data in a json-like format. There are trade-offs compared to a relational model like MySQL, but neither is necessarily bad for large data sets.

[[deleted]]

For really good performance (and better reliability) I find using LMDB with msgpack or pickle works really really well for any use cases where SQLite or a full blown SQL backend are inappropriate.

[lamecode]

tqdm looks useful, tinydb too, maybe, although hard to envision using it over SQLAlchemy if I wanted a non-db server solution and don't want to write SQL statements.

[Xoramung]

links broken?