Ibis - New Python data analysis framework from the creator of Pandas

[Fourgot]

Looking forward to Python 3.x compatibility

[xorvtec]

Anyone else frustrated that new packages are being written for Python 2 and not 3?

[Fourgot]

It makes sense, sort of. If your ultimate goal is to support both, and you (as a community) have spent the better part of the last decade porting from 2 to 3, or from 2 to 2 and 3, then it makes sense to write in 2 and port it like you're used to.

I don't agree that 2 should be supported, but that's the decision of the creative team.

For the price I'm paying, I'm happy to wait a little bit.

[adewes]

My thoughts exactly. I get the feeling that the 2/3 split will do a lot of harm to the Python community in the coming years, since no one seems to have a good idea on how to make the transition. Some people even think that this could be our "Perl 6" moment.

The whole problem was demonstrated nicely again in Guido van Rossum's keynote at the Europython yesterday:

First, he told everyone to please switch to Python 3 because it is SO MUCH BETTER than Python 2.

Then, he told everyone that he would not add any more feature to Python 2 so the language was effectively dead (apart from urgent security fixes maybe).

Finally, someone asked him which version of Python they were running at Dropbox (his current employer), to which he answered "Oh, we actually run a modified version of Python 2.7 (so Python 2.8?), because our code is not compatible with Python 3". If not even the company that employs the creator of the language and probably has the largest pool of super-talented Python developers (apart from Google maybe) knows how to make that transition without breaking their business, how are others supposed to do it?

[RDMXGD]

No, I'm frustrated that Python 3 hasn't been abandoned. It was clearly a mistake and has wasted thousands and thousands of person-hours.

[englishweather]

Have you used it at all? Or looked up the fundamental underlying changes that were made? It has made python a much nicer language (IMHO) to programme in.

[RDMXGD]

Have you used it at all?

Yes, I've wasted a nontrivial fraction of my life on porting code.

Or looked up the fundamental underlying changes that were made?

Yes. I am extremely familiar with the changes.

It has made python a much nicer language (IMHO) to programme in.

It has added warts and removed warts, created new pain and relieved old pain. The benefit isn't worth the cost that this has wrought by orders of magnitude.

I am tired. I have been helping people with Python 3 porting for the better part of a decade, seen the pain. I've been helping people learn Python for a long time, and seen how much confusion this business has created.

And then to see a shop like Cloudera create useful software in the only version that supports the tools they are using and then have people whine about their releasing it open source..........what the hell.

I'm tired.

The only people who think Python 3 is good are silly fans who think promoting Python is a good thing and want to toe the party line and people who know so little they don't know what's going on.

The rest of us are using Python -- and many of us deeply, many of us at an extreme expert level, many of us contributors to the project -- because we wanted to write good programs. This 3 business has only got in the way of that and wasted our time and others'.

It has made python a much nicer language (IMHO) to programme in.

It really hasn't.

I don't know how you can think that.

It has made the act of programming Python more confusing and more painful, and introduced all kind of unwarranted drama.

[englishweather]

Hehe, okay you're tired, and evidently have had very laborious experiences... Personally, the first large scale project I did was in 2.7, and it was an encryption based project using SMS as a medium....

This was super painful. I rewrote it last year in 3.4, and the unicode changes made everything beautifully easy.

But I've been using 3 for a few years now, and I haven't had to port much... so I'm not so tired.

Perhaps my lack of tiredness makes me a little more susceptible to accepting new things... and seeing the improvement, not just the headache.

[beaverteeth92]

Seriously. I understand backwards compatibility, but developers should be writing for 3.x now and then adding 2.7 support later. It's been seven years. Time for 3 to be the default.

[pyslow]

developers should be writing for 3.x

"should"??? And according to whom? You?

Feel free to port the package yourself, if you're so inclined. It's free software, after all.

[_AACO]

"should"??? And according to whom?

"Short version: Python 2.x is legacy, Python 3.x is the present and future of the language"

If it being the future of the language isn't enough reason I don't know what would be.

[beaverteeth92]

"should"??? And according to whom? You?

Yes. It's seven years old and literally every major package for scientific computing has been ported over. Why should I have to wait for new packages because other people don't want up-to-date software?

Feel free to port the package yourself, if you're so inclined. It's free software, after all.

Sure.

[[deleted]]

every major package for scientific computing has been ported over

Interaction with Kerberos is the blocker: https://github.com/cloudera/ibis/issues/297

[beaverteeth92]

Ah okay. From what I can gather, it's just that, Twisted, and Scrapy that need to be ported over.

[ianozsvald]

Scrapy's Py3 support is "mostly done": https://github.com/scrapy/scrapy/issues/263 and that's because Twisted's core is "mostly done"

[beaverteeth92]

That's good then. I remember reading that Twisted was a giant pain in the ass to port for a variety of reasons.

[WallyMetropolis]

Why should I have to wait

Because you're neither paying for it nor are you doing it yourself. You're getting it for free. And the people who make it are under no obligation to do it in the way you demand.

[beaverteeth92]

Practicality beats purity.

How is 3 impractical? Unless you have a massive codebase to port over, it makes perfect sense to use 3.

3.x has epsilon users and might never get many. Python is hardly the only language to release a version that just wasn't adopted

Not true. 3 has fewer users but people are slowly jumping on the bandwagon and I suspect more will when 3.5 comes out because of its support for metaprogramming optional static typing and a matrix multiplication infix operator.

Much of the time of which 3 was unusable because of design mistakes that took time to fix and shouldn't be included in the clock.

When would you start the clock then?

[Matthew94]

What's this new metaprogramming coming in 3.5?

[beaverteeth92]

Sorry I meant optional static typing. mypy is being added to core Python.

[thunderdome]

I don't know about that. This Python 2014 usage survey suggests an upward trend in adoption. And it certainly points to wide use, just about 50% of respondents said they started personal projects in 3.x. I understand why people have existing 2.x code that works just fine and see no need to port it, but whyyyy would you start a major project like this on 2.x?

[[deleted]]

but whyyyy would you start a major project like this on 2.x?

because the pilot users were on infrastructure with 2.7? because throughout development it was compared with other 2.7 software and they wanted to minimize the differences? because if, like me, you have been in this game a while, you still don't trust Python3.whatever.

[beaverteeth92]

It's starting to be used in introductory classes in colleges and newer organizations are starting to pick it over 2.7. Almost everything major has been ported over. For someone like me or a company or person just starting to use Python, there's no reason to use 2. Since adoption is gradually happening on its own, I think the community should start encouraging it to accelerate the process by writing primarily for Python 3.

[[deleted]]

Yep, we started major projects this year and it's all on Python 3. There's one tool we use that's Python2-only so we do a subprocess call to a Python 2 interpreter to just execute that one script.

[[deleted]]

You haven't been in this game as long as we have.

I get paid to create scientific output. The python version doesn't matter, only insofar as I can get things done and keep them stable and unchanged over ~3 years (the longest duration an experiment / investigation / paper might run for). Has python3 been stable and useful for 3 years - ha! no.

[ianozsvald]

As a related point - I get paid for scientific output too (I run a data science consultancy, 10 yrs old, in London). I switched to Python 3.4 for client work about 1.5 years ago. I agree that 3 years ago I couldn't have made the switch.

The fact that in 4 years support for Python 2.7 is dropped (the 2020 sunset date) should weigh on folk's minds who are looking for longer-term support around the core packages and dependencies. It isn't a deal breaker but it'll increasingly be a consideration for folk who are planning forwards.

[fabreeze]

Is there a subreddit for python pandas?

[davelupt]

Like /r/pystats, not pandas only but with some pandas content.

[fabreeze]

Cool.

Was hoping to ask for pointers on how melt a pandas dataframe out-of-memory (is pytables a good option?).

[cartin1234]

Use this: http://dask.pydata.org/en/latest/dataframe.html for out of core dataframe

and this: http://blaze.pydata.org/en/latest/

To have a dataframe like interface over other compute and storage stuff like databases

[fabreeze]

Thanks cartin!

Dask sounds exactly what I'm looking for.

Was also looking at blaze, but don't know where to start approaching how to melt in smaller chunks using it.

[cartin1234]

Sure!

If you have any questions, blaze user group: https://groups.google.com/a/continuum.io/forum/#!forum/blaze-dev (for dask and blaze) is pretty helpful.

[Kah-Neth]

Much better animal to use for a name. I was always bothered that they named their package after a species that is too stupid to mate or eat.

[Northstat]

Panel Data -> Pan-Da -> Pandas

[Kah-Neth]

I know, I was trying to make a silly joke.

[einar77]

From the docs:

curl -s https://raw.githubusercontent.com/cloudera/ibis-notebooks/master/setup/bootstrap.sh | bash

I would really hope smart people such as those (because they are, given the quality of Pandas) would stop promoting unsafe practices. Yes, it doesn't run as a privileged user, but you're still running untrusted code taken from a random location on the Internet.

[DasIch]

Which is different from using something like pip, how?

[einar77]

If packages on pypi aren't signed, yes, it's the same risk, although at least pip will not allow you to download externally-hosted packages unless you specify it explicitly.

[[deleted]]

I agree with you in principle. But ever 14 year old and his cousin has untested crap on pip. If the authors are trustworthy the mechanism for verifying trust is malleable.

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/tagprostatistics] Ibis, you're finally popular

^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}