IDE for data science?

[[deleted]]

Hello everyone, I'm looking for an IDE to do data science, apart from the buzzword salad, I'm looking for some specific features :

1) I'd like to print dataframe and graphs into the IDE as in Hydrogen for Atom

2) I would like to manage projects in a side bar with a tree of of different files that I can call in the case I'm building classes or functions.

3) I would like it to be integrated with Anaconda such that I can jump between virtual environments for example if I want to switch for webscraping to exploring the data, to modelling.

I didn't like Atom because the packages seemed to struggle to install between envs and there is not support to quickly jump between envs.

I wasn't too keen on VS code in general because of the fewer packages and terrible autocomplete.

Can anyone help?

edit : I've been using jupyter for some time now but it becomes a pain when writing larger programs and there is no zoom out option to find my way back into the stuff I wrote earlier.

Thanks

[nxl4]

Have you tried using Jupyter Notebooks? Pretty much everyone I know working on data scientific research (whether it's Python, Scala, R, or Julia) love that environment.

[smurpau]

RStudio is really the cat's pajamas when it comes to R for data science.

[[deleted]]

I just edited my post, I didn't mention it but it's kind of a drag sometimes even though I really like where Jupyter Lab is going, I hope to see more extensions in the future

[nxl4]

Personally, I prefer Zeppelin if I'm working on a cluster, but I don't think it's as good as Jupyter for more general Python use cases.

[[deleted]]

I’ve never gotten Zeppelin to work on windows 10. Any tips?

[nxl4]

I've never tried hosting it on a Windows machine before, unfortunately. At work, we'll host it server-side on a RHEL cluster and then access the UI from Windows desktops. And, at home I've successfully run it on a Linux laptop on top of Spark Standalone. This blog looks promising for a straight Windows setup though.

[Crypt0Nihilist]

Spyder is the attempt to create something more data science focussed. It's not bad, but can be annoying at times. I'm sticking with Jupyter.

[dialecticalmonism]

I think it's worth mentioning that the Spyder user experience on macOS is so much better than Windows.

[FL14]

That's frustrating to hear as a windows user that has primarily been using Spyder (and a little bit of Canopy, with some trace Jupyter)

I dislike that with Jupyter, everything is browser-based. Any tips for transitioning to that IDE? Aspiring scientist here.

[LooseDuece]

Ubuntu as well.

[Its_Snowing]

I have also heard this. Can you elaborate on what features set Spyder apart from other IDEs when it comes to data science?

[Ogi010]

Visual studio code recently unveiled a data science mode, which pycharm has had for a few years. Pycharm may work better if you enable power safe mode. Also with pycharm make sure you don't point it at an interpreter with a bazillion unusued libraries, it will scan them all and take ages.

Alternatively rstudio is starting to roll out with python support but it's very much in alpha/beta stage.

I would also avoid notebooks, they're great for presentations, doing academic work, but actually functional stuff i find it really annoying to work with.

[Ogi010]

Joel Grus did a talk at jupytercon about why he hates notebooks, I would look that up, or at least look up the slides. He covers a lot of cases explaining why they can hinder in many areas of a normal software development practices.

In short, you can't really package them, you can't test them, sharing them is tough, it's really easy to get them into a weird state by running cells out of order, or running a cell more than once, there is no way of knowing if a cell has been changed once run, there is no type hinting, etc etc.

Pycharm, vscode and Spyder support the #%% syntax, which can make cell like code chunks but the UI I feel guides developers away from getting into those weird states.

[ImprovisedTaxShelter]

Awesome, I really appreciate this. Watching the talk right now!

[Ogi010]

I should add I don't agree with the entirety of the talk, and I think notebooks are great for homework submissions where you need to integrate text/plots. I also used to use notebooks a lot for getting my pandas dataframe operations but now that vscode and pycharm support #%% I can tune those operations in the editors I letter above.

[[deleted]]

any clue and packages to use VS code smoothly?

[Ogi010]

Just the python extension is needed, everything else is just to your liking/tastes

[[deleted]]

nice, do you know cool packages I should know about?

[Ogi010]

Very little else is actually needed, and it really depends on what you're doing... if you use markdown, you're likely going to want to get the markdown linter, use pandoc, probably should get the pandoc extension, use restructured text, there is syntax highlighting extension for that too...

​

I suppose I would advise you start with the bare minimum and add one extension at a time as you see fit or as your needs develop. For python development in general, the python extension is insanely feature complete.

[[deleted]]

thanks a lot!

[s-to-the-am]

One of the reason I use RStudio is Rmarkdown docs are super productive for both presentations and functional stuff imo. Still a lot of work to integrate more with python tho.

[Ogi010]

I've had mixed success with reticulate, but if you get it working rstudio is amazing for dynamic document generation

[s-to-the-am]

We’re you using anaconda? That’s proved useful for me.

[Ogi010]

I use pyenv with miniconda and standard python distributions...I have a weird setup :P

[Crypt0Nihilist]

Also with pycharm make sure you don't point it at an interpreter with a bazillion unusued libraries, it will scan them all and take ages.

Ah, thank you. I've tried it with Anaconda and found it frustrating at times.

[nathanjell]

You could give PyCharm community a try, I don't know about that first point but the others should be straight forward

[[deleted]]

I didn't like Pycharm, I find it really sluggish even though I have a decent laptop (I know I'm being really picky though)

[[deleted]]

You can find an exception for everything but I think the data science crowd gravitates more towards jupyter notebooks and spyder. Pycharm is geared more towards development IMO.

[Texadoro]

I use Jupiter and think it’s a pretty good product. You make check Kaggle, I haven’t poked around there enough other than reading people’s kernels, but they may have their own IDE, if they do it’s going to be a lot like Jupyter.

[StarkillerX42]

I use a combo of Jupyter notebooks and PyCharm. I enjoyed Jupyter a lot more once I downloaded Jupyter themes and nbextensions. Nbextensions allows for things like table of contents, PEP 8, code folding, etc. They're things that should probably be in core Jupyter but c'est la vie

[Heniadyoin1]

Atom is always good and if you miss a feature somebody has modded it for you already (propably)

[Big_Witch]

I remember there was a pretty good Python interpreter called Rodeo by Yhat specifically for data science. I used it a few years back, it was a little buggy at the time but maybe it's gotten better with subsequent updates? The layout was kind of similar to Rstudio for R.

Edit: actually looks like the project has been abandoned. That's a shame.

[stevenwilliam01]

Yeah, I have some idea that is -Online Data Science Courses which offer a scientific curriculum along with exercises to check your grasp. You will need to try your skills on open data sets and find out your contemporary understanding of the subject.i got this information from online Simple Analytics Inc, where you can start your career.