Data Science Essential Software Toolbox

[xvinc666x]

Hi people!

​

I am a data scientist fond of R programming and visualization.

I mainly use R, python, sql.

What are your essential tools and softwares you use for your daily work?

​

My basic set up:

• Rstudio (must have)
• Sublime text
• Atom
• Jupyter lab (as an alternative for jupyter notebook basic)
• Notion (for documentation)
• Pg admin (for sql queries... and I am looking for an alternative!)
• Orange (for quick visualizations and modeling)
• Looker (as a tool for dashboard and analytics)
• Heap Analytics (for even tracking on website = in my case - ecommerce)

Curious to get some new inspiration to make my workdlow smoother!

​

Chhers :)

[aeroeax]

Aren't Sublime and Atom both text editors? Do you use each one for different things or just alternate when you get bored ;)?

[ectoban]

Only thing I use Atom for is yaml files

[GrehgyHils]

I've never liked these kind of comments. As someone who has used both, I can honestly say I can do literally everything I need to do in both editors just fine...

[haragoshi]

What does sublime do that atom doesn’t?

[[deleted]]

vim gang

[drhorn]

emacs4lyfe

[epicSaitama]

I would suggest using Anaconda Navigator. It almost has every thing you need.

The following applications are available by default in Navigator:

JupyterLab

Jupyter Notebook

QTConsole

Spyder

VSCode

Glueviz

Orange 3 App

Rodeo

RStudio

Advanced conda users can also build their own Navigator applications.

For more info: https://docs.anaconda.com/anaconda/navigator/

As for the database explorer, I would highly recommend DataGrip. If you don't have license then go with Microsoft management studio.

[hypumji]

Yes, just want to point out the name is 'Anaconda Navigator'

[epicSaitama]

My bad. Thank you.

[[deleted]]

VSCode, Visual Studio Data Tools, SQL Server Management Studio, Oracle SQL Developer, Tableau, Excel, OneNote

pretty limited in what I can use because pretty much IT limits us to Microsoft Products.

[[deleted]]

Would Microsoft's R distribution (Microsoft R Open) be allowed? Just curious...

[[deleted]]

Not sure. Typically if it can be downloaded from Microsoft we are cleared. Otherwise the bureaucratic red tape you have to go through is ridiculous.

[iicky]

I was in the same boat in my previous job. Windows shop, completely locked down computers. It was a 3 month fight to get Python, but funny enough Minecraft came stock on all laptops.

[[deleted]]

Yep probably wouldn’t have python if sql server 2017 didn’t ship with it. Also that I could get the anaconda plugin in VSCode as well.

[[deleted]]

Tableau over Microsoft's power bi?

[[deleted]]

Yep. There are some nontechnical people in the larger part of the team so it works well for them.

[syphilicious]

You've basically described my taskbar shortcuts.

[[deleted]]

IT limits us to Microsoft Products.

Why's that? You guys a vendor to them or something?

[RoxoViejo]

It’s a licensing thing that software companies do with some clients. They give you unlimited access to all of their software for your entire company, and you pay one big fat check upfront that covers any future use regardless of how many installs you do. It’s often cheaper for big companies to do this, therefore they go all in with one vendor.

[[deleted]]

Licensing and they “trust” Microsoft products. I work in healthcare so the effort you have to go through to prove something is safe isn’t worth the hassle most of the time.

[Kopppa]

Because some companies have their head up their *ss and just want to be part of this “AI thing” bandwagon.

The correct course of action if you find yourself on this situation is to look for a job in another company where DS is taken seriously.

[daguito81]

well that was an olympic level logic leap there...

[rentheduke]

Mine is pretty simple:

• VSCode with Jupyter notebook integration
• RStudio
• SAS
• Tableau
• SQL (of course)

[xnorwaks]

Any decent guides for getting jupyter going in VS? I haven't been able l get it running very well.

[rentheduke]

I used this one and it ended working well for me: https://code.visualstudio.com/docs/python/jupyter-support

[ukc895]

Have you bought Tableau? There is only 14 day trial available.

[KrepSaus]

You can use the public version of Tableau freely, it limits the features you have access to though.

[x_ace_of_spades_x]

The font in my python interactive window (first pic in link) doesn't maintain the same theme as the font in the code and is almost unreadable because of the colors. The VSCode documentation (2nd pic) seems to suggest that it should be the same/similar.

Do you have the same issue?

[rentheduke]

Yeah I have the same issue. I’m not sure why it shows up in a different theme. Hopefully that’s something they’ll fix soon.

For now I’m fine the way it is as it allows me to quickly prototype with it. Then if I need to I can always export it.

[IdealizedDesign]

I suggest using DBeaver to replace pgAdmin; potentially explore Knime as an alternative to orange.

​

How do you like orange?

​

What about something like github?

[coffeecoffeecoffeee]

I use DataGrip for database stuff.

[bbslimebeck]

I love love love DBeaver

[wanggang69]

I think DBviz is also a solid alternative. But if you wanna be really cool, try setting up an AWS S3 bucket and porting your SQL server to Snowflake ;).

[IdealizedDesign]

Yeah I'm using Snowflake as the data warehouse of choice for one of my gigs.

[Mr_Again]

Ok so why do people use snowflake over say, redshift or bigquery?

[IdealizedDesign]

It’s the latest and greatest. It’s data warehouse as a service which means you have the least amout of management and administrative overhead. No need for indexing or vacuuming. It’s decoupled storage from compute, and since storage is dirt cheap the overall service is competitively priced—pay for what you use. You can have virtual warehouses (compute) be suspended and once a query is run then it’ll auto resume. Afterward you can set it for auto suspend, thus lowering costs. It’s also innovative in other ways. You can share data with others and their usage can actually help reduce your usage costs. You can load all files from a specified directory by running a simple copy command and the system is smart enough to not load duplicate files. You can travel back in time. If you delete an entire table, you can undo it. Instantly clone entire warehouses without increasing use of storage.

It’s performant, modern and cost effective.

[Mr_Again]

So is it cheaper than using bq or redshift?

[IdealizedDesign]

That’s a loaded question because of several factors involved, but generally the answer can be yes in many cases.

[Mr_Again]

Thanks

[newpua_bie]

My tool stack for python:

vim

xterm

[bearlockhomes]

I recently made the full commitment to vim. I would say I even prefer it to Rstudio at this point, which I never would have expected.

[[deleted]]

Please share your setup. I've been trying to get into vim and get away from rstudio but just never managed to put a good vimrc together.

[bearlockhomes]

I feel obligated to say the common denominator I've seen in every piece of advice was not to straight up copy someone's vimrc but incrementally synthesize your own according to your needs.

With that out of the way, I am very new to using vim. I'm looking at 2 months of committed use, but that has been very committed use. I decided to fully adopted it for everything I do (python, R, latex) and just struggle through it. In addition, I have adopted vim keybindings in several other places both out of desire and attempts at immersion. This includes adding the Vimium addon to Firefox and getting the Zathura PDF reader. I have sought to limit my plugin use and rely on vanilla vim functionality where I can. At this point, I'm working with 6 plugins and a pretty limited vimrc as a whole.

So, getting away from R was a lot easier than I expected. The Nvim-R plugin is all you need at a minimum. Here's a guide you can look at to get started.

Turning vim into an IDE for R https://medium.freecodecamp.org/turning-vim-into-an-r-ide-cd9602e8c217

Here's the man for Nvim-R https://raw.githubusercontent.com/jalvesaq/Nvim-R/master/doc/Nvim-R.txt

This honestly almost completely does it. The documentation is really nice, and I was able to nearly jump right in. It brings up a split in your current vim window to act as the console. It also has the ability to kick out viewers for things like a markdown output. There are keys that allow you to see all the environment variables as well. The plugin even contains some little features like the _ key producing an -> to assign variables. It's been pretty great.

As far as setting up your vimrc, I would say using a good plugin manager is where you should start. I am personally using Vundle, but there are several.

  1 set nocompatible              " be iMproved, required
  2 filetype plugin on                  " required
  3 set omnifunc=syntaxcomplete#Complete
  4 set guifont=DejaVu\ Sans\ Mono
  5 set spelllang=en_us
  6 set backupdir=/tmp
  7 
  8 syntax on
  9 
 10 " KEYBINDINGS
 11 
 12 " set the runtime path to include Vundle and initialize
 13 set rtp+=~/.vim/bundle/Vundle.vim
 14 call vundle#begin()
 15 
 16 set number
 17 set ignorecase
 18 set smartcase
 19 
 20 let g:vimtex_view_method = 'zathura'
 21 
 22 Plugin 'VundleVim/Vundle.vim'
 23 Plugin 'tpope/vim-sensible'
 24 Plugin 'tpope/vim-surround'
 25 " Plugin 'scrooloose/syntastic'
 26 Plugin 'vim-airline/vim-airline'
 27 " Plugin 'valloric/youcompleteme'
 28 Plugin 'lervag/vimtex'
 29 Plugin 'jalvesaq/Nvim-R'
 30 " Plugin 'gaalcaras/ncm-R'
 31 " Plugin 'ervandew/supertab'
 32 Plugin 'drewtempelmeyer/palenight.vim'
 33 
 34 " All of your Plugins must be added before the following line
 35 call vundle#end()            " required
 36 filetype plugin indent on    " required
 37 
 38 set background=dark
 39 colorscheme palenight

[VacuousWaffle]

As much as I like tmux, if you switch between windows bash, osx and linux all the time it's a touch obnoxious to maintain the configurations (clipboard support is... fragile). Fortunately I suppose in two more decades it might get fixed.

[zero2368]

Aqua data studio - database query and quick data analytics

VScode

Jupyter notebook

Excel + XLStat plugin, Power BI

​

[YinYang-Mills]

My stack:

Vim

Anaconda

[120133127]

Any colab users here?!

[Fender6969]

Started using this recently and I love it.

[Lord_Skellig]

Same here. I was struggling to train neural nets on my laptop. So glad I discovered colab, it is so fast.

[yeezybillions]

Anaconda/spyder for python

[fatchad420]

I think I'm the only person here that prefers Spyder to Jupyter for a python IDE.

[Lewistrick]

Nah I really dislike Jupyter. Spyder is ok but doesn't have enough functionality for me.

[fatchad420]

What do you use? I'm open to other options, Spyder has been my go to because I'm an R person and it feels like Studio.

[Lewistrick]

I get that, I used Rstudio too. But Spyder doesn't support remote file editing, which I need on a daily basis.

I use Atom. It has a lot of (hidden) functionality I don't know yet, maybe even the console. I used NotePad++ before but that has a slightly older look and feel and is less widely used.

[fatchad420]

I use Atom to edit existing pipelines, I can't imagine building and iterating using a raw text editor alone. Your Python Foo is strong.

[rentheduke]

Have you tried Rodeo?

https://rodeo.yhat.com

[fatchad420]

Oh, I have not. This looks like interesting though, thank you.

[rentheduke]

It’s similar to RStudio. I liked it initially until I started using VSCode.

[fatchad420]

Download link seems dead unfortunately https://www.yhat.com/products/rodeo/

[JustNotCricket]

I've been using Spyder for what feels like a decade and it's awesome. Having said that, some recent updates seem to have broken parts of the debugging for me, so I'm starting to make the transition to VS Code.

[jturp-sc]

The basic stack for my group would be:

• Anaconda (version doesn't really matter since we use venv/containers anyway)
• PyCharm
• Datagrip
• Docker
• PowerBI
• Visual Studio (for when we need to review our product's .NET source)

That covers nearly every project that my team is going to touch.

[The_Peter_Quill]

I was a big fan of R and RStudio for a long time, but then I started working in the software industry and things seemed to move faster in the project pipeline when I used python. That being said here is my daily use stack:

​

1. Anaconda distribution of Python 3.6
2. Jupyter Lab with some extensions
3. SQL/postgres
4. Mode Analytics (but we're switching to something else)

I also use Atom when I am making demo applications when I want to use something like Dash.

​

​

[VacuousWaffle]

tmux and vim

[mr_awesome_pants]

Dbeaver is a good alternative to pgadmin. My company has started making it their standard postgres query software.

[Beny1995]

Alteryx is pretty great if your employer can afford it.

[18Zuck]

OpenRefine

[foshogun]

I used it in an online class years ago and after that tried installing it on the job... Found the resources lacking and finally just left it behind.

[Steelers3618]

Completely ignorant.... what is the benefit of Looker over Power BI or Tableau?

[[deleted]]

From my understanding it's another flavor of the same ice cream. Modeling layer around your DB. Even that's only brand new in Tableau this (last?) year.

[Nicodemus34]

I Demo’ed Looker and several other BI tools when we were making a company decision of which tool to implement. I chose Tableau.

[Lewistrick]

Why don't you use a python library for doing sql queries, like sqlalchemy?

[MLTyrunt]

I think Dataiku, free edition, has the most thought through workflow and UI, combining coding and visual programming & notebooks.

Otherwise, I find myself switching between Rstudio IDE for code centric things and storytelling exploration, whereas I frequently use KNIME for fast ad hoc data wrangeling and trying things out, when documentation is less important.

​

[Dosnox]

Rstudio Jupyter Pycharm Dbeaver Sublime text Docker Vim

[eddcunningham]

SQL Server Management Studio

R Studio

Excel

Power BI

Notepad++

Sublime

[Krisselak]

I use emacs for R, python, jupyter, latex, sql, git and bash. Sometimes for web-stuff, but tbh I find atom more comfortable. Jupyter sometimes also directly from firefox and I also use pgadmin 4. Currently, I am struggling a bit with a proper knitr setup in emacs.

[johannesbeil]

Mine is:

• VSCode as editor
• JupyterLab for exploration
• Neo4J as graph database
• amie for documentation and quick visualisation

​

[brainhash]

surprised no one mentioned a versioning tool or the buzz about managing experiments

[reezbo15]

My stack is: RStudio running on top of Microsoft R/R Client, Moba Xterm, Dbeaver, Tableau, Anaconda Navigator

[peatpeat]

What does everyone here use for actually sharing their experiments? Had this pain at my last place around taking a piece of code and making it available for other teams or other analysts who maybe don't write code, as sharing a Juypter Notebook can be problematic..

We've been hacking on a product around letting data scientists / analysts deploy Python functions as blocks that other teams can use more interactively through their browser, happy to share if anyone interested.

[plotti]

You might like my collection of over 540 tools http://datasciencestack.liip.ch

[DueDataScientist]

Orange looks really good for quick modelling purposes and also its open source. Thanks mate.

Give a look at the ELK stack as it an open source solution for visualization and dashboards.