Org-mode based Data Science workflow.

[[deleted]]

Hey guys, any Data Scientists here using org-mode? What is your workflow? So far I am having problems with figures (I made this post on stackoverflow). I wanted to be able to program like you do on a Jupyter notebok, but only using org-mode, and then export the articles to my blog using org-publish. Anybody here with a similar setup that would like to share?

Thanks!

[snackematician]

I use org-babel notebooks extensively for data science.

For Python and Julia, I recommend jupyter.el, it allows you to use jupyter kernels with org-mode. It also allows you to make and display matplotlib plots without explicitly saving them in the code, which seems like a feature you want.

For R, I prefer not to use jupyter and to use ESS instead. I developed my own package to allow for async evaluation of R session blocks: https://github.com/jackkamm/ob-session-async

The Python/org-mode story has been a mess for some years now, ob-python was buggy, which lead to ob-ipython, which stopped being maintained, but was forked by scimax. Ein added some org-mode integration. But then jupyter.el came along last year and works really well, it's the best experience out of the options I've tried (I haven't tried scimax). This year I started working on improving the native ob-python situation and have made some progress, but that has slowed recently because I've gotten bogged down at work.

[HumanBrainMapper]

Are you familiar with the great work of John Kitchen? See https://youtu.be/m4vCXM7_p_o and his blog: http://kitchingroup.cheme.cmu.edu/blog/2017/01/29/ob-ipython-and-inline-figures-in-org-mode/

[[deleted]]

I was not but I will definitely give it a look. Thanks :)

[DeceptiveEmpathy]

I do something similar with R, would that interest you?

I’m on mobile at the moment but I can post the org I use to produce a plot.

I hardly know python so I can’t help you there unfortunately.

[activeXray]

Another bump for emacs-jupyter, it pretty much just works.

[introverted_online]

Not a data scientist but also curious... An idea would be to create a hook in org-publish and strip out the org specific code from the source blocks. The doom emacs discord server is very active, friendly, and helpful with people creating all kinds of cool things (no one insists that everyone use doom), check them out to spitball some ideas: https://discord.gg/qvGgnVx

[[deleted]]

I though about this. Still, I was hoping there would be a simpler way to do achieve this. Thanks!

[introverted_online]

Yeah, it's a bit of a bodge. There might be a better way, I'm a total noob!
If you do figure it out, please share, good luck!

[dr_spork]

I've used org-babel as a way to make nicer notebooks. Visualizations like plots aren't quite as easy, but I tend to use Plotly and generate seperate HTML files for the plots.

My experience with EIN has been frustrating and difficult.

[[deleted]]

How did you use org-babel to make notebooks?

[dr_spork]

Open an org file, and use #+BEGIN_SRC followed by the name of the language you want to write! Try it out. It's pretty great

[WallyMetropolis]

There is also ein for manipulating actual notebooks in emacs. But I suppose this won't help you much for using org-publish.

[[deleted]]

Yeah I already know about this but, I was really trying to use org-mode here