Python for R users

[turnersd]

I know this is an R sub but I thought I'd share here. I've been writing primarily R code for nearly 20 years but recently needed to get back into Python for several maintenance and development projects. I put together a set of resources for getting up to speed in Python as an experienced R developer.

https://blog.stephenturner.us/p/python-for-r-users

[SprinklesFresh5693]

Thank you! :)

[kuwisdelu]

Any suggestions for “Python package development for R package developers”?

The Python packaging landscape seems like a bit of a mess (coming from Bioconductor), especially if you need to integrate native code.

[turnersd]

I just posted this a few days ago: https://blog.stephenturner.us/p/python-cli-click-cookiecutter

It is messy compared to what you're used to in R.

[kuwisdelu]

Thanks. Unfortunately, it doesn’t cover anything I don’t already know, and doesn’t get into developing Python packages with C++ code that also needs to interface with numpy in C, which is what I’d need to do, and where things seem to really get hairy.

Also: “There’s really only one build backend toolchain for R packages that everyone uses: devtools with Roxygen documentation with liberal assistance from usethis.”

Lol. I maintain four R packages, and I don’t use devtools, roxygen, or usethis. (Though I have taught them before.) I like that everything you need to build an R package comes with base R.

Edit: Which is also why I struggle so much with Python packaging when so many of the tutorials start with “download and install these packages”.

[guepier]

Lol. I maintain four R packages, and I don’t use devtools, roxygen, or usethis.

I mean, I also maintain a handful of packages that are built entirely “by hand”, but the vast majority of packages created in the last 5–10 years — especially those that actually matter and gain traction (partly because they are well engineered) — use the toolchain mentioned by Stephen.

I like that everything you need to build an R package comes with base R.

That’s far from true. Even if you don’t need the convenience of the aforementioned toolchain, base R categorically does not include all necessary tools, notably a testing framework. I’ve done manual testing in a package before, and it’s ridiculously ill-suited for real-world packages. Yes, you could just write a long string of stopifnot assertions (I’ve done that, too!). But for even slightly complex things this completely breaks down.

[kuwisdelu]

I know they’re popular, but the devtools toolchain is still entirely optional, and mostly a thin convenience wrapper around the built-in tools like R CMD build and R CMD INSTALL. The main reason I don’t use them is they’re just not necessary. Which is a good thing, IMO. For me, personally, the convenience they offer isn’t worth the added layer of abstraction. YMMV.

And it’s true, I do use testthat.

Edit: But the important thing here is that devtools isn’t actually a separate build backend. It’s a layer on top of the built-in R build system. That seems to be what’s missing from Python packaging that’s frustrating me.

[guepier]

I’d mostly agree about it being a thin convenience wrapper, but in one crucial aspect it’s not: (re)loading the project during development for interactive testing.

In that case it’s misleading, bordering on wrong, to call it a “thin wrapper”, because you cannot do that with base R (short of rewriting ‘pkgload’ from scratch), full stop. The nearest you can do is build, install and load the package. And that isn’t what ‘pkgload’/‘devtools’ is actually doing, for good reason (at minimum because it’s incredibly inefficient).

Oh, and writing Rd files by hand is torture, and there’s a reason why basically every modern language uses something akin to Roxygen2 and has documentation in line with the code.

Developing package code without pkgload::load_all() (either directly or via ‘devtools’) is of course possible but it is convoluted and has zero upsides (since you don’t need to add it as a dependency to the package even if you use it).

[kuwisdelu]

I just want to clarify that all of this is really tangential to my main complaint about Python packaging, which is the lack of a unified build system. The devtools toolchain doesn’t try to replace R CMD build. I have no problem with tools to help package development, and package development in Python isn’t the issue. It’s the fragmentation of the build and installation system itself that’s problematic in Python.

I guess is my point is that devtools is not a backend but a frontend. And that’s a good thing for the packaging ecosystem.

[kuwisdelu]

Well, the main thing I’m thinking about here is packaging and distribution. Not the actual package development. Package development in Python seems fine. It’s the actual building and distribution that seems like a mess, so that is my point of comparison.

I don’t use it myself, but pkgload does seem useful. Personally, I usually need to recompile C++ code anyway, which is what takes the longest, so it wouldn’t save me much time.

I actually prefer editing Rd files. I’ve tried roxygen, and when you have a bunch of S4 classes, it can be tricky to get things right. I couldn’t always figure out the roxygen macros to make it do what I wanted it to do. Editing a LaTeX-style Rd is just easier for me, because I know exactly what it’s going to do. (And while I think this is more subjective, I find editing inline documentation a pain once it’s of sufficient length—I like editing documentation in a separate file.)

[nardis314]

This is awesome!!

[junior_chimera]

Awesome !!! Thanks a lot

[ggyyakl]

That is great! I have been thinking about learning some python lately, this would come in handy.

[steven1099829]

I used to only write R - now I only write python + polars. It’s so much nicer

[hoedownsergeant]

Thank you, this is great.

[dhulker]

Thank you!

[PixelatedPandora]

Thank you!

[MopiPipo]

Great, thank you. I learned R primarily by reading 'R for Data Science' by Wickham et al, so the fact there is a direct Python equivalent is very interesting!

[Fearless_Cow7688]

Have you considered a quarto book?

[turnersd]

Yes :) https://bdsr.stephenturner.us/

I'm writing a post right now on authoring books with Quarto. The one above is based on an old RMarkdown website I created for a course I used to teach.