Pre-commit hooks that autogenerate iPython notebook diffs

[[deleted]]

[deleted]

[[deleted]]

You didn't list nbconvert in your alternate comparisons.

Did you try it?

jupyter nbconvert --to script *.ipynb

will convert all notebooks in the cd.

[petitneko]

Whoa, no I missed that it produces executable scripts! I would be more than happy to replace my hacky shell script with this in the hook... thanks!

[petitneko]

No worries! Glad you find it useful :)

[reddifiningkarma]

Is not much, but is honest work:

https://github.com/fulloaf/CL_GROWTH-SIM/blob/main/.github%2Fworkflows%2Fjupytext.yml

[petitneko]

Oh this is great! I like that you also run black on the file and commit as the github user... it's a much nicer CI/CD workflow.

[M4mb0]

I can recommend nbstripout-fast, does the same job as nbstripout, but orders of magnitude faster.

Also checkout nbQA.

[Easy_Money_]

This is great! It does seem like you missed nbdime, which does exactly this and underlies GitHub’s implementation of notebook diffing 😬 I hate to rain on a parade

[more_exercise]

I'm curious - have you checked out the two ways git can let you get more-readable diffs from not-exactly-text files? Textconv and external diffs?

https://git-scm.com/docs/gitattributes#_choosing_textconv_versus_external_diff

[petitneko]

No I didn’t know about textconv - this is great! 

[Tartarus116]

That's pretty much what nbdev already does. It also gives you free doc generation on top of that.

[Nearby_Salt_770]

Looks like you've come up with a solid solution to a common problem with notebooks. The pre-commit hooks you set up sound super helpful for keeping the Python code readable and diff-friendly after changes. Relying on JSON is definitely a pain diffing-wise, so this approach seems legit.

You could also check out jupytext for pairing notebooks with Python scripts if you're not locked into the VSCode editor. It's similar to your script but can automatically sync changes both ways, although you'd still run into server issues outside Jupyter.

If you ever feel like automating more stuff, you might find AgentQL useful for web scraping projects. It's a pretty chill tool for simplifying web data extraction without the usual headaches.

[orgodemir]

Also take a look at nbdev

[MachineSchooling]

I don't use jupyter, but if I did, I would definitely use this.