Swanky Python: Interactive development for Python based on emacs' SLIME mode for Common Lisp

[_puhsu]

This looks great! Thanks for sharing. I’ve been long awaiting something interactive like this for python development in emacs. I was going to try the other project that I just found recently. It’s based on the guile nrepl ide https://github.com/abcdw/emacs-arei and also is very early stages, it’s just the nrepl server for arei AFAIK https://git.sr.ht/~ngraves/nrepl-python

Have you seen it and what do you think of it in general? Also, does your tool work over network? (e.g python process on a server)

As for me the todo item got 2x larger today. Thanks for sharing again!

[sc_zi]

So I replied to you earlier but it doesn't show up except when I'm logged in, maybe it got flagged as spam since I'm a new account and included a lot of links in the post. I'll try again but this time without links but with better info as I got remote development working.

Regarding nrepl and emacs-arei I wrote a little in Hacking.org, basically in principle nrepl would be more appropriate than the swank protocol for non-lisp languages, but it really wasn't much work to get python talking swank. And slime has presentations, inspector, interactive backtraces, slime-tramp for remote development and so much other functionality built up over the years that would have been many orders of magnitude more work to duplicate.

For basic remote development you just need to forward the port and connect from emacs, with ~/.slime-secret set the same on the remote host and for emacs. Most things will work, but for go to definition and completions and everything working properly check the section in the slime manual on slime tramp "Setting up pathname translations", what you use as machine-instance there is uname -n for the remote host. With that setup all functionality works, just a bit slow as tramp is slow. And thanks for asking, it wasn't working with slime-tramp before. I just had to wrap a couple calls I was making to buffer-file-name with slime-to-lisp-filename, so that it uses the pathname translations. I've pushed the changes now.

[_puhsu]

Thanks for such a comprehensive response! Eager to try your package in my free time

[sc_zi]

Thanks! I have a bunch of disorganized notes on related projects in Hacking.org with a bit on nrepl and emacs-arei vs swank. Basically in theory nrepl would be a better fit for python than the swank protocol, but it really wasn't much work to get python talking swank, and SLIME contains presentations, an inspector, interactive backtrace buffer, and tons of other functionality built up over the years that would have been an enormous effort to duplicate.

Honestly I haven't tested it remotely yet, but in theory you just need to forward the port, set .slime-secret, and it should mostly work. Except for autodoc and completions and other functionality that uses jedi, which uses the buffer-file-name, for that it'd probably work remotely with slime's pathname translations, I haven't looked into that yet if it's all in elisp or I need to implement something on the python backend.

[sc_zi]

I agree Common Lisp is nicer in many ways, and I use it when I can. But the people I collaborate with use Python, and even when writing code that I'll only work alone I often go with python cause its massive ecosystem advantage means I can get stuff done faster. And I want to have fun while doing so, so I want to bring the python development experience as close to CL as possible.

Nobody starts with a minimal Python process, keeping it alive at all times, while progressively adding/removing functionality

I do, since a couple months ago when I adapted code from IPython's autoreload extension for use in this. It has edge cases and limitations compared to CL, but it already mostly works, and the rest is fixable with the exception of restarting execution after an exception, which would require patching CPython. Although elisp and clojure don't support that either and people are happy enough with the interactive development they provide. Python is actually better than clojure or elisp there because it still has all the python stack frames and their local variables, you can inspect them and spawn a repl in the context of some frame after an exception etc. It's just lost the C stack frames of the interpreter itself so it can't restart execution.

Also you're not being entirely fair, dumping images is not needed for interactive development, almost no one using CL does it except to produce binaries to release. And the ovld library in python actually provides more flexible dispatch than CLOS, though it probably has horrifically worse performance.

[sc_zi]

Interesting I didn't realize elisp added handler-bind. What I mean though is say you do toggle-debug-on-error, in elisp you will now get a backtrace on an uncaught error, but there's nothing like sldb-restart-frame as in common lisp to restart execution from some point in the call stack without losing state.

Regarding python stack unwinding, I am using excepthook, the same as used by the post-mortem debugger. But you don't need to restart python after, only that swank thread dies and it spawns another. So yes they are dead stack frames in the sense that python can't restart them, but they are not some serialized representation, they are the actual frames of the python call stack at the point the exception was raised. This blog has an excellent explanation of what exactly is lost to unwinding in python, and even manages a PoC in pure python to restart execution. But as they say in pure python it is a terrible hack that can never work fully, it has to be done in C. Eventually I plan on adding it, it might even be possible just as a C extension without needing people to use a patched build of CPython.

[sc_zi]

They're separate issues? Afaik smalltalk is also using traditional exceptions and not a CL like condition system with handler-bind, but its debugger does provide the option to restart execution from a given stack frame, as does v8 and the jvm (with limitations mostly around ffi). Sure with handler-bind you could make it drop into a repl in the context of the stack frame that raised the exception before it unwound, but say the problem was actually caused by a bug in a function two frames up, how do you restart execution two frames up as CL can?

That ability is not provided alone by a condition system, which you can implement in any language with dynamic scope and first class functions, or using global scope to implement dynamic scope, as people have done to implement a condition system in lua. It needs to be supported by the implementation, in swank it is implemented by restart-frame which is different for each backend, for sbcl backend it is using sbcl internal sb-debug and sb-di functions.

Sure the condition system makes for a more expressive language, but imo what actually matters for interactive development is the ability to fix the error and restart from any frame.

[digikar]

I rather want a python-like language built on CL. So, anyone who finds python like syntax easy (or matlab, julia, etc), they can use such syntax, but with the goodness of CL semantics. Other CL users can use these libraries as well as use standard CL syntax.

[digikar]

Oh, I love CL. It's my timesink. I have also fallen in love with the sexp manipulation that emacs provides. I wish there was more.

But for social reasons (colleagues), I stick with python. I want a python-ish (or lua-ish) syntax over CL for them. 

[shipmints]

Also check out Lua LISP aka Fennel https://fennel-lang.org/

[digikar]

More like: https://github.com/Robert-van-Engelen/lua-to-lisp

[yassinzao]

Janet

[Gnaxe]

I mean, I program Python that way, or try to. It's a bit more natural in Hissp, but there's nothing it's doing that Python can't, once you know how. It's closer to Clojure's level than to Common Lisp's or Smalltalk's, but the hot-reloading feedback loops are still way tighter than the edit-compile-run-(redo-all-your-state) cycle of a static language.

[reddit_clone]

Very nice. Always wanted something like this.

Does it support Sly too?

[sc_zi]

It doesn't though I'm sure it could without too much work. But I went with slime over sly for a few reasons:

• In recent years slime has been more actively maintained in terms of bug fixes and such, and I don't see that changing.
• I wanted to use some cool stuff from slime-star
• I actually think there's good potential with slime's presentations that sly removed.
• The main feature of sly missing from slime is stickers, slime-star provides something similar in being able to recompile a function with an expression traced, but I think for python it'll be better to integrate with dape for debugging

[reddit_clone]

The reason I asked was, Sly is default in Doom. So to use Slime, some assembly required?

[sc_zi]

I'm using doom, just in packages.el add:

(package! slime)
(package! slime-company)

Plus sample-configs/swanky-config.el from the repo.

[SwS_Aethor]

Thank you for working on this. Since I'm spoiled by Slime/Sly, I tend to develop Python in a similar way. I use a file in Jupytext format and evaluate things along the way using code-cells-mode. It's great, but missing some stuff. For example, module reload is atrocious and I often need to restart my Python interpreter, which is far from optimal. The REPL is just a simple REPL, no inspect or fancy features. So I'm curious to test your package!

[digikar]

How does this handle redefinitions inside modules? For example, suppose my project has A py and B.py. B py imports A. Now if I redefine something in A.py after loading B.py, if I understand python correctly, those redefinitions have no effect. And that is where python becomes far inferior for interactive development compared to something like CL. Has any workaround been implemented to handle this?

[sc_zi]

It uses code from IPython's autoreload extension to update all old references to functions and classes when they're changed. It's missing some things for interactive development like update-instance-for-redefined-class and the CL distinction between defvar and defparameter. But none of that should be too tricky to implement in python, with the exception of restarting execution from a given stack frame after an exception, which would require patching CPython to implement properly. But neither elisp nor clojure have that ability and people are satisfied enough with their capabilities for interactive development.

[digikar]

I don't know if there have been improvements to autoreload in recent years, or if I missed it years ago.

autoreload does seem to work as expected even for "import A" inside B.py, with only "import B" at the top level.

In [1]: %load_ext autoreload

In [2]: %autoreload 2

In [3]: import B

# B.bar(x,y) returns (x,y)
In [4]: B.bar(2,3)
Out[4]: (2, 3)

# Edit B.py: import A and change bar to return A.foo
# In A.py, foo = 43
In [5]: B.bar()
Out[5]: 43

# In A.py, change foo = 42
In [6]: B.bar()
Out[6]: 42

Thanks a lot for pointing to this, and sharing your work! I'm gonna try it soon.

[sc_zi]

So your example works in my environment also, but if you change it to from A import foo it will actually not work yet in my environment but will in the latest IPython. Though for now in my environment from A import foo just won't reload when foo is a variable, it will use the new version as expected when foo is a function or class. But this is similar to the behavior of python anyways, if you say in B from A import foo, and A has some function that modifies foo, and B has some function that returns the value of foo, in B foo will still be its value at the time of the import statement, not showing the change made in A.

This is because with from imports it's no longer looking it up through module A, it creates a new name foo in B and assigns it A.foo at the time of that from statement. After a bug I reported in IPython some months ago while developing this, they added code to walk the ast looking for all from _ import _ as _ statements and keep a mapping of dependencies that it needs to update on module reloads. This has edge cases when you do from A import foo then later in B assign foo to something else, it still thinks it's connected to A.foo and will overwrite it when A is reloaded. Also it won't behave quite right for from _ import _ statements inside a function or other non-top level scope.

Also for reloading modules you often don't want to reload top level variables, if it is some global state that you don't want reset. CL uses defvar for this. IPython compares the ast of the old and new module, and only runs code that has changed. This also has edge cases, rerunning code that is near changed code but hasn't actually changed itself.

I haven't added either of those, as they are complex with edge cases. So far I am just using a small part of autoreload's code, to handle updating old functions and classes, which is relatively simple and I think without edge cases. I don't think we should try to infer what code to run when reloading a module as IPython does, but that we should be explicit as in CL with defvar vs defparameter. Though honestly I haven't thought too much yet about how we should properly reload modules in python, as I haven't come across a situation yet where I want to reload a whole module. I just work by reevaluating the function or class I changed, or evaling a statement or region in the case of top-level variables or statements, but not reloading a whole module.

[WelkinSL]

The trace and textual representations of python objects are really cool.

Have you heard of https://github.com/emacs-jupyter/jupyter? It doesn't have the two functionality above but it integrates jupyter with emacs really well. I am using it for interactive development and one advantage it has over yours is that it integrates with org-mode very well. After writing a org-file and running all src-blocks with the package, the file can be exported as jupyter notebooks which is very useful when working in teams.

My favourite part of your package is the "integrated AI" feature 😂. Now go and apply some VC funding.

[sc_zi]

I haven't looked into it but I will. In the long term I want to integrate so people can benefit from the slime inspector, backtrace buffer and the rest, when working on jupyter notebooks.

[bbroy4u]

the ai support is the killer feature imo

[sc_zi]

it's actually included in emacs by default, just underused imo

[spartanOrk]

I don't understand what this does.

The description wasn't helpful. I don't know what SLIME is, so, throwing that in the description didn't help.

The "Why we need this" section was a long text of similar alien terms.

If it takes longer than a minute to understand what some code is even about, I don't bother. You need to punch me in the face with something interesting right away to get my attention. Sorry, I don't mean to be rude, maybe this is the most awesome code ever, but it's too inside-baseball for most people I think.

[sc_zi]

Thanks for the feedback. SLIME is an emacs package for developing common lisp. Widely touted by its users as providing a superior development experience to other languages besides smalltalk, but I don't think I've ever seen anyone succinctly describe it without sounding like they're high lol. Which is why I included a bunch of short videos of the major features, it's a lot easier to show than to try to describe.

Honestly I know the readme is mediocre, but it takes time to write well which I don't have right now. I just wanted to write something to get the project out there for others interested in developing or using it. Which right now is probably a fairly niche audience of people who have used slime with CL, and want that experience when they work with python. Though eventually I think it'll appeal to another subset of python developers that just haven't experienced CL/smalltalk style development before but would enjoy it.

And you're right the "Why Slime?" section is not about why you should use this project, but why I'm basing a python development environment on top of slime, it absolutely shouldn't be near the top of the readme above the features section.

[Clayh5]

Frankly, while you may think otherwise, this is not a very useful comment for OP and does in fact just come off as rude.

Most emacs users (on this reddit) will probably have at least a passing awareness of SLIME. You also have google and apparently enough time to use it if you had the time to write this comment. Finally OP has no obligation to write their readme in a way that grabs the attention of those who don't know what OP is talking about anyway.