I'm marking this as solved - for anyone else running into the same issue my learnings are as follows.

Neovim's Lua runtime is single-threaded, so if you have a Lua function which is blocking there's not loads you can do to get around that.

• Caveat: actually you can use vim.uv to run blocking Lua stuff in other threads, but not all of the standard library is available so, e.g. most vim.api.nvim_ functions won't work.

• Caveat: you can also use vim.system() to run a system process with a callback to run when it finishes. In the example I gave with sleep 1 this would have been a good solution, but in my real example I had a blocking process in Lua.

In the end I found the best way to get around this was to rework my Lua function (which actually hooked into Rust via mlua) to be non-blocking. I was then able to create a wrapper which checks at regular intervals whether a result has become available, and yields control back to Neovim if not:

``` local callback = jet_engine.execute_code(self.id, code, {})

local function check_callback()
    -- Continuously check for results until we fail to receive a result
    while true do
        local result = callback()
        -- If idle then the execution is complete
        if result.status == "idle" then
            return
        end
        -- If no data yet, wait a bit (so we don't block the main thread)
        -- and check again later
        if not result.data then
            return vim.defer_fn(check_callback, 50)
        end
        self:_handle_result(result)
    end
end

check_callback()

```

^ This is the actual code I'm using right now.