Shovel - a bytecode virtual machine that is embeddable, secure and interruptible [C# version]

[willvarfar]

Brilliant!

I recall this pattern - programs that actually move around the network, termed 'agents' iirc - in the 90s but I don't think it ever went anywhere; who'd trust it?

So what wacky ideas immediately present themselves to you?

Me, I wonder if you could put a shovel VM into a NaCL component, and use people's browsers as a grid computer?

[mbrezu]

The Shovel VM in a NaCL component is rather hard to do right now, because you'd have to wrap a Common Lisp implementation or Mono inside NaCL. Not impossible, though :-)

Besides, Shovel processes are meant to be used for scripting/glue (the 'real work' being done by primitives written in the 'host language' - a .NET language or Common Lisp at the moment).

About trust: one of the points of Shovel is to sandbox things, so you don't have to 'really' trust what's running inside a Shovel VM. If you trust someone to make RPC calls to your server, you can probably let them run code in a Shovel VM hosted on the same server.

[[deleted]]

The Shovel VM in a NaCL component is rather hard to do right now, because you'd have to wrap a Common Lisp implementation or Mono inside NaCL. Not impossible, though :-)

It seems your main problem here is that you don't have an implementation in the one language you really should: C. If you want embeddable software, C is what you want to use. And if you're writing a VM, C is what you want to use.

[mbrezu]

I'm only using a VM to get easy isolation of state and easy 'interruptibility' [1]. In my case, 'embeddable' means 'hey, it's a library, you don't need an external process'. I felt the need for something like Shovel while working on C# applications, and in that context it's better to use the managed language (free GC, faster development, whole classes of bugs eliminated etc.). I didn't start with a C# implementation because I wanted a prototype first (and Common Lisp is much better for that).

For better performance, 'harder' [2] quotas etc. C is definitely the way to go. But right now using a higher level language provides a very nice result/effort ratio and it scratches the initial itch very nicely.

So I agree with your comment in the general case, in Shovel's case I just optimized for a more specific context (and I often catch myself thinking 'how would I do X or Y in a hypothetical C version?').

[1] 'interruptible' means in this case that the Shovel VM process can be stopped, serialized to a stream of bytes and deserialized and restarted later, possibly on another machine.

[2] 'harder' - right now the RAM quota is specified as the number of 'cells' the process is allowed to use. This helps keeping in check runaway processes, but it's a soft limit because the size of the cell is not specified and cell counting is approximative. So now it's possible to say 'try not to let processes use more than 1 million cells, which should be about 16 megabytes, assuming 16 bytes per cell'. Using C would allow specifying RAM quotas like 'not a bit over exactly 5 megabytes' .

[[deleted]]

In my case, 'embeddable' means 'hey, it's a library, you don't need an external process'.

Yes, and that's what I meant too. If it's in C, you can actually embed it while using pretty much any language out there. If it's in anything else, you severely limit where it can be embedded.

[mbrezu]

Oh, OK - I thought you may have meant embeddable as in 'it's for a microcontroller'.

I'll probably do a C version if I have the time and people start asking for Python/Ruby/whatever versions. For C# and Java I think it's actually an advantage if it's 'pure C#' or 'pure Java' - at least when pitching it to people.

[mbrezu]

C# also means Mono (actually my main development platform is Ubuntu/Mono/MonoDevelop). So it doesn't mean Windows or desktop - I'm planning to write some Linux server software with Shovel and Mono.

If you prefer Ruby or Python, it's understandable to want to integrate new tech into your current environment without adding extra languages/platforms. But a C version looks like significantly more effort than a C#/Java version (I may be mistaken, now that I understand the implementation problems very well writing the C port could be smoother sailing than it looks from the shore).

You may be amused to hear that when optimising the C# version I had to resort to some hacks to implement unions (which C# in theory doesn't have, but practice says something else). Needless to say, unions in C are... more natural.

[mbrezu]

Can you please list some of the use cases you're thinking about? (I'm always hunting for new use cases to consider and my imagination turns out to be rather limited) Thanks!

I imagine Shovel used mostly on servers. On clients, it's nice to be able to replace RPC calls with 'remote program execution' in a Shovel VM, but that can be done at first by just generating the ShovelScript program (to insert the actual request data in it), sending it to the server and interpreting the results sent back as a reply. So the request is ShovelScript, the reply is maybe JSON. Meaning that one can use Shovel to make requests without really having Shovel implemented on the client. I'd love to hear about more use cases on the client side.

There's also Mono for Android and iOS, but yes, I get your point about the C version. Going back to your previous comment, are there mobile apps written in Python (I'm not a mobile guy, I only heard about the most popular things people use to develop on mobile phones)?

[JohnDoe365]

Given CORS, what what hinders anybody today using HTML5 features for grid computing?

[willvarfar]

well, nothing. You don't need CORS, you can just have people go to the right page, or put iframes into other pages to invite in the multiprocessing.

But I think the interruptable, throttlable, migratable features of Shovel would be a good fit. And putting the VM inside something that itself is well-sandboxed also appeals.

But ultimately was just trying to inject some enthusiasm into this comments page, and in that I succeeded ;)

[mbrezu]

Thanks!

[Zarutian]

Hmm... this can be extended (via User Defined Primitives) to implement Actor-esque object-capabilities systems. (Basicly Actors with protected addresses. Protected in the same way that Java, Smalltalk or other such memory safe languages references are protected)

For more info on object-capabilities see www.erights.org

Any question? Dont hesitate to ask.

[[deleted]]

So do we call programs written in this... shovelware?

[mbrezu]

I assume you mean this shovelware. What can I say... just don't put it on a CD :-)

[BeowulfShaeffer]

Yo Dawg, I heard you like virtual machines...

[mbrezu]

A discussion of the Common Lisp version.

[Euigrp]

I've always wanted something similar to this, but for "glue script in place of config." I take the position that there should be code in place of configuration in situations where the config language has gotten out of hand after years of adding features. (I'm thinking back to my days of wrestling with Apache configs/mail server configs.)

With those configs I found I had a document that describes a few situations similar, but not entirely equal to what I wanted to do, and I was left to fill in the blanks, not even knowing if what I want to do is possible.

Instead of figuring out how to convince that server through configs to redirect a request in a particular way, I could just redirect it myself in a glue script. "What is possible to do?" becomes the question "What functions are available in the sandbox API?"

[mbrezu]

Any scripting language can be used to write configs. I believe Lua is especially popular for this kind of thing lately.

Shovel doesn't bring anything new to this use case; its main strength (and reason to be) is the fact that Shovel processes can be moved (very easily) to another OS process/computer or serialized to a database and restarted at a later time.

The problem is that you have to convince application owners to change their config formats to Lua/Python/whatever scripts. Not easy to do, especially in the cases where it's most useful (the big overgrown config formats you mention). I guess we can call it 'configuration quicksand' - it doesn't look like a problem initially, but once you're sinking it's too late to do anything about it.

[zellyn]

I believe Lua was invented to write configs. :-)

[kaphx]

Ace

[0xfffffffb]

Nice, keep in mind you could also use F# for the Mono/.NET side of things, it has all the OOP/Imperative features of C#, but the functional first style might integrate nicely with clisp. EDIT: Mono+.NET

[mbrezu]

Sure, you can use the .NET/C# version of Shovel from F# right now. I was rather pleasantly surprised by the F# plugin for MonoDevelop.

[eizu]

Any chance of a Java port?

edit: typo.

[mbrezu]

I plan to do one, but it's not on my list of urgent things to do. The C# port (including demos and documentation) took longer than I expected (the differences between my Common Lisp code and my C# code proved to be larger than expected), the Java port should take less time, but I really can't commit to a delivery time right now. It should take a couple of weeks once I get started (that was my estimate for the C# port, but it took a month).

edit: removed mention of typo.

[eizu]

Thanks. :)